KR102263897B1 - Fingerprint layouts for content fingerprinting - Google Patents

Abstract

A method for identifying media content is disclosed. The method includes receiving fingerprint media content from a content provider to obtain a first fingerprint set, wherein the first fingerprint set includes a first fingerprint subset in a first fingerprint layout and a second fingerprint subset in a second fingerprint layout. includes The method may include receiving a second fingerprint set, wherein the second fingerprint set may include a third fingerprint subset of the first fingerprint layout and a fourth fingerprint subset of the second fingerprint layout. The method includes matching a first fingerprint in a first fingerprint subset to a second fingerprint in a third fingerprint subset or by matching a third fingerprint in a second fingerprint subset to a fourth fingerprint in a fourth fingerprint subset to access media content. can be identified.

Description

Fingerprint layout for content fingerprinting

delete

A media consuming device, such as a smart television (TV), may access broadcast digital content and receive data, such as streaming media, from a data network. Streaming media refers to a service through which an end user can request media content, such as a movie or news, over a telephone line, cable channel, Internet channel, or the like. For example, a user can watch a movie without leaving their residence. In addition, users can access various types of educational content, such as video lectures, without having to physically attend classes at a school or educational institution.
As the number of media consuming devices continues to increase, video content creation and delivery may likewise increase. With the increasing use of media consuming devices to access broadcast digital content and streaming media, content providers or network providers may distribute broadcast digital content or contextually relevant content to viewers consuming streaming media. For example, local broadcasters may include contextually relevant advertisements and interactive content along with broadcast digital content or streaming media.

The present invention will be more fully understood from the detailed description provided hereinafter and drawings of various embodiments of the invention. However, the drawings are not intended to limit the present invention to specific embodiments, but are for explanation and understanding only.
1 is a system diagram of a content distribution network according to an embodiment.
2 is a system diagram of a content overlay network according to an embodiment.
3A illustrates compositing of frames to form media content and an on-screen display to create an image to be displayed on an electronic device according to one embodiment.
3B illustrates a composite of an on-screen display and media content comprising a window according to one embodiment.
3C illustrates a composite of an on-screen display and media content comprising a window according to one embodiment.
3D illustrates compositing of media content, on-screen display, and overlay content to create an image to be displayed on an electronic device according to one embodiment.
4 shows a fully subdivided fingerprint layout of a frame according to one embodiment.
5A illustrates a top subdivided fingerprint layout of a frame according to one embodiment.
5B illustrates a top subdivided fingerprint layout of the frame shown in FIG. 5A with a plurality of blocks, according to one embodiment.
5C illustrates a bottom subdivided fingerprint layout of the frame shown in FIG. 5A according to one embodiment.
5D illustrates a bottom subdivided fingerprint layout of the frame shown in FIG. 5C with a plurality of blocks, according to one embodiment.
6A is a flowchart of a method for identifying media content according to one embodiment.
6B is a flow diagram of a method for identifying media content using a top subdivided fingerprint layout according to one embodiment.
7 schematically shows a block diagram of a system according to one embodiment.

Currently, there are hundreds of program selections offered by cable television (TV) providers and satellite operators. In addition, there are over a hundred major television market segments with dozens of regional television channels. A content provider or broadcaster can create a content feed from a central location and distribute the content feed to multiple locations. For example, a content provider may distribute a television channel to a plurality of network providers. Network providers may be local broadcasters, multi-channel networks, and other content owners or distributors.

The term “content” may refer to media or multimedia. The term “content” may also be a specific term referring to the subject matter of a media rather than the media itself. Likewise, the terms “media” and some compound words including “media” (eg, multimedia or hypermedia) refer to content rather than a channel through which information is passed when it is delivered to an end user/audience. An example of a type of content commonly referred to as a type of media is a “motion picture,” also referred to as a “movie.” Another example is "video," which typically includes video frames and audio samples. “Broadcast” may refer to the transmission of information (eg, media content) to a radio, television, or other electronic device. For example, the electronic device may broadcast media content using a transmitter. In another example, the electronic device may stream media content over the Internet or other private or public network. “Broadcast” may also refer to media content, such as video and audio content, delivered to a distributed audience using a content delivery system, such as over the air (OTA), cable, satellite, or Internet-based networks, in a one-to-many model. may be For example, a broadcast may be media content received from a radio or television transmitter to an electronic device.

Broadcasting or streaming media content, such as TV shows or Internet broadcasts, can be an attractive place to advertise products and services, provide information to viewers, or any combination thereof. To generate revenue for providing a variety of programming choices, content providers (eg, television networks) may place advertisements (AD) in their media content. In general, advertisers have used to send one stream of media content to all viewers on a channel, and local broadcasters target viewers based solely on ratings provided by viewer analytics and rating providers, such as The Nielsen Company®.

While TV advertising is an emerging place to advertise a product or service, traditional TV advertising lacks targeted and personalized AD for different viewers. Automatic content recognition (ACR) systems may use fingerprints to identify media content and present targeted and/or personalized advertisements to viewers. For example, the ACR system may identify media content using content fingerprinting. In one embodiment, the ACR system may be software running on an electronic device, such as a server or a TV. In another embodiment, the ACR system may be a service hosted by a vendor or service provider and accessed by an electronic device via the Internet. For example, the ACR system may be software as a service (SaaS), a platform as a service (PaaS), or an infrastructure as a service (IaaS).

Content fingerprinting may include a processing device that uniquely identifies the media content as its “fingerprint” by identifying, extracting, and compressing the fingerprinting of the media content. As one example, a fingerprint component may be a visual video feature, such as key frame analysis, color, and motion change during a video sequence. The ACR system may identify video content by matching one content fingerprint or content sequence fingerprint from the media content being displayed by the electronic device with a fingerprint stored in a reference database.

However, when the electronic device overlays the on-screen display on the media content, the on-screen display may disrupt the ACR system for identifying the media content. For example, the electronic device may overlay the on-screen display on the media content received from the content provider. The on-screen display may be a logo, subtitle text, emergency message, volume bar, channel number, configuration menu, and the like.

Conventionally, when an electronic device generates a content fingerprint and sends it to the ACR system, the content fingerprint of the media content may include an on-screen display, and since the on-screen display does not include a stored fingerprint, the ACR system detects the content fingerprint. It can disrupt matching with stored fingerprints of media content. When the content fingerprint includes an on-screen display and no stored fingerprint, at least a portion of the stored fingerprint is different from the content fingerprint.

Described herein are methods, systems, and apparatus that allow an ACR system to identify media content including an on-screen display that is overlaid or embedded in the media content. In one embodiment, the device may generate the fingerprint using a different frame format. The device may be communicatively coupled to the electronic device, and the device may communicate a fingerprint to the ACR system. The ACR system may identify the displayed media content by comparing the fingerprint with other fingerprints. The ACR system may use different frame formats to avoid disturbances due to on-screen display within the media content. By using different frame formats for fingerprints, false positives can be avoided when matching fingerprints, as described in the embodiments herein. By using a different frame format it is also possible to avoid missing the actual match when matching the fingerprint.

1 is a system diagram of a content distribution network 100 according to one embodiment. In the content distribution network 100 , a content provider 102 may broadcast a content feed to a local operator 106 . The local operator 106 may include a headend 104 and an ACR fingerprinting server 105 . The headend 104 of the local operator 106 may receive a content feed from the content provider 102 . The headend 104 may generate a local content feed based on the received content feed. For example, headend 104 may be a regional affiliate broadcaster that receives network channels along with programs and advertisements from national or global broadcasters. For example, the headend 104 may be the source of a broadcast feed before the broadcast facility that transmits the broadcast feed. As another example, the headend 104 may be a distribution amplifier. A distribution amplifier may receive a source feed and make multiple outputs for different devices from the same source feed. Output feeds may be routed to various distribution outlets, such as over-the-air (OTA) broadcasts, delivery to cable operators, delivery to satellite operators, and/or delivery to online streaming operators.

The headend 104 may communicate the local content feed to an ACR fingerprinting server 105 , an over-the-air (OTA) broadcaster 108 , and/or a multi-channel video programming distributor (MVPD) 110 . For example, the ACR fingerprinting server 105 may be a device installed in a broadcast facility. The ACR fingerprinting server 105 may receive a feed from the headend 104 . The ACR fingerprinting server 105 may fingerprint the feed and send the fingerprint to the cloud-based web service. In one embodiment, the ACR fingerprinting server 105 may be installed inside a broadcast facility to fingerprint the feed before it is delivered to a display device, such as a television (TV).

The OTA broadcast station 108 and/or the MVPD 110 may communicate the local content feed to the electronic device 115 . Some examples of electronic devices 115 include electronic devices 118 and 120 , a set top box 114 that streams media content to electronic devices 118 and 120 , and a set-top box 114 that allows users to stream local content feeds, such as wirelessly. and other devices 116 that enable it. For example, the electronic device 115 may include a display (eg, a liquid crystal display or a touch screen display) for displaying information.

For example, the OTA broadcaster 108 may broadcast a local content feed using a traditional local television channel or radio channel. In this example, electronic devices 118 and 120 may include antennas (eg, TV antennas or radio antennas) for receiving local content feeds. As another example, the MVPD 110 (eg, a cable or satellite station) may communicate a local content feed to the set top box 114 . In this example, set top box 114 may format a content feed for electronic devices 118 and 120 and communicate the formatted content feed to electronic devices 118 and 120 . Electronic devices 118 and 120 may include display devices for displaying local content to a viewer, such as a television screen or a touch screen.

The various components of the content distribution network 100 may be integrally configured with or connected to the electronic devices 118 and 120 . For example, the electronic devices 118 and 120 may be smart televisions. For example, a smart TV may include an antenna, a set-top box 114, or an integrated ACR system and display device as a single unit. In another example, a smart TV may include a processing device and radio frequency (RF) circuitry. The processing device may execute an operating system (OS) and/or one or more applications, such as applications for accessing video-on-demand (VOD) content. The processing device may communicate over a wireless local area network (WLAN) using an antenna in a radio frequency (RF) circuit. For example, the processing device may connect to the Internet using RF circuitry to access interactive media or streaming media on demand online.

The ACR fingerprinting server 105 may analyze the local content feed and determine fingerprint information (eg, fingerprint). The ACR fingerprinting server 105 may communicate the fingerprint to the ACR system 124 or 126 . ACR systems 124 and 126 may be different ACR systems chosen by the device manufacturer, such as a smart TV manufacturer. For example, the ACR system 124 or 126 may include processing logic or processing devices to execute instructions or perform processes. As another example, ACR systems 124 and 126 may be ACR web services for different ACR vendors.

The ACR fingerprinting server 105 may enable the use of indexing of clients from multiple ACR vendors. For example, different consumer electronics manufacturers (CEMs), such as TV manufacturers, may independently select ACR vendors to be integrated into their electronic devices. To enable ACR-based functionality on a particular CEM's device, the ACR system may index broadcast feeds using an ACR vendor selected by the CEM and integrated into the CEM's television. One advantage of integrating multiple ACR vendors into the ACR fingerprinting server 105 is that entities such as advertisers, advertising agencies and content providers can interface with a single platform and deliver advanced advertising and content experiences to electronic devices from multiple CEMs. that it can

For example, the ACR system 124 may be coupled to a reference database 125 . In another example, the ACR system 126 may be coupled to a reference database 127 . Reference databases 125 and 127 may store fingerprints and media content. The content management device 122 may be a server component that receives messages from the ACR systems 124 and/or 126 . The message may include information indicating the media content that the electronic device 118 or 120 is trying to display or is currently displaying. For example, in response to the content management device 122 receiving the message, the content management device 122 creates an impression or association between the electronic device 118 or 120 and the media content stored in the database 123 . can be recorded In another example, in response to the content management device 122 receiving the message, the content management device 122 may create and submit a query to the database 123 for available overlay content to be overlaid over the media content. .

In one embodiment, the ACR system 124 or 126 may send a fingerprint request to the electronic device 118 or 120, respectively, where the fingerprint request includes a frame format for the fingerprint. The electronic device 118 or 120 or the ACR system integrated in the electronic device 118 or 120 may generate a fingerprint in a frame format and transmit the fingerprint back to the ACR system 124 or 126 . The ACR system 124 or 126 may use the fingerprint for media content identification.

A frame format may include a frame that is divided into a plurality of segments, which will be described in more detail in the following paragraphs. For example, the frame format may exclude one or more of the plurality of segments from being used by the ACR system 124 or 126 to match a content fingerprint to a stored fingerprint. For example, one or more excluded segments may include logos, subtitle text, urgent messages, and the like. In another embodiment, the electronic device 118 or 120 may transmit the entire fingerprint to the ACR system 124 or 126 . The ACR system 124 or 126 may reformat the fingerprint to exclude one or more segments of the fingerprint defined by the frame format. For example, the ACR system 124 or 126 may request a variety of different fingerprint layouts. As another example, the ACR system 124 or 126 may reformat the fingerprint into a variety of different formats to be used for content recognition.

To find a match between the content fingerprint from the electronic device 115 and the fingerprint stored in the reference database, the ACR system 124 or 126 may perform a one-to-one comparison between the content fingerprint and the stored fingerprint. For example, the ACR system 124 or 126 may use an O(n) algorithm (eg, a linear algorithm) for comparison, where n is the number of elements in the reference database. When an ACR system uses an O(n) algorithm to find entries in a database, the ACR system may have to search all or a relatively large number of stored fingerprints in a reference database to find a match.

The content management device 122 filters the overlay content using the device information for the electronic device 118 or 120 . The content management device 122 may transmit the overlay content to the electronic device 118 or 120 through the ACR system 124 or 126 . In one embodiment, the electronic device 118 or 120 may display the overlay content to the viewer. In another embodiment, the electronic device 118 or 120 may replace the media content with the overlay content and display the overlay content to the viewer.

In one embodiment, the ACR system 124 or 126 may cause a trigger to be displayed on the electronic device 118 or 120 . The electronic device 118 or 120 may recognize the trigger and send a request to the content management device 122 to record an impression or association between the electronic device 118 or 120 and the media content in the database 123 . For example, the electronic device 118 or 120 may send a request to the content management device 122 to record whether the viewer changes channels when the electronic device 118 or 120 displays selected media content. .

Automatic content recognition may be performed in a distributed architecture as described herein. In one embodiment, the ACR system may be located on any of the electronic devices 115 , 118 or 120 . In one embodiment, the ACR system may perform automatic content recognition. In another embodiment, a local ACR system or local ACR fingerprint matcher is integrated into the electronic device 115 , 118 or 120 to perform automatic content recognition. In another embodiment, the ACR system may perform ACR fingerprint matching as described herein. In another embodiment, the ACR system may be a client-side matching component.

Electronic device 115 , 118 , or 120 may be a media device, such as a television, mobile phone, or dongle. The ACR system may be software or firmware that is executed or operated on the electronic device 115 , 118 , or 120 . The ACR system may fingerprint media content (eg, audio content, video content, subtitles, emergency messages, etc.) during playback of the media content. The ACR system may communicate with the ACR system 124 or 126 to trigger an event to facilitate identification of content and cause the content overlay system to perform dynamic ad replacement and/or overlay content overlay over media content. The interaction between the ACR system and the ACR fingerprinting server 105 is described in more detail below.

The ACR fingerprinting server 105 may parse the local content feed, which may include an ordered sequence of frames from the local content feed, and may capture the fingerprint. The ACR fingerprinting server 105 may communicate the fingerprint to the ACR system 124 or 126 . ACR systems 124 and 126 may be different ACR systems selected by the device manufacturer, such as a smart TV manufacturer. For example, ACR fingerprinting server 105 may format fingerprints for different ACR systems 124 and 126 .

ACR systems 124 and 126 may establish communication connections with different electronic devices 115 . The electronic device 115 may communicate the fingerprint information to the ACR systems 124 and 126 , respectively. When the ACR system 124 or 126 receives the ACR fingerprint information from the electronic device 115 , the ACR system 124 or 126 may match the received fingerprint with that generated by the ACR fingerprinting server 105 . . When a match occurs and the ACR system 124 or 126 identifies the content, the ACR system 124 or 126 may communicate the ACR event to the content management device 122 .

As another example, the ACR system 124 or 126 may receive ACR fingerprint information from one of the electronic devices 115 and match the received fingerprint with the fingerprint generated by the ACR fingerprinting server 105 . When a match occurs and the ACR system 124 or 126 identifies the content, the ACR system 124 or 126 may notify the electronic device 115 of the ACR event. The electronic device 115 may communicate ACR event information to the content management device 122 . Alternatively or additionally, the ACR system 124 or 126 may communicate ACR event information directly to the content management device 122 . The ACR event information includes information indicating advertisement in the local content feed, information indicating selected or flagged content in the local content feed, or information indicating change of the content channel to new media content in the electronic device 115. may include

ACR event information from different ACR systems 124 and 126 may have different formats and the content management device 122 may normalize the data into a common format. The content management device 122 may store normalized data in the database 123 . For example, the content management device 122 may receive disparate data sets from the ACR systems 124 and 126 , wherein the disparate data sets are similar but not identical data, such as identical content, but Contains differently formatted data. The content management device 122 may process and reformat the heterogeneous data set to generate a single data model or format (eg, a reformatted data set). The content management device 122 may store the reformatted data set in the database 123 .

In one embodiment, to normalize heterogeneous data sets from the ACR systems 124 and 126 , the content management device 122 may remove or filter data from the data sets. For example, some data sets may include fields or data that may be irrelevant to the content management device 122 . In this example, the content management device 122 may remove or filter the irrelevant data. In another embodiment, to normalize heterogeneous data sets from ACR systems 124 and 126 , content management device 122 may map fields within the data sets. For example, when the content management device 122 receives the first data set from the ACR system 124 and receives the second data set from the ACR system 126 , the data of the first data set and the second data set At least some of the fields may be common to both the first data set and the second data set. However, the common data field may be located in different places in the first data set and the second data set. In this example, the content management device 122 may map different data fields of the first data set and the second data set to normalized fields. When the content management device 122 maps the data field, the same data field is mapped to the same data field location in the database 123 .

In another embodiment, to normalize heterogeneous data sets from ACR systems 124 and 126 , content management device 122 may derive data from the data sets. For example, data from ACR systems 124 and 126 may not include all fields necessary to populate data fields in the database. However, the content management device 122 may use other fields in the data set from the ACR systems 124 and 126 to derive data for these data fields.

For example, database 123 may include data fields for state fields within a country, designated market area (DMA) fields, and country and/or city fields, but from ACR systems 124 and 126 . The data set of can contain only zip code (ZIP) codes. In this example, the content management device 122 may derive data for a field in the database using the ZIP code. In another example, the data set may include Internet Protocol (IP) addresses of ACR systems 124 and 126 without including any geographic location information. In this example, the content management device 122 may derive state, DMA, country, city and ZIP code information using a geo-IP lookup service.

As another example, the database 123 may include demographic fields, such as an age field, a gender field, a household income field, and the like. However, data sets from ACR systems 124 and 126 may not contain demographic fields or demographic data. In this example, the ACR systems 124 and 126 may provide the IP address of the electronic device 115 to the content management device 122 . The content management device 122 may determine demographic data for filling the data field of the database by using the IP address.

In another example, the fields in the first data set from the ACR system 124 include local time zone information, such as a mountain summer time (MDT) zone and the second data set from the ACR system 126 is another information from time zones, such as Coordinated Universal Time (UTC) zones. The database may store all data using UTC, and the content management device 122 may convert the local time into UTC before storing the data in the database 123 .

In another embodiment, the content management device 122 uses the normalized data to generate reports or data on the user's viewing behavior (viewing data) across different ACR technology vendors and smart TVs or other internet-connected video devices. can do. The content management device 122 and the electronic device 115 may include a communication interface for communicating information, for example, overlay content between the electronic device 115 and the content management device 122 . For example, the communication interface may communicate information using a cellular network and/or a wireless network. For example, if the communications network has a third generation partnership project (3GPP) release 8, 9, 10, 11, or 12 or IEEE® (Institute of Electronics and Electrical Engineers, Inc.) 802.16p, 802.16n, 802.16m-2011, It may be a cellular network employing 802.16h-2010, 802.16j-2009, 802.16-2009. In another embodiment, the communication network may conform to IEEE® standards developed by the Institute of Electronics and Electrical Engineers, Inc., such as IEEE 802.11-2012, IEEE 802.11ac, or IEEE 802.11ad standards. It may be a wireless network (eg, a network using the Wi-Fi® technique developed by the Wi-Fi Alliance). In another embodiment, the communication network may deploy a Bluetooth® connection developed by the Bluetooth Special Interest Group (SIG), such as Bluetooth v1.0, Bluetooth v2.0, Bluetooth v3.0, or Bluetooth v4.0. have. In another embodiment, the communication network comprises Zigbee® connections developed by the ZigBee Alliance, such as IEEE 802.15.4-2003 (Zigbee 2003), IEEE 802.15.4-2006 (Zigbee 2006), IEEE 802.15.4-2007 ( Zigbee Pro).

For example, the content management device 122 may instruct the electronic device 115 to replace the segment of the local content feed received from the OTA broadcasting station 108 or the MVPD 110 with the overlay content. In another example, the content management device 122 may instruct the electronic device 115 to overlay or superimpose the overlay content on a segment of the local content feed. Content management device 122 may aggregate ACR event information across multiple ACR systems 124 and 126 and communicate overlay content to different electronic devices 115 (eg, electronic devices from different device manufacturers). .

The content management device 122 may also establish communication connections with other devices 116 . For example, other devices 116 may communicate with electronic devices 118 and 120 and provide an additional screen (eg, a second screen) to display overlay content. For example, the electronic devices 118 and 120 may receive a local content feed from the OTA broadcasting station 108 or the MVPD 110 and display the local content feed to a viewer. Other devices 116 may also communicate ACR event information to the ACR systems 124 and 126, as mentioned above, when an ACR event occurs. When the content management device 122 receives the ACR event information, the content management device 122 may communicate the overlay content to other devices 116 .

For example, electronic devices 118 and 120 may continue to display local content while other devices 116 display overlay content. In another example, both electronic devices 118 and 120 and other devices 116 may display overlay content. In another example, electronic devices 118 and 120 and other devices 116 may display segments of overlay content and segments of local content feeds. In another example, electronic devices 118 and 120 and other devices 116 may display different local content feeds and different overlay content.

For example, electronic devices 118 and 120 and/or other devices 116 display overlay content when electronic devices 118 and 120 and/or other devices 116 receive overlay content. can do. In another example, electronic devices 118 and 120 and/or other devices 116 may delay displaying the overlay content for a threshold period of time. In one embodiment, the threshold time period may be a specified time period. In another embodiment, the content management device 122 may select a time period for delaying the electronic devices 118 and 120 and other devices 116 from displaying the overlay content.

2 is a system diagram of a content overlay network 200 according to one embodiment. The content overlay network 200 may include an electronic device 210 , a first ACR system 220 , a content database 230 , an overlay content server 240 , a record database 250 and an overlay content management system 260 . can The ACR system 220 provides broadcast digital content to the electronic device 210 . The broadcast digital content may include overlay content that the ACR system 220 superimposes or overlays on a media segment for the electronic device 210 to display to a viewer. A media segment may be a segment or a portion of media content. For example, a media segment may be a segment or portion of media content for a TV program or movie, such as a TV show clip or movie clip. In another example, the media segment of the media content may be an advertisement that is part of the media content for a TV show or movie.

In one embodiment, the ACR system 220 is integrated into the electronic device 210 . In another embodiment, the ACR system 220 is coupled to the electronic device 210 . For example, the ACR system 220 may be directly connected to the electronic device 210 using an input port of the electronic device 210 . For example, the ACR system 220 is an HDMI dongle that is plugged into a High-Definition Media Interface (HDMI) input terminal of the electronic device 210 . In this example, the ACR system 220 may include an HDMI input port, an HDMI output port, and a network interface for receiving broadcast content or media segments from a content provider. In another example, the ACR system 220 may be indirectly connected to the electronic device 210 via another device connected to the electronic device 210 , for example, an audio and/or video (A/V) device.

The electronic device 210 may display the targeted or personalized overlay content segment to the viewer using the ACR system 220 . The overlay content segment may be an audio segment, a video segment, and/or a graphics segment of the content. For example, an overlay content segment may be a media object or file containing an audio segment or a video segment of content that is an alternate AD. In another example, the overlay content segment is an audio segment or video segment of a larger audio object or larger video object that includes other overlay content segments.

In one embodiment, the ACR system 220 may utilize digital workmarking of ACR broadcast content. For digital watermarking, a content provider or broadcasting station may process the broadcast content and insert the watermark data into the content signal before broadcasting the content signal. The electronic device 210 or the ACR system 220 may detect the watermark data to identify the media segment and synchronize it with the broadcast content.

In another embodiment, the ACR system 220 may identify the media content of the broadcast content using content fingerprinting when the electronic device displays the media content. Fingerprinting the content may include the ACR system 220 matching the sequence of the content fingerprint for the media content being displayed by the electronic device 210 with the fingerprint stored in the reference database. An advantage of content fingerprinting is that a media segment may be processed (eg, real-time processing) while the media segment is being broadcast (eg, processed in real-time) or processed (eg, substantially real-time processing) of a media segment within a threshold period of time, eg, seconds, after the media segment is broadcast. that it can The content overlay network 200 may adjust the overlay content segment (eg, real-time or substantially real-time adjustment) based on viewer behavior during content broadcast when the electronic device 210 displays the content. For example, the viewer action may include the electronic device 210 receiving a command to change a channel or a command to stop displaying the media content from the user device. In another example, the viewer action may include the electronic device 210 receiving an instruction from the user device to display time-shifted content. When a viewer selects or changes content, the content overlay network 200 may switch overlay content overlaid on the media content.

The overlay content server 240 is connected to the ACR system 220 . The overlay content server 240 may receive overlay content from a content provider or advertiser. The overlay content server 240 may store a media segment, an overlay content segment, and format information for the media segment and the overlay content segment. The overlay content server 240 may transmit the media segment to the electronic device 210 in an overlay format selected from the overlay format.

The records database 250 is coupled to the overlay content server 240 . The records database 250 may maintain a record of the media segment or the overlay content segment displayed by the electronic device 210 . In one embodiment, the electronic device 210 has a unique identifier (UID) associated therewith that uniquely identifies the electronic device 210 . The UID may also be associated with a media segment or an overlay content segment to indicate a media segment or an overlay content segment displayed by the electronic device 210 .

In another embodiment, the history database 250 may receive messages from the overlay content server 240 . For example, the message may include a media segment ID identifying the media segment and a UID associated with the media segment ID. As another example, the message may include an overlay content segment ID identifying the overlay content segment and a UID associated with the overlay content segment ID. History database 250 may store various media segment IDs and overlay content segment IDs along with their associated UIDs for subsequent retrieval by overlay content server 240 . The overlay content server 240 may search the history database 250 to determine when the respective media segment ID or media segment or overlay content segment associated with the overlay content segment ID was displayed on the electronic device 210 along with the UID. have.

In one embodiment, the overlay content management system 260 may include an input device for receiving configuration information for the media segment and the sequence of the overlay content segment. For example, the input device may be a software interface (eg, a web portal) on the electronic device, where a user may input setting information to set a sequence of media segments and overlay content segments in the overlay content server 240 . For example, overlay content management system 260 may be a website accessible to end users to create, view, update, and delete overlay content and replacement video.

For example, a user may use an input device to set rules or criteria for overlay content server 240 to use to select an overlay content segment from a set of overlay content. In one embodiment, a rule or criterion may include a sequence for displaying items in an overlay content set, such as media content and overlay content segments. In another embodiment, the rules or criteria may include different overlay content segments taking into account the user's demographic information. For example, when the user is a male, the electronic device 210 may display the selected overlay content segment. In another example, when the user is a woman, other overlay content segments may be displayed. In another embodiment, the rules or criteria may include displaying different segments of the overlay content taking into account time of day. For example, in the morning, the electronic device 210 may display an overlay content segment related to breakfast, in the evening, the electronic device 210 may display an overlay content segment related to dinner, and at night The electronic device 210 may display an overlay content segment related to the adult theme. In another embodiment, the overlay content server 240 may include an input device for receiving media content and configuration information for arranging the overlay content segments in a linear sequence. For example, the user may send setting information to the overlay content server 240 using a graphical user interface (GUI) to align the sequence of the media segment and the overlay content segment.

3A illustrates one compositing of a frame comprising media content 302 and an on-screen display 304 to create an image to be displayed on the entire device, according to one embodiment. Compositing may be a technique for creating a display image by stacking two or more videos or graphics on multiple tracks, such as media content 302 and on-screen display 304 . A processing device, such as a processing device integrated into a smart TV or ACR system, may mix video or graphics into one image. In one embodiment, each track may be one layer. For example, the top layer or top layer, such as the on-screen display 304 , may be the dominant layer. The main layer may be displayed before the media content 302 . For example, the on-screen display 304 may cover at least a portion of the media content 302 (eg, a bottom layer or a bottom layer).

Layers of images may be received and composited on one or more devices. In one embodiment, the media content 302 may be stored or reside on a first device and may be transmitted to a second device via an interface, such as a high definition multimedia interface (HMDI) connection. The second device may synthesize the received media content 302 with the on-screen display 304 to display the final composite image.

In another embodiment, the media content 302 and the on-screen display 304 may be stored or reside on the first device. The first device may synthesize the received media content 302 with the on-screen display 304 and display it as a final composite image. Thereafter, the first device may transmit the final composite image to the second device through the interface. In another embodiment, the media content 302 and the on-screen display 304 may be stored or present on a single device, where the single device composites the media content 302 and the on-screen display 304 and combines the composite image. can be displayed.

The processing device that composes the media content 302 or the on-screen display 304 may include changing the opacity or transparency of at least a segment of the media content 302 or the on-screen display 304 . For example, compositing may include changing the opacity level of a segment of an on-screen display 304 (eg, a main layer) so that a viewer can view a portion of the media content 302 . For example, the processing device may form a window 306 within the on-screen display 304 corresponding to a location of the bottom of the on-screen display 304 . The window 306 may include information (eg, text, graphics or video) that may be superimposed on the media content 302 . For example, the information in window 306 may include subtitle text. The processing device may set the opacity level of the window 306 to be opaque or solid and may set the opacity level of the remaining portion 308 of the on-screen display 304 to be transparent. When the opacity level of the remaining portion 308 is transparent, the media content 302 may be visible to the viewer.

3B illustrates a composite of media content 302 including window 310 and on-screen display 304 , according to one embodiment. Unless otherwise noted, some of the features of FIG. 3B are the same as or similar to some of the features of FIG. 3A when indicated by the same reference numerals. The processing device may form a window 310 in the on-screen display 304 that corresponds to a location on top of the on-screen display 304 . The pane 310 may include information (eg, text, graphics, or video) that may be superimposed on the media content 302 . For example, the information in the window 310 may include text for an emergency message. The remaining portion 308 of the on-screen display 304 is made transparent so that at least a portion of the media content 302 can be viewed through the remaining portion 308 .

3C illustrates a composite of media content 302 including windows 312 - 318 and on-screen display 304 , according to one embodiment. Unless otherwise noted, some of the features of FIG. 3C are the same as or similar to some of the features of FIG. 3A when denoted by the same reference numerals. The processing device may form windows 312 - 318 in the on-screen display 304 corresponding to the locations of the corners of the on-screen display 304 . The panes 312 - 318 may include information (eg, text, graphics or video) that may be overlaid on one or more corners of the media content 302 . For example, information in one or more of windows 312 - 318 may include text and/or graphics for a logo. The remaining portion 308 of the on-screen display 304 is made transparent so that at least a portion of the media content 302 can be viewed through the remaining portion 308 .

3D illustrates the compositing of media content 302 , on-screen display 304 , and overlay content 318 to create an image to be displayed on an electronic device according to one embodiment. For example, the top layer or main layer may be the overlay content 318 . For example, the overlay content 318 may cover or mask the media content 302 (eg, the bottom layer or bottom layer) and the on-screen display 304 (eg, the middle layer). In this example, the on-screen display 304 may cover or mask the media content 302 .

For example, the processing device may adjust the opacity level of a portion of the overlay content 318 such that a viewer may view the portion of the on-screen display 304 below or behind the overlay content 318 . In yet another example, the processing device may configure the overlay content 318 and the portion of the on-screen display 304 such that a viewer may view the overlay content 318 , the on-screen display 304 and the portion of the media content 302 . You can adjust the opacity level.

4 shows a fully subdivided fingerprint layout of a frame 400 according to one embodiment. Frame 400 may be a composite of media content 302 and on-screen display 304 ( FIGS. 3A-3D ). The subdivided fingerprint layout may include the ACR system subdividing the frame 400 into a plurality of blocks 410 - 440 using the fully subdivided fingerprint layout. Each of blocks 410 - 440 may represent a different segment or location formed within frame 400 . For example, blocks 410-440 may be rectangular or square. As another example, blocks 410-440 may be polygonal, circular, oval, and the like. In another example, each of blocks 410 - 440 may be approximately the same size. In another example, one or more of blocks 410 - 440 may be of different sizes. The number, shape, and size of blocks in one frame are not limited, and the number and shape and size of blocks may vary.

In one embodiment, frame 400 may be subdivided into a 4x4 grid with 16 blocks 410-440. Corner blocks 434 , 436 , 438 , and/or 440 may include an on-screen display, such as a logo. For example, a corner of the on-screen display 304 may be defined as an available location for displaying a logo.

For example, when electronic device 118 ( FIG. 1 ) receives a request from ACR system 124 to form a fingerprint for frame 400 using a fully subdivided fingerprint layout, electronic device 118 may A complete fingerprint including blocks 410-440 may be generated. In this example, the electronic device 118 may remove one or more of the edge blocks 434 , 436 , 438 , and/or 440 from the fingerprint and send the fingerprint to the ACR system 124 . In another example, when the electronic device 118 receives a request from the ACR system 124 to generate a fingerprint for the frame 400 using a fully subdivided fingerprint layout, the electronic device 118 generates a corner block ( 434 , 436 , 438 , and/or 440 ). The electronic device 118 may transmit the fingerprint to the ACR system 124 .

For example, when frame 400 is divided into a 4x4 grid with 16 blocks 410-440 and blocks 434, 436, 438, and 440 are removed, there is a total of 12 in a fully subdivided fingerprint layout. There are two blocks. In this example, the fully subdivided fingerprint layout has 12 block ranks with 12! different permutations (ie, 12!=479001600 permutations). In one embodiment, the ACR system may store 7 days of media content for fingerprinting, assuming that the media content has 30 frames per second. For example, for a 7-match video, the maximum total number of frames is: 7 days * 24 hours * 60 minutes * 60 seconds * 30 frames = 18144000 frames, where 479001600 permutations/18144000 frames = 26.4 times. In another example, for a 3x3 block, the fingerprint may have different permutations of 9!. In another example, for a 5×5 block, the fingerprint may have 25! different permutations.

The number of blocks in the subdivided fingerprint layout is not intended to be limiting. The number of blocks can vary from two blocks to hundreds of thousands of blocks. For example, ACR system 124 or electronic device 118 may be completely configured to provide a sufficient number of permutations to avoid a threshold number of collisions in which a plurality of fingerprints in reference database 125 match fingerprints for frame 400 . The number of blocks to be used for the subdivided fingerprint layout can be selected. In one embodiment, a 3x3 grid of blocks may have an insufficient number of permutations that may result in a number of collisions exceeding a threshold number of collisions.

In another embodiment, the ACR system 124 or the electronic device 118 may select the number of blocks to be used for a fully subdivided fingerprint layout to avoid each block containing a small number of pixels. For example, ACR system 124 or electronic device 118 may be reformatted to remove edge blocks 434, 436, 438, and/or 440 from the fingerprint, and blocks 410-432 to be grayscale blocks. have. For example, each grayscale block may have an average value.

The average value of the grayscale block may be increased or decreased based on the luminosity or brightness level of the grayscale block. For example, a white grayscale block may have an average value of 0 and a black grayscale block may have an average value of 455. In this example, the average value of the grayscale block may use a smaller amount of data than the color block. The average value of the grayscale block can also retain a larger amount of lightness information for fingerprinting. For example, the brightness information for the grayscale block may be substantially similar to the brightness information for the color fingerprint of the frame 400 . Minor variations in average grayscale values for blocks with small block areas (eg, small number of pixels) can change the fingerprint and cause incorrect matching with the fingerprint stored in the reference database.

The average values associated with blocks 410 - 432 may be ranked or weighted. For example, blocks 410-432 may be ranked using an ordinal ranking. An ordinal ranking is a sequence of mean values, where each mean value may be an ordinal rank of a respective grayscale mean value. For example, the sequence of values may be 10, 20, 40, 30 with an ordinal rank of 1 2 4 3 . As another example, the sequence of values may be 12 28 43 33 with an ordinal rank of 1 2 4 3 .

As another example, the average values of the associated blocks 410 - 432 may be ranked in ascending or descending order. The ranks of the twelve grayscale mean s values associated with blocks 410-432 may be concatenated into a sequence, such as a hex string. The hexadecimal string may be a fingerprint value for frame 400 .

5A illustrates a top subdivided fingerprint layout of a frame 500 according to one embodiment. The top subdivided fingerprint layout may include a frame 500 subdivided into a plurality of blocks 510 - 526 . The top subdivided fingerprint layout may include an upper segment 528 with blocks 510 - 524 and a lower segment 530 with blocks 526 .

For example, upper segment 528 of frame 500 may be subdivided into a 2x4 grid having eight segments 510 - 524 . Block 526 may more likely include an on-screen display, such as a logo, emergency message, or caption. For example, when the electronic device 118 receives a request from the ACR system 124 to generate a fingerprint of the frame 500 using the top subdivided fingerprint layout, the electronic device 118 may block 526 from the fingerprint. can be removed or excluded.

For example, the plurality of blocks 510 - 526 may be rectangular or square. In another example, blocks 510-526 may be polygonal, circular, elliptical, or the like. In another example, each of the plurality of blocks 510 - 524 and 526 may be approximately the same size. In another example, one or more of the plurality of blocks 510 - 524 and 526 may be of different sizes. The number of the plurality of blocks 510-524 and 526 and the shape and size of the plurality of blocks 510-524 and 526 are not limited and may be changed.

5B shows a top subdivided fingerprint layout of frame 500 shown in FIG. 5A with blocks 532-554 according to one embodiment. Unless otherwise noted, some of the features of FIG. 5B are the same as or similar to some of the features of FIG. 5A when indicated by the same reference numerals. Blocks 532-554 may be partitioned vertically as compared to the lattice partitioning of blocks 510-524 of FIG. 5A. The top subdivided fingerprint layout of FIG. 5B is substantially the same as FIG. 5A in all other respects.

5C shows a bottom subdivided fingerprint layout for frame 500 as shown in FIG. 5A with blocks 510-524 according to one embodiment. Unless otherwise noted, some of the features of FIG. 5C are the same as or similar to some of the features of FIG. 5A when indicated by the same reference numerals. For a bottom subdivided fingerprint layout, blocks 510 - 524 may be located in the lower segment 530 compared to blocks 510 - 524 located on the upper segment 528 of FIG. 5A . The bottom subdivided fingerprint layout of FIG. 5C is substantially identical to FIG. 5A in all other respects.

5D shows a bottom subdivided fingerprint layout for frame 500 shown in FIG. 5C with blocks 532-554 according to one embodiment. Unless otherwise noted, some of the features of FIG. 5D are the same as or similar to some of the features of FIG. 5C when denoted by the same reference numerals. Blocks 532-554 may be partitioned vertically compared to the lattice partitioning of blocks 510-524 of FIG. 5C. Blocks 532-554 may be located on lower segment 530 compared to blocks 510-524 located on upper segment 528 of FIG. 5B . The bottom subdivided fingerprint layout of FIG. 5D is substantially the same as FIG. 5C in all other respects.

6 shows a flow diagram of a method 600 for identifying media content according to one embodiment. Method 600 may be at least in part by processing logic, including hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed by a processing device), firmware, or a combination thereof. can be performed with Method 600 may be performed by all or part of content distribution network 100 of FIG. 1 . For example, method 600 may be performed by all or part of the processing logic of electronic device 118 or 120 , ACR system 124 or 126 , or reference database 125 or 127 ( FIG. 1 ). Unless otherwise noted, some of the features of FIG. 6A are the same as or similar to some of the features of FIG. 1 when denoted by the same reference numerals.

Referring to FIG. 6A , method 600 begins with broadcasting media content (block 610 ) by content operator 102 or local operator 106 to ACR system 124 or electronic device 118 . do. The method includes enabling the ACR system 124 to receive media content (block 612). The method may include the ACR system 124 fingerprinting the media content to obtain a first set of fingerprints (block 614 ). The ACR system 124 may generate a first fingerprint using the fully subdivided fingerprint layout, generate a second fingerprint using the upper subdivided fingerprint layout, and generate a third fingerprint using the upper subdivided fingerprint layout. The first fingerprint set may include a first fingerprint, a second fingerprint, and a third fingerprint.

The method may include the ACR system 124 storing the first set of fingerprints and metadata associated with the media content in the reference database 125 (block 616 ). The method may include the electronic device 118 receiving the media content (block 618 ). The method may include the electronic device 118 fingerprinting the media content to obtain a second set of fingerprints (block 620). The electronic device 118 may generate a fourth fingerprint using the fully subdivided fingerprint layout, generate a fifth fingerprint using the upper subdivided fingerprint layout, and generate a sixth fingerprint using the upper subdivided fingerprint layout. The second fingerprint set may include a fourth fingerprint, a fifth fingerprint, and a sixth fingerprint.

The method may include the electronic device 118 sending the second set of fingerprints and metadata associated with the media content to the ACR system 124 (block 622 ). The metadata may include a first fingerprint set, a second fingerprint set, or schedule data associated with media content. The schedule data may include a schedule of airing times of media content or advertisements. As another example, the schedule data may include data obtained from a third-party server or database. As another example, the schedule data may include data obtained directly from a broadcasting system through an internal scheduling system. In another embodiment, the metadata is an event that occurs on the electronic device 118 , such as when the electronic device 118 receives a command from an input device, such as pausing media content, fast forwarding media content, rewinding media content, etc. may include one.

The method includes the ACR system 124 identifying media content displayed on the electronic device 118 using a subdivided fingerprint layout, a top subdivided fingerprint layout, a bottom subdivided fingerprint layout, or a combination thereof, as described above. step (block 624). In one embodiment, the ACR system 124 may compare blocks 510 - 524 of the second fingerprint using the top subdivided fingerprint layout to all top subdivided fingerprint layout fingerprints in the reference database 125 . When blocks 510-524 of the fifth fingerprint match corresponding blocks 510-524 of the second fingerprint, the ACR system 124 may identify the media content associated with the second fingerprint. In another embodiment, the ACR system 124 may similarly use the subdivided fingerprint layout to compare blocks 410 - 432 in the first fingerprint with all subdivided fingerprint layout fingerprints in the reference database 125 . When blocks 410-432 of the fourth fingerprint match corresponding blocks 510-524 of the first fingerprint, the ACR system 124 may identify media content associated with the first fingerprint. In yet another embodiment, the ACR system 124 may similarly use the lower subdivided fingerprint layout to compare blocks 510 - 524 of the third fingerprint to all lower subdivided fingerprint layout fingerprints in the reference database 125 . When blocks 510-524 of the sixth fingerprint match corresponding blocks 510-524 of the third fingerprint, the ACR system 124 may identify the media content associated with the third fingerprint.

In one embodiment, the ACR system 124 may sequentially query each type of fingerprint layout from the reference database 125 when searching for a fingerprint match. For example, the ACR system 124 may perform a first query on all fingerprints in the reference database 125 having a subdivided fingerprint layout to determine whether any subdivided fingerprint layout fingerprint can match a fourth fingerprint. can When the ACR system 124 identifies a fingerprint match, the ACR system 124 may stop querying and identify the media content associated with the matching fingerprint. If the ACR system 124 does not identify a fingerprint match, the ACR system 124 performs a second query for all fingerprints in the reference database 125 with the top subdivided fingerprint layout to perform a second query for any top subdivided fingerprint layout fingerprint. It may be determined whether the fifth fingerprint is matched. If the ACR system 124 identifies a fingerprint match, the ACR system 124 may stop the query and identify the media content associated with the matching fingerprint. If the ACR system 124 does not identify a fingerprint match, the ACR system 124 performs a third query for all fingerprints in the reference database 125 with the bottom subdivided fingerprint layout fingerprint to any bottom subdivided fingerprint layout fingerprint. It may be determined whether the sixth fingerprint is matched. When the ACR system 124 identifies a fingerprint match, the ACR system 124 may stop the query and identify the media content associated with the matching fingerprint. The fingerprint layouts used in the first query, the second query, and the third query are not limited, and the fingerprint layouts may be used in any order.

In another embodiment, the ACR system 124 may perform parallel queries for each type of fingerprint layout in the reference database 125 when searching for a fingerprint match. For example, the ACR system 124 may simultaneously perform the first query, the second query, and the third query. Once the ACR system 124 identifies a fingerprint match for the first query, the second query, or the third query, the ACR system may stop the query and identify the media content associated with the matching fingerprint.

The method may include the ACR system 124 identifying overlay content to overlay onto the media content (block 626 ). For example, the overlay content may be stored in a reference database 125 or in a memory device coupled to the ACR system 124 . The overlay content may be correlated with the identified media content. The method may include transmitting the overlay content to the electronic device 118 (block 628 ). The method may include the electronic device 118 superimposing the overlay content over at least one segment of the media content ( 630 ).

Use of the ACR system 124 , reference database 125 , and electronic device 118 of FIG. 6A is not limiting and other components (eg, components of FIG. 1 ) may perform one or more steps of FIG. 6 . can be implemented For example, the ACR system 124 may be replaced with the ACR system 126 , the electronic device 118 may be replaced with the electronic device 120 , and the reference database 125 may be replaced with the reference database 127 . can be

6B is a flow diagram of a method 640 for identifying media content using a top subdivided fingerprint layout according to one embodiment. Method 640 may be at least in part by processing logic, including hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed by a processing device), firmware, or a combination thereof. can be performed with Method 640 may be performed by all or part of content distribution network 100 of FIG. 1 . For example, method 640 may be performed by all or part of the processing logic of electronic device 118 or 120 , ACR system 124 or 126 , or reference database 125 or 127 ( FIG. 1 ). Unless otherwise noted, some of the features of FIG. 6B are the same as or similar to some of the features of FIG. 1 when denoted by the same reference numerals.

Referring to FIG. 6B , method 640 begins with ACR system 124 receiving a first fingerprint in a top subdivided fingerprint layout (block 642 ). The method includes the ACR system 124 querying the reference database 125 for a second fingerprint that matches the first fingerprint, wherein the second fingerprint has a top subdivided layout (block 644). The method may also include the ACR system 124 identifying the overlay content correlated to the second fingerprint (block 646 ). For example, the reference database 125 may store a plurality of fingerprints. The fingerprint may be correlated with different overlay content. The overlay content may be stored in a reference database 125 or another memory device. The method may include the ACR system 124 transmitting the overlay content to the electronic device 118 (block 648 ). The format of the first fingerprint and the second fingerprint is not limited. The format of the first fingerprint and the second fingerprint may be a full subdivided fingerprint layout, a top subdivided fingerprint layout, a bottom subdivided fingerprint layout, or some other fingerprint layout.

7 shows a schematic representation of a machine in an exemplary form of a computer system 700 upon which a set of instructions may be executed to cause the machine to perform one or more of the methods described herein. In alternative embodiments, the machine may be connected (eg, networked) to another machine on a LAN, intranet, extranet, or Internet. A machine may operate as a server or electronic device in a client-server network environment or as a peer machine in a peer-to-peer (or distributed) network environment. A machine may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, smartphone, web appliance, server, network router, switch or bridge, or to be performed by the machine. It may be any machine capable of executing a set of instructions (sequentially or otherwise) specifying an operation. Also shown as a single machine, the term "machine" also includes any collection of machines that individually or jointly execute a set (or a plurality of sets) of instructions to perform any one or more of the methods herein. should also be considered

The computer system 700 may correspond to the ACR fingerprinting server 105 of FIG. 1 . Computer system 700 may correspond to at least one segment of a cloud-based computer system. Computer system 700 includes processing device 702 , main memory 704 (eg, read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (eg, synchronous DRAM (SDRAM) or DRAM (RDRAM)). ), static memory 706 (eg, flash memory, static random access memory (SRAM), etc.), and data storage device 718 , which communicate with each other via bus 730 .

Processing unit 702 represents one or more processing units, such as a microprocessor, central processing unit, and the like. More specifically, the processing device may include a complex instruction set computing (CISC) microprocessor, a simplified instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or any other processor or instruction set implementing the instruction set. It may be a processor that implements the combination. The processing device 702 may be one or more special purpose processing devices, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, and the like. In one embodiment, processing apparatus 702 may include one or more processing cores. The processing device 702 is configured to execute instructions 726 of mirroring logic stored in the main memory 704 to perform the operations described herein.

Computer system 700 may further include a network interface device 708 communicatively coupled to network 720 . Computer system 700 may also include a video display unit 710 (eg, a liquid crystal display (LCD) or cathode ray tube (CRT)), an alphanumeric input device 712 (eg, a keyboard), a cursor control device 714 (eg, , a mouse), a signal generating device 716 (eg, a speaker), or other peripheral device. Computer system 700 may also include a graphics processing unit 722 , a video processing unit 728 , and an audio processing unit 732 . In another embodiment, computer system 700 is a chipset (not shown), which refers to an integrated circuit or group of chips designed to work with processing device 702 and designed to control communications between processing device 702 and external devices. ) may be included. For example, the chipset connects the processing device 702 to a high-speed device, such as main memory 704 and a graphics controller, and connects the processing device 702 to the peripheral's low-speed peripheral bus, such as a USB, PCI or ISA bus. It can be a set of chips on the motherboard that connect to

Data storage device 718 may include computer-readable storage medium 725 having stored instructions 726 implementing any one or more of the methods of functionality described herein. Instructions 726 may also reside completely or at least partially within main memory 704 and/or processing unit 702 while being executed by computer system 700 , and may also reside in main memory 704 and processing unit 702 . ) may also be a computer-readable storage medium.

The computer-readable storage medium 725 may be used to store instructions 726 using a software library including methods for calling logic and/or applications. Although computer readable storage medium 725 is referred to as a single medium in the illustrative embodiments, the term "computer readable storage medium" or "computer readable medium" refers to a single medium or a plurality of media (eg, , centralized or distributed databases and/or associated caches and servers). "Computer-readable storage medium" also includes any medium that can store, encode, or transport a set of instructions that are executed by a machine to cause the machine to perform any one or more of the methods of the embodiments of the present invention. should be understood as The term "computer-readable storage medium" will be understood to include, but is not limited to, solid-state memory, optical and magnetic media.

From the above description, many details have been provided. However, it will be apparent to one of ordinary skill in the art that the embodiments may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring the description.

Although embodiments may be described with reference to specific integrated circuits, such as computing platforms or microprocessors, other embodiments may be applied to other types of integrated circuits and logic devices. Similar techniques and descriptions of the embodiments described herein may be applied to other types of circuits or semiconductor devices. For example, the disclosed embodiments are not limited to desktop computer systems or Ultrabooks™, but may also be used in other devices such as handheld devices, tablets, other thin notebooks, system-on-a-chip (SOC) devices, and embedded applications. have. Some examples of handheld devices include cellular phones, Internet Protocol devices, smartphones, digital cameras, personal digital assistants (PDAs), and handheld PCs. Embedded applications typically include microcontrollers, digital signal processors (DSPs), system-on-chips, network computers (NetPCs), set-top boxes, network hubs, wide area network (WAN) switches, or the functions and operations described below. any other system capable of performing.

Although embodiments have been described herein with reference to processors or processing devices, other embodiments may be applied to other types of integrated circuits and logic devices. Similar techniques and descriptions of embodiments of the present invention may be applied to other types of circuits or semiconductor devices that may benefit from higher pipeline throughput and improved performance. The description of embodiments of the present invention may apply to any processor or machine that performs data manipulation. However, the present invention is not limited to processors or machines performing 512-bit, 256-bit, 128-bit, 64-bit, 32-bit, and/or 16-bit data operations, but may be applied to any processor and machine in which manipulation or management of data is performed. can be applied. In addition, the following description provides examples and the following drawings show various examples for illustrative purposes. These examples, however, should not be construed as limiting as they are intended to provide embodiments of the invention rather than to provide an exhaustive list of all possible implementations of the invention.

Some portions of the detailed description are presented as algorithms and symbolic representations of operations on bits of data in a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art to most effectively convey the meaning of their work to another person skilled in the art. An algorithm is here and generally conceived as a coherent sequence of actions leading to a desired result. An operation is one that requires physical manipulation of a physical quantity. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, or otherwise manipulated. It has proven convenient at times, for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. The blocks described herein may be hardware, software, firmware, or a combination thereof.

It should be understood, however, that all of these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As is apparent from the preceding description, the terms "sensing", "initiating", "determining", "continue", "interrupt", "receive", "record", etc. are used throughout the description, unless specifically stated otherwise. The description used is computing that manipulates and transforms data expressed in physical (eg, electronic) quantities in registers and memories of the computing system into computing system memory or registers or other such information storage transfers or other data in electronic devices. It will be understood that reference to the operations and processes of a system or similar electronic computing device.

The word “example” or “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" is not necessarily to be construed as preferred or preferred over other aspects or designs. Rather, the words "example" or "example" are intended to present the invention in a clear manner. As used herein, the term “or” means an inclusive “or” rather than an exclusive “or.” That is, "X comprises A or B" is intended to include any natural inclusive permutation, unless otherwise specified or clear from context. That is, if X includes A, X includes B, or X includes both A and B, then “X includes A or B” is satisfied under any of the preceding instances. In addition, when the indefinite articles "a" and "an" are used in this application and in the claims, they should generally be construed to mean "one or more" unless otherwise specified in the singular form or clear from the context. In addition, the use of the terms "an embodiment" or "an embodiment" or "an embodiment" or "an embodiment" throughout does not mean the same embodiment or embodiment unless otherwise stated. Also, the terms "first", "second", "third", "fourth", etc., when used herein, are meant as labels for distinguishing different elements, and must have an ordinal meaning according to these numerical designations. it may not be

Embodiments described herein may also relate to apparatus for performing the operations herein. The apparatus may be specially constructed for the required purpose or it may comprise a computer selectively activated or reconfigured by a computer program stored on the computer. Such computer programs may include non-transitory computer readable storage media including, but not limited to, disks of any type, such as floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read- It may be stored in only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical card, flash memory, or any tangible medium suitable for storing electronic instructions. The term “computer-readable storage medium” should be understood to include a single medium or a plurality of media (eg, centralized or distributed databases and/or associated caches and servers) that store one or more sets of instructions. "Computer-readable medium" also includes any medium that can store, encode, or transport a set of instructions that is executed by a machine to cause the machine to perform any one or more of the methods of embodiments of the present invention. should be understood as The term "computer-readable storage medium" is capable of storing solid-state memory, optical media, magnetic media and a set of instructions that, when executed by a machine, cause the machine to perform any one or more of the methods of the embodiments of the present invention. It will be understood to include, but not be limited to, any medium in which

The algorithms and displays provided herein are not inherently related to any particular computer or other device. Various computing systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the operations. The required structure for a variety of these systems will be apparent from the description below. In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the description of the embodiments described herein.

The foregoing description provides examples of specific details, such as specific systems, components, methods, and the like, in order to provide a good understanding of some embodiments. However, one of ordinary skill in the art will recognize that at least some embodiments may be practiced without these specific details. In other instances, well-known components or methods have not been described in detail in order not to unnecessarily obscure the embodiments of the present invention and are presented in simple block diagram format. Accordingly, the specific details provided above are by way of example only. Certain embodiments may vary from these exemplary details and are still considered to be within the scope of the present invention.

It is to be understood that the foregoing description is for the purpose of explanation and not limitation. Many other embodiments will be apparent to those skilled in the art after reading and understanding this description. Accordingly, the scope of the present embodiment should be determined with reference to the following claims and the full scope of equivalents accompanying these claims.

Claims (19)

A computing system comprising:
one or more processors, and
a computer-readable medium storing instructions that, when executed by the one or more processors, cause the computing system to perform functions;
The functions are
the ability to access a first video frame comprising first pixel values each corresponding to first positions within the first video frame;
selecting second pixel values respectively corresponding to second locations within the first video frame based on a first fingerprint layout identifying second locations, wherein the first pixel values include the second pixel values and the second location. at least one pixel value not included in the two pixel values, wherein the first locations correspond to the second locations and (a) not included in the second locations and (b) the at least one pixel value contains at least one position where -
generating a first fingerprint file identifying the first video frame by processing the second pixel values;
determining that the first fingerprint file matches a second fingerprint file identifying a second video frame;
a function of storing data indicating that the first video frame matches the second video frame;
the function of selecting third pixel values respectively corresponding to third positions in the first video frame based on a second fingerprint layout identifying third positions, the first pixel values being the third pixel values and the second at least one additional pixel value not included in the three pixel values, wherein the first positions are at the third positions and (a) not included in the third positions and (b) at least one additional pixel value. contains at least one corresponding position —
generating a third fingerprint file identifying the first video frame by processing the third pixel values; and
a function of determining that the third fingerprint file does not match any fingerprint file stored in a database;
and generating the first fingerprint file comprises generating the first fingerprint file in response to determining that the third fingerprint file does not match any fingerprint file stored in a database. According to claim 1,
wherein the function of accessing the first video frame comprises the function of receiving the first video frame from a media device. 3. The method of claim 2,
The functions are
selecting the media content based on an association between the media content and the first video frame; and
and sending instructions to the media device to play the media content. According to claim 1,
wherein the second pixel values include brightness values, and the function of generating the first fingerprint file by processing the second pixel values comprises the function of generating the first fingerprint file by processing the brightness values. system. 5. The method of claim 4,
Processing the brightness values is:
determining whether a first brightness value of the brightness values is greater than or less than a second brightness value of the brightness values; and
and generating the first fingerprint file based on the determination. According to claim 1,
The functions further include generating the second fingerprint file prior to determining that the first fingerprint file matches the second fingerprint file. According to claim 1,
wherein the first video frame is a rectangular video frame comprising four corner regions, and wherein at least one position not included in the second positions comprises one or more of the four corner regions. According to claim 1,
wherein the first video frame is a rectangular video frame comprising (i) an upper region from a left edge of the rectangular video frame to a right edge of the rectangular video frame and (ii) a lower region from the left edge to the right edge; and at least one location not included in the second locations includes the sub-region. According to claim 1,
wherein the first video frame is a rectangular video frame comprising (i) an upper region from a left edge of the rectangular video frame to a right edge of the rectangular video frame and (ii) a lower region from the left edge to the right edge; and at least one location not included in the second locations comprises the upper region. A computing system comprising:
one or more processors, and
a computer-readable medium storing instructions that, when executed by the one or more processors, cause the computing system to perform functions;
The functions are
the ability to access a first video frame comprising first pixel values each corresponding to first positions within the first video frame;
selecting second pixel values each corresponding to second locations within the first video frame based on a fingerprint layout identifying second locations, wherein the first pixel values are the second pixel values and the second pixel at least one pixel value not included in the values, wherein the first locations are at least corresponding to the second locations and (a) not included in the second locations and (b) the at least one pixel value. contains one location ― ,
generating a first fingerprint file identifying the first video frame by processing the second pixel values;
determining that the first fingerprint file matches a second fingerprint file identifying a second video frame, and
a function of storing data indicating that the first video frame matches the second video frame;
the second pixel values include brightness values, and the function of generating the first fingerprint file by processing the second pixel values includes the function of generating the first fingerprint file by processing the brightness values;
Processing the brightness values is:
calculating a first average brightness value for the first group of brightness values and a second average brightness value for the second group of brightness values;
determining whether the first average brightness value is greater than or less than the second average brightness value; and
and generating the first fingerprint file based on the determination. 11. The method of claim 10,
wherein the first fingerprint file comprises a sequence of numbers indicating whether the first average brightness value is greater than or less than the second average brightness value. A computing system comprising:
one or more processors, and
a computer-readable medium storing instructions that, when executed by the one or more processors, cause the computing system to perform functions;
The functions are
the ability to access a first video frame comprising first pixel values each corresponding to first positions within the first video frame;
selecting second pixel values each corresponding to second locations within the first video frame based on a fingerprint layout identifying second locations, wherein the first pixel values are the second pixel values and the second pixel at least one pixel value not included in the values, wherein the first locations are at least corresponding to the second locations and (a) not included in the second locations and (b) the at least one pixel value. contains one location ― ,
generating a first fingerprint file identifying the first video frame by processing the second pixel values;
determining that the first fingerprint file matches a second fingerprint file identifying a second video frame, and
a function of storing data indicating that the first video frame matches the second video frame;
wherein the first video frame is a rectangular video frame comprising (i) an upper region from a left edge of the rectangular video frame to a right edge of the rectangular video frame and (ii) a lower region from the left edge to the right edge; At least one location not included in the second locations includes the lower region,
and the second pixel values correspond to portions of the first row of the upper region and portions of the second row of the upper region that are between the lower region and the first row, respectively. A computing system comprising:
one or more processors, and
a computer-readable medium storing instructions that, when executed by the one or more processors, cause the computing system to perform functions;
The functions are
the ability to access a first video frame comprising first pixel values each corresponding to first positions within the first video frame;
selecting second pixel values each corresponding to second locations within the first video frame based on a fingerprint layout identifying second locations, wherein the first pixel values are the second pixel values and the second pixel at least one pixel value not included in the values, wherein the first locations are at least corresponding to the second locations and (a) not included in the second locations and (b) the at least one pixel value. contains one location ― ,
generating a first fingerprint file identifying the first video frame by processing the second pixel values;
determining that the first fingerprint file matches a second fingerprint file identifying a second video frame, and
a function of storing data indicating that the first video frame matches the second video frame;
wherein the first video frame is a rectangular video frame comprising (i) an upper region from a left edge of the rectangular video frame to a right edge of the rectangular video frame and (ii) a lower region from the left edge to the right edge; At least one location not included in the second locations includes the lower region,
and the second pixel values each correspond to portions of a row of an upper region extending from an upper edge of the rectangular video frame to the lower region. A computing system comprising:
one or more processors, and
a computer-readable medium storing instructions that, when executed by the one or more processors, cause the computing system to perform functions;
The functions are
the ability to access a first video frame comprising first pixel values each corresponding to first positions within the first video frame;
selecting second pixel values each corresponding to second locations within the first video frame based on a fingerprint layout identifying second locations, wherein the first pixel values are the second pixel values and the second pixel at least one pixel value not included in the values, wherein the first locations are at least corresponding to the second locations and (a) not included in the second locations and (b) the at least one pixel value. contains one location ― ,
generating a first fingerprint file identifying the first video frame by processing the second pixel values;
determining that the first fingerprint file matches a second fingerprint file identifying a second video frame, and
a function of storing data indicating that the first video frame matches the second video frame;
wherein the first video frame is a rectangular video frame comprising (i) an upper region from a left edge of the rectangular video frame to a right edge of the rectangular video frame and (ii) a lower region from the left edge to the right edge; At least one location not included in the second locations includes an upper region,
and the second pixel values correspond to portions of a first row of the lower region and portions of a second row of the lower region between the upper region and the first row, respectively. A computing system comprising:
one or more processors, and
a computer-readable medium storing instructions that, when executed by the one or more processors, cause the computing system to perform functions;
The functions are
the ability to access a first video frame comprising first pixel values each corresponding to first positions within the first video frame;
selecting second pixel values each corresponding to second locations within the first video frame based on a fingerprint layout identifying second locations, wherein the first pixel values are the second pixel values and the second pixel at least one pixel value not included in the values, wherein the first locations are at least corresponding to the second locations and (a) not included in the second locations and (b) the at least one pixel value. contains one location ― ,
generating a first fingerprint file identifying the first video frame by processing the second pixel values;
determining that the first fingerprint file matches a second fingerprint file identifying a second video frame, and
a function of storing data indicating that the first video frame matches the second video frame;
wherein the first video frame is a rectangular video frame comprising (i) an upper region from a left edge of the rectangular video frame to a right edge of the rectangular video frame and (ii) a lower region from the left edge to the right edge; At least one location not included in the second locations includes the upper region,
and the second pixel values respectively correspond to portions of a row of the lower region extending from the lower edge of the rectangular video frame to the upper region. As a method,
accessing a first video frame comprising first pixel values each corresponding to first positions within the first video frame;
selecting second pixel values each corresponding to second locations within the first video frame based on a first fingerprint layout identifying second locations, wherein the first pixel values include the second pixel values and the second location. at least one pixel value not included in the two pixel values, wherein the first locations correspond to the second locations and (a) not included in the second locations and (b) the at least one pixel value contains at least one position where -
generating a first fingerprint file identifying the first video frame by processing the second pixel values;
determining that the first fingerprint file matches a second fingerprint file identifying a second video frame;
storing data indicating that the first video frame matches the second video frame;
selecting third pixel values each corresponding to third positions in the first video frame based on a second fingerprint layout identifying third positions, the first pixel values being the third pixel values and the second pixel values at least one additional pixel value not included in the three pixel values, wherein the first positions are not included in the third positions and (a) the third positions and (b) the at least one additional pixel value contains at least one position corresponding to − ,
generating a third fingerprint file identifying the first video frame by processing the third pixel values; and
determining that the third fingerprint file does not match any fingerprint file stored in a database;
wherein generating the first fingerprint file comprises generating the first fingerprint file in response to determining that the third fingerprint file does not match any fingerprint file stored in the database. 17. The method of claim 16,
wherein the second pixel values include brightness values, and generating the first fingerprint file by processing the second pixel values comprises generating the first fingerprint file by processing the brightness values. 18. The method of claim 17,
Processing the brightness values is:
determining whether a first brightness value of the brightness values is greater than or less than a second brightness value of the brightness values; and
and generating the first fingerprint file based on the determination. A non-transitory computer-readable medium storing instructions that, when executed by a computing system, cause the computing system to perform functions, comprising:
The functions are
the ability to access a first video frame comprising first pixel values each corresponding to first positions within the first video frame;
selecting second pixel values respectively corresponding to second locations within the first video frame based on a first fingerprint layout identifying second locations, wherein the first pixel values include the second pixel values and the second location. at least one pixel value not included in the two pixel values, wherein the first locations correspond to the second locations and (a) not included in the second locations and (b) the at least one pixel value contains at least one position where -
generating a first fingerprint file identifying the first video frame by processing the second pixel values;
determining that the first fingerprint file matches a second fingerprint file identifying a second video frame;
a function of storing data indicating that the first video frame matches the second video frame;
the function of selecting third pixel values respectively corresponding to third positions in the first video frame based on a second fingerprint layout identifying third positions, the first pixel values being the third pixel values and the second at least one additional pixel value not included in the three pixel values, wherein the first positions are not included in the third positions and (a) the third positions and (b) the at least one additional pixel value contains at least one position corresponding to − ,
generating a third fingerprint file identifying the first video frame by processing the third pixel values; and
a function of determining that the third fingerprint file does not match any fingerprint file stored in a database;
wherein the function of generating the first fingerprint file comprises generating the first fingerprint file in response to determining that the third fingerprint file does not match any fingerprint file stored in the database. computer readable media.

Images