CN113596525B

CN113596525B - System and method for estimating user attention

Info

Publication number: CN113596525B
Application number: CN202110678008.3A
Authority: CN
Inventors: 埃亚·奥伦; 法尔扎·罗哈尼
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2015-12-14
Filing date: 2016-11-04
Publication date: 2024-05-24
Anticipated expiration: 2036-11-04
Also published as: DE112016002225T5; GB2555545A; WO2017105667A1; EP3295415A1; GB201721018D0; JP2018531468A; CN107710261A; CN107710261B; US10405045B2; US11089371B2; US20170171620A1; JP6615339B2; CN113596525A; US20190387277A1

Abstract

The present disclosure relates to systems and methods for estimating user attention. The present disclosure provides systems and methods for estimating audience member engagement with content or distinguishing users consuming content from users that have been distracted or left. User presence or attention may be estimated based on user interactions with content or skipping content, where user interactions are compared to high quality engagement data from a small audience measurement assessment team, or extrapolated based on a time engagement curve. An attention gap may be estimated that represents a user that is not present for, or does not participate in or notice the presentation of content at the client device. This enables the measurement system to distinguish users consuming and likely to like the content from users not consuming or likely to dislike the content, even when both sets of users' client devices receive and present the content item.

Description

System and method for estimating user attention

Description of the division

The application belongs to a divisional application of Chinese patent application 201680034538.1 with the application date of 2016, 11 and 4.

Background

With unicast internet delivery, instances of content presentation can be accurately measured, and the content delivery server accurately reports the number of client devices delivering any content item. While it may be readily assumed that each delivery instance results in the user focusing on or actively consuming the content, this may not always be the case. The user's attention may be drawn to elsewhere so that the user cannot actively consume the content, or the user may leave the immediate vicinity without pausing or stopping the media stream, resulting in the presentation of the content to an empty room.

In addition, a content selection system for personalizing or customizing content delivery may use presentation statistics to determine which users or demographics are interested in what types of content. Assuming that all users are engaged in or interested in content delivered to their client devices may exaggerate the actual interest rate of some items, resulting in inaccurate determination of user attention.

Disclosure of Invention

User presence or attention may be estimated based on user interactions with content or skipping content, where user interactions are compared to high quality engagement data from a small audience measurement assessment team or extrapolated based on a time engagement curve. An attention gap may be estimated that represents a user that is not present for or does not participate in or pay attention to the content presentation at the client device. This enables the measurement system to distinguish users consuming and likely to like content from users not consuming or likely to dislike content, even when both groups of users' client devices receive and present content items.

In one aspect, an audience measurement system may use content with a known interest rate that can be determined with high quality via a small evaluation team or survey. The same content may then be presented to an average viewer, with the viewer engagement measured via interactions with the content (e.g., click through, skip, survey selection, etc.). For example, using the skip measured viewer interaction rate may directly identify the percentage of users actively participating in the content but not interested in the content (e.g., users explicitly skipping the content); but users who consume and enjoy the content without skipping the content may be excessively represented by including users that do not exist or participate in the content in the number. Note that the gap may be identified as the difference between the known interest rate and the measured interaction rate. In the assumption that if all users are engaged in the content, the viewer interaction rate will be similar to that of a small high quality assessment team, and thus, the attention gap represents a portion of the users that are not engaged in the content or are not present for the presentation of the content. Thus, the percentage of participating or interested users may be determined and the content provider charged according to the adjusted active view or selection algorithm. The same attention gap may be used to estimate the engagement of previous or subsequent content presentations.

In another aspect, the audience measurement system may utilize a time participation curve based on similar or identical content of different durations. This enables engagement or attention measurements to be made without the need to use an assessment panel to determine the baseline. It may be assumed that if the content item has an infinite duration, each user focusing on the content presentation will eventually choose to skip or terminate the presentation, and thus, for 100% audience engagement or interest, there will eventually be a corresponding 100% skip rate. Thus, any difference between a theoretical 100% skip rate and a skip rate that can be extrapolated from the skip rate of shorter duration content represents a gap in attention, or a percentage of users that are not engaged or present. In one such embodiment, the viewer may be presented with similar or identical content of different lengths (such as 30 seconds, 1 minute, 2 minutes, etc.) and measured skip rates. The best fit curve may be determined for similar content of infinite duration and extrapolated so that the attention gap may be calculated. This gap can then be applied to earlier measurements to estimate the percentage of participating, interested users of the respective content presentation.

These techniques may be used alone or in combination, and any type of content (e.g., streaming audio or video, television programming, pre-content, post-content, or interstitial content, etc.) and any type of interaction (skip, rewind, click-through, vote, social media "like", share, etc.) may be used.

These embodiments are not mentioned to limit or define the scope of the disclosure, but to aid in understanding the disclosure. Particular embodiments may be developed to realize one or more of the following advantages.

Drawings

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the present disclosure will become apparent from the description, the drawings, and the claims, in which:

FIG. 1A is a diagram of an embodiment of a media player that allows user interaction;

FIG. 1B is a diagram illustrating an example of interactions with content over time according to one embodiment;

FIG. 2A is a block diagram of a content delivery and engagement measurement environment, according to one embodiment;

FIG. 2B is a block diagram of the client and server devices shown in FIG. 2, according to one embodiment; and

FIG. 3A is a diagram illustrating an example of measurements of an evaluation team interacting with content, according to one embodiment;

FIGS. 3B and 3C are diagrams illustrating estimating audience engagement by using the gap between measured content interactions and expected content interactions, according to some embodiments;

FIG. 4 is a flow chart of an embodiment of a method for estimating user attention;

FIGS. 5A and 5B are diagrams illustrating estimating audience engagement by using time engagement extrapolation, according to one embodiment; and

FIG. 6 is a flowchart of an embodiment of a method for estimating audience engagement using time engagement extrapolation, according to one embodiment.

Like reference numbers and designations in the various drawings indicate like elements.

Detailed Description

Media or other content may be provided to a client device, including: a desktop computer, a laptop computer, a tablet or smart phone, a video game console, a smart television, a wearable computer, or any other type or form of client device capable of presenting content to a user. The content may be of any type and form, including: text, audio, video, interactive applications, games, or other media. In many implementations, content may be created by a publisher, commonly referred to as a content provider; and may be delivered to client devices through a content distribution system having an infrastructure and bandwidth for delivering content to thousands or millions of client devices simultaneously.

In many implementations, the content distribution system may bill the content provider based on the number of recipients or the size of the audience, which is typically measured by the number of client devices delivering the content. With streaming media or similar content delivered in response to a request (e.g., a hypertext transfer protocol (HTTP) GET request, a real-time streaming protocol (RTSP) PLAY request, a real-time transport protocol (RTP) request, or other types and formats of requests), in some embodiments, the number of requests (or responses) may be measured directly by the content distribution system. In other systems, such as those utilizing multicast protocols or broadcast systems, in some embodiments, a device, such as a cable demodulator or decoder or set-top box, may transmit an identification of the channel tuning or reception of the content to the content distributor.

The content provider may wish to determine the size of the audience in terms of content presentation or viewing. However, while content may be delivered to and presented by the client device, the user may or may not actually be present or may be distracted by other content or events. For example, many implementations of streaming or multimedia broadcasting include additional pre-content, post-content, or interstitial content. The audience member may view other devices, leave the room (e.g., hold a snack), or be inattentive during presentation of the additional content. Thus, while some devices may receive and present content, it is assumed that each such presentation represents a possible incorrect browsing by the audience member.

In addition, content providers, producers, and publishers often make production decisions based on popularity of content to viewers or related demographics, such as creating content similar to popular content, or canceling presentation of undesirable content. Similarly, a content selection system for personalizing or customizing content delivery may use presentation statistics to determine which users or demographics are interested in what types of content. The actual interest rate of some items may be exaggerated by the wrong assumption that all users are engaged in or interested in all content delivered to their client devices.

In some implementations, content may be presented by a client device with interactive controls. For example, FIG. 1A is an illustration of an embodiment of a media player 100 that allows user interaction. The media player 100 may be a stand-alone software application, an embedded player within a web page rendered by a web browser, a display environment of a smart television, or any other type and form of interface for presenting content to a user and allowing the user to interact with the content or content presentation in at least a limited manner.

The media player 100 may display content 102, such as multimedia content, audio content, text content, or any other type and form of content. In many implementations, the content 102 may have a temporal aspect or duration, such as audio or video content. Examples of such content include live or pre-recorded television programs, movies, music, radio programs, podcasts, or any other such content. In many implementations, the media player 100 can represent a temporal aspect of the content 102 with a timeline 104 or similar interface to display the play time 106 and/or allow interaction with the play (e.g., rewind, fast forward, pause, initiate play at a particular time, etc.).

Different or additional content may be presented sequentially or at predetermined times during longer content, particularly for broadcast or streaming content, such as pre-content before another program, post-content after a program, or interstitial content during a program break.

In many implementations of the media player 100, a user may have the ability to terminate or skip a portion of content or interact with or control the presentation of content. For example, pre-recorded content typically has a predetermined duration or maximum length 108, such as 30 seconds. In some implementations, the user may view the complete pre-recorded content, or may select to skip a portion of the content by selecting a "skip" button 110 or similar interface element. While skip button 110 is shown on one corner of the displayed media content 102, in many embodiments skip button 110 may be placed elsewhere within media player 100 or separately from the media player (e.g., on a remote control). Selection of skip button 110 may cause media player 100 to terminate playback of the current content and request the next item of content or begin playing the buffered next item of content. In other implementations, a "dislike" or "irrelevant" button may be used to indicate that the user is not interested in the content being played, and the media player 100 may provide an identification of the preference to the content selection system for adjusting the personalized profile or to the content provider. In some such implementations, the media player may continue to present the media after selecting the "dislike" or "irrelevant" interface element.

While "skip," "dislike," "irrelevant," or similar negative interface elements may indicate that the user is not interested in the content, in other embodiments, the media player 100 may include positive interaction controls to allow the user to explicitly indicate that the user is interested in the content. For example, the media player 100 may include a control that triggers a function related to the content, such as requesting a website.

Selection of a negative or positive interface element may not only indicate an explicit preference of the user, but the selection also implicitly identifies the user as present and consuming or participating in the content. However, in many instances, the user may have a positive preference for content, consume the content, or be attracted to the content, but may not select the corresponding interface element. For example, some media players 100 may not include positive interface elements so that a user may only skip media (indicating a negative preference) or view media in its entirety. The latter behavior may indicate that the user has a positive preference for media, or may indicate that the user leaves the room or detaches the user from the presentation without stopping playing.

FIG. 1B is a diagram illustrating an example of a negative interaction 120 (e.g., selection of "skip," "dislike," "irrelevant," or similar interface elements) with rendered content over time 125, according to one embodiment. In many cases, the user subset 122 may select negative interface elements as soon as possible (e.g., to avoid viewing content), and thus, the interaction or "skip" rate may initially rise rapidly. Another subset 124 of users may view the content and decide at some point during the presentation of the content that they are not interested in the content and may select interface elements. During this time, the skip rate may similarly rise, although the skip rate at different rates depends on the level of interest of the content.

At some point during the playing of the content, when most users 122 and 124 have selected the interface element, a few users that are actively attracted to the content but are not interested in the content will remain or will not select the interface element. Thus, the skip rate may drop and eventually tend to settle (e.g., few users intending to press the skip button will wait until the last few seconds of the content item). However, the remaining subset 126 of users (e.g., the total viewer size of the received content or the number of client devices minus the sum of the subsets 122 and 124) represents users that are actively attracted to and interested in the content as well as users that are not engaged in the presentation of the content (e.g., are absent or distracted).

To distinguish users participating in and interested in content from users not participating in content, user presence or attention may be estimated via comparison of high quality participation data from a small audience measurement assessment panel, or extrapolated based on a time participation curve. An attention gap may be estimated that represents a user that is not present for or does not participate in or notice the presentation of content at the client device. This enables the measurement system to distinguish users consuming and likely to like content from users not consuming or likely to dislike content, even when both groups of users' client devices receive and present content items.

The following description of the various portions of this specification and its corresponding contents may be helpful in reading the following description of the various embodiments:

section a describes a network environment and computing environment for delivering content and estimating audience attention;

section B describes embodiments of systems and methods for estimating audience attention via comparison of engagement data from an assessment team;

section C describes embodiments of systems and methods for estimating viewer attention via time participation extrapolation;

A. network and computing environment

Fig. 2A is a block diagram of a content delivery and engagement measurement environment 200, according to one embodiment. Network 205 may connect one or more client devices 210A-210N (commonly referred to as client device 210) to a content distribution system or an audience measurement system (commonly referred to as audience measurement system 212). Audience measurement system 212 may receive content from one or more content providers 218 directly or via any of networks 205. The audience measurement system 212 may also be in communication with an evaluation group provider 214 that may perform single source evaluation group measurements or studies with one or more group members 216A-216N (commonly referred to as group members 216). Although only one measurement system or server 212 and one evaluation group provider 214 are illustrated, in many embodiments, multiple providers or servers may communicate via one or more networks 205.

Still referring to fig. 2A and in more detail, the network 205 may be any form of computer network or combination of networks that relay information between client devices 210, one or more audience measurement servers 212, and other devices not shown. The network 205 may include the internet and/or other types of data networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), a cellular network, a satellite network, or other types of data networks. Network 126 may also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and/or transmit data within network 205. The network 205 may further include any number of hardwired and/or wireless connections. The client device 210 may communicate wirelessly (e.g., via WiFi, cellular, radio, etc.) with transceivers that are hardwired (e.g., via fiber optic cable, CAT5 cable, etc.) to other computing devices in the network 205. In some implementations, the network 205 may be a virtual network, such as a virtual network between multiple virtual machines executed by a single physical machine, or an abstract network, such as transmitting data offline via a physically removable medium (e.g., SNEAKERNET, transmitting data via tape media, CD-ROM, flash media, external hard drive, floppy disk, etc.).

Client device 210 may be referred to in different ways as a client, a device, a client device, a computing device, a user device, or any other such terminology, and may be a desktop computer, a laptop computer, a tablet computer, a smart phone, a video game, a smart television or set-top box, a server, a workstation, or any other type and form of computing device capable of communicating over network 205. In some implementations, the client device 210 may execute an application, service, server, daemon (daemon), routine, or other executable logic for communicating over the network 205, such as a web browser, mail client, video player, music player, video game, or any other such application. Such applications may include command line interfaces, graphical user interfaces, or any combination of these or other interfaces. In embodiments where the client device is a smart television or set-top box, the client device may receive content via a first interface such as terrestrial, satellite, or cable broadcast; and may communicate with the audience measurement server via a second interface, such as an ethernet or WiFi interface, via the network 205. In other implementations, the client device 210 may receive content via the network 205 and may send the interaction identification via the network 205.

The content provider 218 may include one or more computing devices connected to the network 205 and configured to provide content to the client 210 directly or via either the audience measurement system or the content distribution system 212. The content provider 218 may be referred to in different ways as a content provider, a server, a web server, a data server, a publisher, a service provider, or by other similar terms. In many implementations, the content provider 218 may include a plurality of computing devices configured as a server farm or cloud, and may include routers, load balancers, network address translators, firewalls, or other such devices. The content provider 218 may be a computer server (e.g., FTP server, file sharing server, web server, etc.) or a combination of servers (e.g., data center, cloud computing platform, etc.). Content provider 218 may provide any type and form of content including text, images, video, audio, multimedia, or other data, or any combination thereof. The content may include search results, blog or forum content, news articles, movies, television programs, podcasts, video games, or other interactive content, websites, social networks, or any other type and form of content. Content provider 218 may be an online search engine that provides search result data to client device 210 in response to a search query. In another implementation, the content provider 218 may be a first party web server that provides web page data to the client device 210 in response to a request for a web page.

The audience measurement server or system 212 may include one or more computing devices connected to the network 205 and configured to measure and analyze audience data to determine audience size and participation rate. Audience measurement server 212 may be referred to in different ways as a content distribution system, a distribution and measurement system, a measurement server, a web server, a data server, a service provider, or by other similar terminology. In many implementations, the audience measurement server 212 may include a plurality of computing devices configured as a server farm or cloud, and may include routers, load balancers, network address translators, firewalls, or other such devices. In some implementations, the audience measurement server 212 may be a content provider 218.

In some implementations, the audience measurement server 212 may receive an identification of a request for content and/or a device identifier from the client 210 via the network 205. In one such embodiment, client 210 may execute a plug-in or other application to send an identification of a request for content to audience measurement server 212. In another embodiment, a request for content may be sent from the client 210 to the audience measurement server 212, which audience measurement server 212 may then redirect the request to the appropriate content provider 218 while recording information about the request. In yet another embodiment, a request for content may be sent from the client 210 to the content provider 218, which content provider 218 may respond with content and embedded instructions that cause the client 210 to send a second request to the audience measurement server 212. Content, such as web pages, delivered from a content provider may include embedded pixel-by-pixel images with addresses of the audience measurement server 212 such that when the client's web browser renders the content, the web browser sends a request for the image to the audience measurement server 212, possibly including parameters or cookies, device identifiers, or other information.

In other implementations, the audience measurement server 212 may receive an identification of a request for content and/or a device identifier from the content provider 218. Content provider 218 may execute a measurement proxy (not shown) that may include an application, service, server, daemon, or other executable logic for measuring requests from client devices that are received by content provider 218. The measurement proxy may send the identification of the request to the audience measurement system 212, either individually or in aggregate.

The evaluation group provider 214 may include one or more computing devices for conducting market or audience research with one or more evaluation group participants or members 216. The panelist who has agreed to participate in the panelist may indicate to the panelist that he or she has viewed or listened to a particular content item, such as a television program, a broadcast program, a movie, a commercial, an identified time period such as one minute, ten minutes, half hours, or any other such time period, or any other such information. In some implementations, the panelist can provide a diary or log, or wear or carry a portable device that detects content played nearby and logs for subsequent transmission to the panelist provider. The panel provider 214 may aggregate demographic information about each panel participant who watched or listened to the event and anonymize the information to create a viewer profile indicating characteristics such as specified gender, age, percentage of viewers from a particular location, or other such information. Audience measurement of content may be useful for measuring audience size or popularity of content for planning content delivery scheduling or other such purposes.

In some implementations, the panel provider 214 can aggregate and/or anonymize panel diaries or logs and send demographic information of the content box to the audience measurement server 212. In many implementations, the evaluation group provider 214 can send the identification of the corresponding broadcast block and/or information about the content block to the audience measurement server 212, such as a program type, a program title, a program description, keywords associated with the program, a website or other document associated with the program, a production and/or manufacturer associated with the program, or any other such information.

In some implementations, the evaluation panel provider 214 can gather preference information about the content items, such as positive or negative preferences of the content, from the panel members 216. In some implementations, this may be an explicit preference identification, while in other implementations, the panelist 216 may be allowed to select skipped content items, indicating a negative preference. In some such implementations, panelists may use the client device 210 and the media player 100 as discussed above.

Because the assessment team provider typically pays participation fees to the assessment team members, the team members 216 may be fully involved in the content. Thus, such audience measurement and preference data may be of high quality, albeit for small sample sizes.

In many embodiments, demographic information about the group members (and other users) may be anonymous or disambiguated to protect the privacy of the group members or device users. In many such implementations, the user may be provided with the following opportunities in similar situations where personal information about the user of the client device may be collected for measurements or may be used to select third party content: whether programs or features that control gathering personal information (e.g., information about the user's social network, social actions or activities, the user's preferences, or the user's current location) can gather personal information, or whether or how to send measurement data to an audience measurement server and/or an evaluation team provider. In addition, the particular data may be processed in one or more ways prior to being stored or used by the audience measurement server so that personal identity information may be removed when parameters (e.g., demographic parameters) are generated. In some implementations, the identity of the user may be anonymized such that personal identity information of the user cannot be determined, or where location information is obtained, the geographic location of the user may be generalized (such as to a city, zip code, or state county level) such that a particular location of the user cannot be determined. Thus, a user can control how information about the user is collected, and how the information is used by the audience measurement server, the panel provider, and the content provider.

Fig. 2B is a block diagram of the client and server devices shown in fig. 2A, according to one embodiment. Referring first to client device 210, the client device may be a computing device that is a client, an evaluation group participant, or a "regular" audience member that is not an evaluation group participant or content. Client device 210 may be any number of different types of consumer electronic devices configured to communicate via network 205, including, but not limited to, a laptop computer, a desktop computer, a tablet computer, a smart phone, a digital video recorder, a television set-top box, a video game console, or any other type and form of computing device or combination of devices. In some implementations, the type of client device 210 may be categorized as a mobile device, a desktop device, or a device that is intended to remain stationary or that is configured to access the network 205 primarily via a local area network, or another category of electronic devices, such as media consumption devices.

In many implementations, the client device 210 includes a processor 222 and a memory 224. Memory 224 may store machine instructions that, when executed by processor 222, cause processor 222 to perform one or more of the operations described herein. The processor 222 may include a microprocessor, ASIC, FPGA, or the like, or a combination thereof. In many implementations, the processor 222 may be a multi-core processor or an array of processors. Memory 224 may include, but is not limited to, electronic storage, optical storage, magnetic storage, or any other storage device capable of providing program instructions to processor 222. Memory 224 may comprise a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, EEPROM, EPROM, flash memory, optical media, or any other suitable memory from which processor 222 may read instructions. The instructions may include code from any suitable computer programming language such as, but not limited to, C, C ++, C#, java, javaScript, perl, HTML, XML, python, and Visual Basic.

The client device 210 may include one or more network interfaces 226. Network interface 226 may include any type and form of interface, including ethernet, including 10Base T, 100Base T, or 1000Base T ("gigabit"); any variant of 802.11 wireless, such as 802.11a, 802.11b, 802.11g, 802.11n, or 802.11ac; cells, including CDMA, LTE, 3G, or 4G cells; bluetooth or other short range wireless connection; or any combination of these or other interfaces for communicating with the network 205. In many embodiments, the client device 210 may include multiple network interfaces 226 of different types enabling connection to various networks 205, such as the internet, or networks 205 via different sub-networks. As discussed above, the client device 210 also includes other interfaces for receiving terrestrial, satellite, cable analog, or digital broadcasts.

The client device 210 may include one or more user interface devices 228. The user interface device 228 may be any electronic device (e.g., keyboard, mouse, pointing device, touch screen display, microphone, etc.) that communicates data to a user and/or converts received sensory information from a user into electronic signals by generating sensory information (e.g., visualization on a display, one or more sounds, haptic feedback, etc.). According to various embodiments, one or more user interface devices may be internal to the housing of the client device 210, such as a built-in display, touch screen, microphone, etc., or external to the housing of the client device 210, such as a monitor connected to the client device 210, a speaker connected to the client device 210, etc.

The client device 210 may include the application 230 in the memory 224 or may execute the application 230 with the processor 222. The application 230 may be an application, applet, script, service, daemon, routine, or other executable logic for receiving content and for sending responses, commands, or other data. In some implementations, the application 230 can be a web browser, and in another implementation, the application 230 can be a video game. The application 230 may include functionality for displaying content received via the network interface 226 and/or generated locally by the processor 222 and for transmitting interactions received via the user interface device 228, such as requests for websites, selections of search response options, entering text strings, and the like.

The application 230 may be or include an embedded media player 100, such as a plug-in within a web browser or a local media player. The application 230 may provide a user interface for interacting with content played in the media player, such as skip controls, dislike buttons, or any similar interface.

In some implementations, the application 230 can include a data collector or collection agent. The acquisition agent may include application plug-ins, application extensions, subroutines, browser toolbars, daemons, or other executable logic for acquiring data processed by the application 230. In some implementations, the application 230 may be a separate application, service, daemon, routine, or other executable logic separate from the application 230 but configured to intercept and/or collect data processed by the application 230, such as a screen grabber, packet interceptor, AOI hooking process, or other such application. The acquisition agent may be configured to intercept or receive data entered via the user interface device 228, such as internet search queries, text strings, survey response selections, or other values, or data received and processed by the application 230, including accessed websites, time spent interacting with websites or applications, pages read, or other such data. In many implementations, the collection may store some or all of these data, or identifiers of these data, in a behavioral history database or other data structure, and may include an identification of the website visited, the network link followed, the search query entered, or other such data. In some implementations, the data may be anonymous or disambiguated to reduce personal identity information. In other embodiments, the acquisition agent may be executed by a server, or may be executed by an intermediary device, such as a router, cable modem, or other such device, disposed between the client and the server. The data requests and responses may be parsed by an acquisition agent executing on the intermediate router as the requests and responses traverse the router. In some implementations, this may allow monitoring of all data streams to/from the home without the need to install an acquisition agent on multiple devices within the home. In other implementations, as discussed above, the client device 210 may not execute the acquisition agent; in such embodiments, the request data may be obtained by the audience measurement server without the use of an acquisition proxy, such as via rendering an embedded image in the content, redirecting the request, or other such method.

The client 210 may include a device identifier 234 or be identified with the device identifier 234. The device identifier 234 may be an alphanumeric string, a data string, a serial number, a Media Access Control (MAC) address, an Internet Protocol (IP) address, a user name or account name, a Globally Unique Identifier (GUID), a cookie, a random or pseudo-random number, or any other type and form of identifier, including combinations of these or other identifiers. In some implementations, the device identifier 234 may be fixed to the device or preconfigured in the device, such as a manufacturer serial number or MAC address, while in other implementations, the device identifier 234 may be dynamically set by the content provider, the evaluation team provider, the audience measurement server, the application 230, or other entity, such as a cookie or a user name. In some implementations, a unique or new device identifier 234 may be provided for each communication of the content provider and/or the audience measurement server, while in other implementations, the device identifier 234 may not change, or may change periodically (e.g., hourly, daily, weekly, etc.), or at other intervals (e.g., upon restarting the client device, logging into an internet service, etc.). In some implementations, the device identifier 234 can be associated with one or more other device identifiers 234 (e.g., a device identifier of a mobile device, a device identifier of a home computer, etc.). In many implementations, the device identifier 234 may be generated by the content provider and/or transmitted to the device 210, as discussed above. In other implementations, as discussed above, the client 210 may request a device identifier or cookie 234 from an audience measurement server or content provider and may send the device identifier or cookie 234 to the audience measurement server provider or content provider in association with the request for content.

A block diagram of an embodiment of the audience measurement server 212 or content provider 218, broadcast provider, or similar device is also illustrated in fig. 2B. As with the client device 210, the server 212 may include one or more processors 222, memory 224, a network interface 226, and a user interface 228. In some implementations, referred to as headless servers, the server 218 may not include the user interface 228, but may communicate with the client 210 having the user interface 228 via the network 205. Memory 224 may include content storage such as storage for web pages, images, audio files, video files, data files, or any other type and form of data. In some implementations, the memory 224 may store one or more applications (not shown) for execution by the processor 222 of a server, including an FTP server, a web server, a mail server, a file sharing server, a peer-to-peer server, or other such applications for delivering content stored in a content store.

In some implementations, the server 212 may execute the measurement engine 250. The measurement engine 250 may include an application, service, server, daemon, routine, or other executable logic for measuring audience for content items, including receiving device identifiers and/or requests for content items or information about such requests, aggregating or categorizing content identifiers according to device identifiers, and measuring audience for content items during a time period. In one embodiment, the measurement engine may count the number of content identifiers identifying particular content items received within a time period (such as one hour) associated with different device identifiers to count the size of the audience receiving the content items. The audience measurement may be provided to a third party, such as a content provider.

In some implementations, the server 212 can maintain a measurement database 244. Measurement database 244 may include any type and form of database, flat file, data array, or other data structure for storing a plurality of content identifiers with corresponding device identifiers, cookies, and/or session identifiers. In many embodiments, measurement database 244 may also include a timestamp of the received or transmitted content identifier. The measurement database 244 may also include demographic information or characteristics received from the client device in connection with a request for content or previously received from the client device and associated with a device identifier. The characteristics may be obtained explicitly via investigation or profile questions, or may be obtained implicitly via a request associated with the device that is similar to a request associated with a device having known characteristics. A characteristic or feature may be identified by a value, such as where the characteristic may have several different potential values. In some implementations, characteristics associated with broadcast events or blocks from different devices may be aggregated and/or anonymized.

The server 212 may also maintain a content store 246 that may store any type and form of content including audio or video content as discussed above. In some implementations, the content store 246 may be in one or more external storage devices, or may be distributed among one or more servers 212 or cloud storage devices. Content from content store 246 may be provided to client device 210, evaluation group provider 214, or group member 216.

The server 212 may execute a comparator or comparison engine 252. Comparator 252 may include an application, service, server, daemon, routine, or other executable logic for comparing measured data such as measured skip rate of content provided to a viewer, skip rate from panelist measurements, or other such information. Comparator 252 may include a bit-wise comparator for comparing integer values or bit strings, a mathematical comparator, an analog comparator, or any other type and form of application or software for comparing values to each other and/or to a threshold value.

Server 212 may also execute a correlator or correlation engine 256, which correlator or correlation engine 256 may include applications, services, routines, daemons, or other executable logic for determining correlations between various information including multivariate demographics. The correlator 256 may use any type and form of algorithm to determine the correlation between the presence of a characteristic in the aggregate device measurement data and the statistical value of the presence of a characteristic in the demographic data. The correlation engine 256 may use a Pearson correlation algorithm to compare the frequency of the characteristics in the data with the frequencies of the characteristics in other data. The generated correlation coefficient may be compared to a threshold and a confidence score associated with the characteristic may be increased or decreased in response to the coefficient being above or below the threshold.

Server 212 may also execute an extrapolator 258, which extrapolator 258 may include applications, services, routines, daemons, or other executable logic for extrapolating predicted values of given input parameters from a dataset. Extrapolation 258 can use any type and form of algorithm to determine extrapolation, including determining a best fit curve, calculating limits for a given function, or any other such function.

A block diagram of an embodiment of a server of the panelist provider 214 is also illustrated in fig. 2B. As with the client device 210, the server 214 may include one or more processors 222, memory or storage 224, a network interface 226, and a user interface 228. In some implementations, referred to as headless servers, the server 214 may not include the user interface 228, but may communicate with clients 210 or team members 210 having the user interface 228 via the network 205. Memory 224 may include content storage such as storage for web pages, images, audio files, video files, data files, or any other type and form of data. In some implementations, the memory 224 may store one or more applications (not shown) for execution by the processor 222 of a server, including an FTP server, a web server, a mail server, a file sharing server, a peer-to-peer server, or other such applications for delivering content stored in a content store.

In some implementations, the evaluation group provider 214 can maintain a database 240 of evaluation group information. Database 240 may be any type and form of database and may include information about panelists including demographic information, content preferences, or any other such information. As discussed above, in many embodiments, the panelist information may be obfuscated, the panelist may be encrypted, the panelist may be anonymous, or the panelist information may be obscured to preserve the privacy of the panelist.

The evaluation team provider 214 may store test content 242, which test content 242 may include any type and form of content presented to the team member 216. In some implementations, the test content 242 can be a subset of the content from the content store 246. For example, as discussed in more detail below, the test content 242 may be content that is widely preferred or disliked by panelists or members of the audience as opposed to content that may become more contradictory by the members of the audience. Evaluation team provider 214 may store information regarding the test content, including skip rate, preference rate, or other such information. In many implementations, rather than storing test content 242, evaluation group provider 214 may store only an identification of the test content (e.g., in a database) and may direct a client or group member to content store 246 maintained by server 212.

B. engagement estimation via evaluation team measurements

In one aspect, the audience measurement system may use content with a known interest rate that may be determined at high quality via a small evaluation team or survey. As discussed above, because participation fees may be paid to the panelist, the panelist's participation in the content may be equal to 100% (the panelist likes or dislikes the content, but nonetheless actively consumes the content, rather than being distracted or absent during presentation). Preferences may be determined explicitly (e.g., a "like" or "dislike" interface element or such control thereof) or implicitly (e.g., by enabling a team member to skip or terminate presentation of its dislike content).

FIG. 3A is a diagram illustrating an example of a measurement of an evaluation team's interactions with content ("dislike" or "skip" rate 300; or conversely, "like" or "watch completely" rate 302), according to one embodiment. The test content may be referred to as low quality content 304 that may be disliked by a majority of panelists or high quality content that may be liked by a majority of panelists. Content that is contradictory to panelists (or that is similar in number to the panelists who like and dislike the content) may be discarded because the content is too ambiguous for the engagement test to be performed. The content may be explicitly designed to be of high quality or low quality (e.g., 30 seconds of siren and/or flashing) or may be selected in response to a greater skip rating of the content (e.g., low quality content) or a greater skip-free rating of the content (e.g., high quality content).

As shown, the panelist may skip the low quality content 304 at a very high level n 306 (equivalent to a very low level of viewing rate). In some cases, n may be equal to 100% for truly good content; however, often at least a few panelists may dislike the content or skip the content, resulting in less than 100% n 306. Similarly, panelists may skip high quality content 310 at a very low rate m 312 (equivalent to a very high viewing rate). Also, while it is highly likely that no panelist will skip some very high quality content, in many cases at least a few panelists will skip content, resulting in a low but non-zero rate m 312. The low quality and high quality content may be classified based on a threshold value, which may be set to be sufficient to avoid blurring or measurement errors. In various embodiments, the low quality content may have a skip rate of greater than 70%, 80%, or 90%, or any other such value; and the high quality content may have a skip rate of less than 30%, 20%, or 10% or any other such value. More extrema may lead to improved measurements.

The same test content may then be presented to an average viewer, with the viewer engagement measured via interactions with the content (e.g., click through, skip, survey selection, etc.). As discussed above, using the skip measured viewer interaction rate may directly identify the percentage of users that are actively participating in the content but are not interested in the content (e.g., users that explicitly skip the content); but users who consume and enjoy the content without skipping the content may be excessively represented by including users that do not exist or participate in the content in the number. Note that the gap may be identified as the difference between the known interest rate and the measured interaction rate. In the assumption that if all users are engaged in the content, the viewer interaction rate will be similar to that of the team members, and thus, the attention gap indicates that the users are not engaged in the content or are not part of the users present for the presentation of the content. Thus, the percentage of participating or interested users may be determined and the content provider charged according to the adjusted active view or selection algorithm. The same attention gap may be used to estimate the engagement of previous or subsequent content presentations.

Fig. 3B is a diagram illustrating estimating audience engagement by using the gap between measured content interactions and expected content interactions for low quality content 304, according to some embodiments. The expected skip rate n 306 may be determined based on the interactions of the fully engaged team members with the content as discussed above in connection with fig. 3A. The same content may be provided to an average viewer with the ability to dislike the content or skip the content. Since some common viewers are likely to be out of mind (e.g., distracted or not present) during the presentation of the content, a smaller percentage of the common viewers will definitely dislike or skip the content, resulting in a measured skip rate i 324 that is less than the intended skip rate n (or conversely, a viewing rate that is greater than the intended viewing rate). The difference between the expected rate n and the measured rate m represents the attention gap 326, or the percentage of users that have taken off the body and are likely to have skipped the content if already present.

Other content, referred to as subject matter 330, may be provided to the same viewer-client device either before or after the test content 304 (e.g., immediately before or after or during a short time frame in which any user, such as a commercial, that is likely to similarly be off from the subject matter 330). The skip rate j 328 (or vice versa) may be measured for the viewer client device. Attention gap 326 may be added to the skip rate (or the attention gap may be subtracted from the viewing rate) to calculate a skip rate of interest k 330 (or a viewing rate of interest 332). The skip rate or viewing rate of interest indicates the percentage of audience members that would skip content or view content entirely if such dislocated or non-existent users were present to be able to select to skip content.

Fig. 3C is another diagram illustrating estimating audience engagement by using the gap between measured content interactions and expected content interactions of high quality content 310, according to some embodiments. Similar to the graph shown in fig. 3B, the measured skip rate i 324 may be lower than the expected skip rate m 312, with the difference equal to the attention gap 326. The attention gap 326 may be applied as discussed above to determine the skip rate of interest or the viewing rate of interest for other subject matter.

Fig. 4 is a flow chart of an embodiment of a method 400 for estimating user attention. Dividing the method into two phases, evaluating training 401 of the panel or training of the attention gap model based on content-based panel member measurements; and application of audience measurement 402 or models. Although training 401 is shown as being performed prior to measurement 402, in some embodiments training 401 may be performed after an earlier audience measurement and attention estimates may be made to stored data from the earlier audience measurement.

During the training of the model team members, in step 404, content items may be shown or provided to the team members. In some implementations, the content may be provided to the evaluation team as a group (e.g., in a group setting), while in some implementations, the content may be provided to individual client devices of members of the evaluation team. In some implementations, the content may be provided via a media player as discussed above in connection with fig. 1A. Each evaluation team member may consume the content and indicate a positive or negative preference for the content, or may view the content (indicating a positive preference) or choose to skip or terminate the content presentation (indicating a negative preference). As discussed above, each interaction may trigger a media player, acquisition agent, or similar application to send a request or identifier to the evaluation team provider for measurement.

In step 406, the request or identifier may be received by an evaluation group provider. Identifiers of the preferences of the team members may be stored, such as in an evaluation team information database. In some implementations, the skip rate or the viewing rate of the content items can be updated based on the reference preferences. In one such embodiment, for a skip rate equal to the number of received skip interactions divided by the number of team members, the number of team members may be incremented and the number of skip interactions is incremented in response to receiving a skip indicator or the number of skip interactions is not incremented in response to receiving a view or positive preference indicator (or an indicator of content provided to a team member and no negative preference indicator is received). In other embodiments, the viewing rate may be similarly updated. In many embodiments, steps 404 and 406 may be iteratively repeated for each member of the evaluation team, or until the number of participating team members exceeds a predetermined threshold (which may be based on the number of team members equal to a predetermined portion of the corresponding audience or demographic population) sufficient to obtain statistical accuracy of the evaluation team.

If data from additional evaluation team members is not required, the evaluation team provider or audience measurement system may determine the expected rate of content items equal to the aggregate skip rate or viewing rate in step 408, depending on the implementation. The expectation rate may be provided to the audience measurement system or stored in measurement data associated with the content item.

As discussed above, content items may be predetermined or created as low quality or high quality content items, or may be categorized as low quality or high quality in response to an aggregate skip rate or viewing rate from an evaluation group being above a predetermined threshold, respectively.

Larger evaluation teams or repeated tests may increase the accuracy of the expectation. Thus, in some implementations, in step 410, the evaluation team provider or the audience measurement system may compare the iteration count of steps 404 through 408 to a threshold, and may repeat steps 404 through 410 in response to the number of iterations being below the threshold.

During the audience measurement phase 402, a test content item may be shown to the audience member (e.g., content shown to an evaluation team, the content expectation being known) or provided to the audience member or user's client device as discussed above at 412. In some implementations, a client device may request a content item, such as a subject content item, and a content provider or distributor may choose to provide both the test content item and the subject content item. In other implementations, the content provider or distributor may choose to provide the test content in response to a first request and choose to provide the subject content in response to a second request.

In step 414, the audience measurement server may receive an indicator, such as a skip or negative preference indicator, from the client device. The indicator may be received via any of the methods or systems discussed above, such as via a parameter of the request (e.g., an HTTP request for a small image or a GET request with a parameter and/or corresponding value) or any such method.

In step 416, the audience measurement server may determine a measured skip rate (or vice versa) at which to present the test content to non-evaluation group viewers. As discussed above, the rate may be determined or updated as a function of the previous rate of the content divided by the number of requests for the content or the number of transmissions of the content (or the inverse of the value for the viewing rate). As shown, in many embodiments, steps 412 through 422 may be repeated for additional audience members, and the measured rates may be updated accordingly.

In step 418, an attention gap may be determined that is equal to the measured skip rate of presenting the test content item to non-evaluation group viewers minus the expected skip rate (or measured view rate minus expected view rate) obtained from the evaluation group for the test content item.

In step 420, the subject matter may be shown or presented to the audience member. As discussed above, the subject content item may be provided to the audience member within a predetermined time window from the presentation of the test content item to ensure the accuracy of the calculated attention gap. The subject matter item may be provided directly, or a client device of the audience member may request the subject matter item for display.

The client device may present the subject content item and the user may interact negatively with the content, interact positively with the content explicitly, interact positively with the content implicitly (e.g., by viewing the content comprehensively), or be off-line from the content item (e.g., distracted, out of the room, etc.). If the user explicitly interacts with the content, the audience measurement server may receive an identification of an explicit indicator (e.g., skip, dislike, like, etc.) in step 422. Explicit or implicit indicators may be recorded or aggregated with other indicators of the subject content item, such as an increment in a number representing the size of the viewer population and a number representing skipped or non-skipped (e.g., for determining a skip rate or viewing rate). As discussed above, in some embodiments, steps 412 through 422 may be repeated for additional audience members. In yet another embodiment, steps 412 through 422 may be iteratively repeated before or after step 424 and/or steps 426 through 430. Thus, in some implementations, the attention rate may be recalculated for each presentation of the subject matter item; while in other embodiments, a certain amount of measurement data may be collected and aggregated prior to determining the attention rate.

In step 424, the attention rate may be determined based on the measured skip rate of the subject content item plus the calculated attention gap determined by testing the content item. Attention rate may represent the percentage of users who have watched the content in its entirety and participated in the content and are not distracted or absent.

As discussed above, in some implementations, the rate charged by the content distributor to the provider of paid content (e.g., the subject content item) may be adjusted based on the attention rate. Thus, in step 426, in some implementations, the measurement system may determine whether the attention rate is less than a specified minimum attention rate in the contract or is otherwise associated with the content item. If the attention rate is not less than the specified minimum attention rate in the contract, then in some embodiments, the content provider may charge a contract rate in step 428. If the attention rate is less than the specified minimum attention rate in the contract, then in step 430 the contract rate may be reduced by a predetermined amount or percentage corresponding to the attention gap. In other implementations, based on the attention rate, the content provider or distributor may select the subject matter for wider distribution or may select other content that may be of interest to the user based on the attention rate being above a predetermined threshold. For example, in response to a high attention rate, the content distribution system may provide other content related to the subject matter for automatic presentation based on a high likelihood that the viewer will continue to participate in the content.

Thus, by displaying the tested and measured content of the evaluation team to a viewer with an unknown engagement level at a known skip rate for the fully engaged team members, the attention rate thereof can be determined by the identification of the gap between the expected skip rate and the measured skip rate. This attention rate may be used to adjust the measure of skip rate or viewing rate of other content, thereby enabling the system to distinguish users consuming the content without skipping the content from users that are distracted or not present during the presentation of the content.

In one embodiment, the present disclosure describes a method of estimating a user's attention to content presented by a client device. The method comprises the following steps: the first content item is transmitted by the audience measurement system to each of a plurality of client devices. The method further comprises the steps of: a first skip indicator of the first content item is received by the audience measurement system from each of a first subset of the plurality of client devices, the first subset of the plurality of client devices being smaller than the plurality of client devices. The method further comprises the steps of: a first skip rate for the first content item is determined by the audience measurement system based on a ratio of a number of the first subset of the plurality of client devices to a number of the plurality of client devices. The method further comprises the steps of: a second content item is transmitted by the audience measurement system to each of the plurality of client devices, the second content item being associated with a measured skip rate obtained by the evaluation group study. The method further comprises the steps of: a second skip indicator of the second content item is received by the audience measurement system from each of a second subset of the plurality of client devices, the second subset of the plurality of client devices being smaller than the plurality of client devices. The method further comprises the steps of: a second skip rate for the second content item is determined by the audience measurement system based on a ratio of the number of the second subset of the plurality of client devices to the number of the plurality of client devices. The method further comprises the steps of: a ratio difference between the second skip rate and the measured skip rate is determined by the audience measurement system. The method further comprises the steps of: calculating, by the audience measurement system, an attention rate for the first content item based on the ratio difference and the first skip rate, the attention rate indicating that the plurality of users are viewing the first content item on corresponding client devices; and adding, by the audience measurement system, an entry to a database maintained by the audience measurement system, the entry identifying a third content item and a corresponding attention rate.

In some embodiments, the method comprises: identifying, by the audience measurement system, a first subset of a population of client devices sharing a common characteristic; and selecting, by the audience measurement system, a plurality of devices corresponding to the first subset. For example, the plurality of devices may include devices of users sharing a common demographic or interest characteristic with a group of panelists.

In other embodiments, the method comprises: transmitting, by the audience measurement system, a third content item to each of the plurality of client devices, the third content item being associated with a second measured skip rate obtained from a second evaluation group study, the second measured skip rate being less than the measured skip rate. The method further comprises the steps of: a third skip indicator of the third content item is received by the audience measurement system from each of a third subset of the plurality of client devices, the third subset of the plurality of client devices being smaller than the plurality of client devices. The method further comprises the steps of: a third skip rate for the second content item is determined by the audience measurement system based on the number of the second subset of the plurality of client devices divided by the number of the plurality of client devices. The method further comprises the steps of: a second ratio difference between the third skip rate and the second measured skip rate is determined by the audience measurement system. In some such embodiments, calculating the attention rate includes: the attention rate is calculated based on the second ratio difference. In yet another embodiment, the method comprises: the attention rate is calculated by adding a first difference between the first skip rate and the difference and a second difference between the first skip rate and the second difference.

In some embodiments, the method comprises: determining that the device type of the plurality of client devices is a mobile device; and reducing the attention rate by the adjustment rate based on determining that the device type is mobile device. For example, in some such implementations, the system may assume that the user of the mobile device is more prone to getting out of the way than, for example, an individual consuming content on a smart television. Such device type specific modifiers may be applied to attention rate calculations.

In some embodiments, the method comprises: determining, by the audience measurement system, that the first content item was automatically played; and calculating the attention rate based on determining that the first content item is automatically played. For example, some systems automatically arrange additional content for presentation even if the user has left the room. The measurement system may determine whether presentation of the content item was requested manually or automatically, and may reduce the attention rate calculation by a predetermined amount or factor in response to a determination that content was requested automatically. In other embodiments, the method comprises: determining, by the audience measurement system, whether the first content item is a leading content item or a gap content item associated with the primary content; and calculating the attention rate based on determining whether the first content item is a leading content item or a gap content item.

In some embodiments, the method comprises: an attention rate report is generated by the audience measurement system for the first content item based on the attention rates retrieved from the database. The method further comprises the steps of: determining, by the audience measurement system, that the attention rate of the first content item is greater than a threshold attention rate; and in response to determining that the attention rate is greater than the threshold attention rate, transmitting, by the audience measurement system, the first content item to each of the second plurality of client devices.

In some embodiments, the method comprises: generating, by the audience measurement system, an invoice for a content provider associated with the first content item based on the attention rate; and sending, by the audience measurement system, an invoice for the first content item to the content provider. In yet another embodiment, the method comprises: a discount per presentation rate on the invoice of the content provider is calculated by the audience measurement system based on the attention rate.

In some embodiments, the method comprises: transmitting, by the audience measurement system, a fourth content item having the first duration to each of the plurality of client devices; receiving, by the audience measurement system, from each of a fourth subset of the plurality of client devices, an indication that a user of the client device selected to skip a portion of the fourth content item; determining, by the audience measurement system, a fourth skip rate for the fourth content item based on a ratio between a size of the fourth subset and a size of the plurality of client devices; transmitting, by the audience measurement system, a fifth content item having a second duration different from the first duration to each of a fifth plurality of client devices; receiving, by the audience measurement system, from each client device in a fifth subset of the second plurality of client devices, an indication that a user of the client device has selected to skip a portion of the fifth content item; determining, by the audience measurement system, a fifth skip rate for the fifth content item based on a ratio between a size of the fifth subset and a size of the fifth plurality of client devices; calculating, by the audience measurement system, an extrapolated skip rate for the third duration from the fourth skip rate for the first duration and the fifth skip rate for the second duration; wherein calculating the attention rate of the first content item is further based on a size of the first plurality of client devices, a size of the first subset, and an extrapolation skip rate.

C. participation estimation via time participation extrapolation

In another aspect, the audience measurement system may utilize a time participation curve based on similar or identical content of different durations. This allows participation in measurements or attention without the need to use an evaluation team to determine the baseline or expected rate of the test content item.

Fig. 5A and 5B are diagrams illustrating estimating audience engagement by using time engagement extrapolation, according to one embodiment. Referring to fig. 5A, an aggregate skip rate 500 over a content duration 502 for one or more audience members for a related content item is illustrated. The content may be related but have different durations, such as 15 second content items and 30 second content items in the same content category. In another embodiment, the shorter content may be repeated to fill a longer duration.

Related content of multiple durations may be provided to multiple client devices for display to a user and a skip rate or similar negative interaction rate is identified for each duration (510). As shown, a small number of users may skip very short content, while a large number of users may skip longer content. It may be assumed that if the content item has an infinite duration, each user noting the content presentation will eventually choose to skip or terminate the presentation, and thus, with 100% audience engagement or interest, there will eventually be a corresponding 100% skip rate for the infinite duration content. Thus, any difference between a theoretical 100% skip rate and a skip rate that can be extrapolated from the skip rate of shorter duration content represents the attention gap 520, or the percentage of users that are not engaged or present.

As shown, the best fit curve 512 or function may be determined based on the skip rate of a plurality of measured samples 510 or related content of different predetermined durations. Curve 512 may be extrapolated for an infinite duration of content similar to curve 514 such that an attention gap 520 may be calculated. Referring now to fig. 5B, the calculated gap 520 may then be applied to the earlier measurements 510 to estimate the percentage of users of interest that are engaged in the presentation for each content for each duration (530). As discussed above, the contract allocation billing or content selection algorithm may be adjusted based on the estimated attention rate 530 instead of the measured but exaggerated viewing rate.

Fig. 6 is a flowchart of an embodiment of a method 600 for estimating audience engagement using time engagement extrapolation, according to one embodiment.

In step 604, a content item of a first length may be provided to a plurality of client devices. As discussed above, providing the content item may include: a request for a content item is received from each client device, the content item is selected, and the content item or a uniform resource locator of the content item is sent to the client device. The measurement server may also receive an indication of a negative preference or a content item skip and may measure or determine a duration-specific skip rate based on a ratio of a number of negative preference indications received to a number of client devices receiving content for the duration in step 606.

In step 608, in some embodiments, the measurement server may determine whether the difference between two consecutive duration-specific skip rates (e.g., skip rate _t and skip rate _t-1) is less than a predetermined threshold. For example, the measurement server may determine whether the slope between two consecutive measurements is near flat or within a predetermined slope threshold. This may indicate that all users participating in the content and who will press skip for the current duration also participate in the content for the previous duration and will press skip.

If the difference is not below the threshold (e.g., if the slope between measurements is too steep), another duration may be selected in step 610 and the relevant content item may be presented to the plurality of audience members in repeating steps 604 through 610. This may be iteratively repeated until the difference between consecutive skip rate measurements in step 608 is less than a threshold, or thus when the best fit curve between two consecutive skip rate measurements is close to horizontal.

In step 612, the extrapolator of the measurement system may calculate an attention gap for the presentation of the content item based on an extrapolation of the best-fit curve of the skip rate, or a limit equal to infinity where t is close to 100%, minus the skip rate measured at various times t.

In some implementations, the measurement system can associate a gap of attention with the content item. In many implementations, the association may have a predetermined validity period such that the attention gap may be periodically discarded or updated, with additional groups of client devices receiving content for each duration.

In some embodiments, as shown in fig. 5B, the attention gap may be applied to measurements made to determine the attention gap. In such an implementation, in step 614, the audience measurement system may retrieve multiple views of the content item or the viewing rate of the duration of the content item without skipping the content or otherwise negatively interacting with the content (e.g., from an audience measurement database of content). In step 616, the determined attention gap may be applied to the retrieved value or viewing rate, resulting in a more accurate participation and viewing rate than the ratio of the audience members that were dislocated prior to or during the content presentation.

As discussed above, in some implementations, the rate charged by the content distributor to the provider of paid content (e.g., the subject content item) may be adjusted based on the attention rate. Thus, in step 618, in some implementations, the measurement system can determine whether the attention rate is less than a specified minimum attention rate in the contract or is associated with the content item. If the attention rate is not less than the specified minimum attention rate in the contract, then in some embodiments, the content provider may charge a contract rate at 620. If the attention rate is less than the specified minimum attention rate in the contract, then in step 622 the contract rate may be reduced by a predetermined amount or percentage corresponding to the attention gap. In other implementations, based on the attention rate, the content provider or distributor may select the subject matter for a broader distribution or may select other content that may be of interest to the user based on the attention rate being above a predetermined threshold. For example, in response to a high attention rate, the content distribution system may provide other content related to the subject content for automatic presentation based on a high likelihood that the viewer will continue to participate in the content.

Thus, by extrapolating the skip rate of related content of different durations to the theoretical skip rate of content of infinite length, the measurement system may determine the attention gap or attention rate that the audience member is not engaged in the content based on the difference between the theoretical skip rate and 100%. The attention rate may then be determined by reducing the measured skip rate for each duration item by the attention gap, thereby enabling the system to distinguish users consuming content without skipping from users that are distracted or absent during the content presentation, without the need to test the content or pay an evaluation team.

In one embodiment, the present disclosure describes a method of estimating a user's attention to content presented by a client device. The method comprises the following steps: a first content item having a first duration is transmitted by an audience measurement system to each of a plurality of client devices. The method further comprises the steps of: an indication is received by the audience measurement system from each of a first subset of the plurality of client devices that a user of the client device selected to skip a portion of the first content item. The method further comprises the steps of: a first skip rate for the first content item is determined by the audience measurement system based on a ratio between a size of the first subset and a size of the plurality of client devices. The method further comprises the steps of: a second content item having a second duration different from the first duration is transmitted by the audience measurement system to each of a second plurality of client devices, the second content item being related to the first content item. The method further comprises the steps of: an indication is received by the audience measurement system from each of a second subset of the second plurality of client devices that a user of the client device selected to skip a portion of the second content item. The method further comprises the steps of: a second skip rate for the second content item is determined by the audience measurement system based on a ratio between a size of the second subset and a size of the second plurality of client devices.

The method further comprises the steps of: an extrapolated skip rate for a third duration is calculated by the audience measurement system from the first skip rate for the first duration and the second skip rate for the second duration. The method further comprises the steps of: a third content item is transmitted by the audience measurement system to each of a third plurality of client devices. The method further comprises the steps of: an indication is received by the audience measurement system from each of a third subset of the third plurality of client devices that a user of the client device selected to skip a portion of the third content item. The method further comprises the steps of: determining, by the audience measurement system, an attention rate for the third content item based on a size of the third plurality of client devices, a size of the third subset, and the extrapolated skip rate; and adding, by the audience measurement system, an entry to a database maintained by the audience measurement system, the entry identifying a third content item and a corresponding attention rate.

In some embodiments of the method, determining the attention rate further comprises: the attention rate is calculated by subtracting the size of the third plurality of client devices from the difference between the size of the third plurality of client devices and the size of the third subset divided by the extrapolation skip rate. In many, but not all, embodiments of the method, the third duration is an infinite duration. In other embodiments, calculating the extrapolation skip rate further comprises: an extrapolation skip rate is calculated for a third duration that is greater than each of the first duration and the second duration.

In some embodiments, calculating the attention rate further comprises: determining that the device type of the plurality of client devices is a mobile device; and reducing the attention rate by the adjustment rate based on determining that the device type is mobile device. In other embodiments, the method comprises: determining, by the audience measurement system, that the third content item was automatically played; and calculating the attention rate based on the determination that the third content item is automatically played.

In some embodiments, the method comprises: determining, by the audience measurement system, whether the third content item is a leading content item or a gap content item associated with the primary content; and calculating the attention rate based on determining whether the third content item is the leading content item or the gap content item.

In other embodiments, the method comprises: an attention rate report is generated by the audience measurement system for the third content item based on the attention rates retrieved from the database. In yet another embodiment, the method further comprises: determining, by the audience measurement system, that the attention rate of the third content item is greater than a threshold attention rate; and in response to determining that the attention rate is greater than the threshold attention rate, transmitting, by the audience measurement system, a third content item to each of the second plurality of client devices.

In yet another embodiment, the method comprises: generating, by the audience measurement system, an invoice for a content provider associated with the third content item based on the attention rate; and sending, by the audience measurement system, an invoice for the third content item to the content provider. In yet another embodiment, the method comprises: a discount per presentation rate on the invoice of the content provider is calculated by the audience measurement system based on the attention rate.

As discussed above, in many such embodiments, the collected data may be anonymous or disambiguated to preserve privacy, particularly for individuals other than the panelist. In many such implementations, the user may be provided with the following opportunities in similar situations where personal information about the user of the client device may be collected for measurements or may be used to select third party content: whether or not a program or feature that controls the possible collection of personal information (e.g., information about the user's social network, social actions or activities, the user's preferences, or the user's current location) can collect personal information, or whether or how to send measurement data to an audience measurement server and/or an evaluation team provider. In addition, the particular data may be processed in one or more ways prior to being stored or used by the audience measurement server so that personal identity information may be removed when parameters (e.g., demographic parameters) are generated. The identity of the user may be anonymous so that the user's personal identity information cannot be determined, or in the case of location information being obtained, the user's geographic location may be generalized (such as to a city, zip code, or state county level) so that the user's particular location cannot be determined. Thus, a user can control how information about the user is collected, and how the information is used by the audience measurement server, the panel provider, and the content provider.

Implementations of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on one or more computer storage media, executed by, or controlling the operation of, data processing apparatus. Alternatively or additionally, the program instructions may be encoded on a manually-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by data processing apparatus. The computer storage medium may be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Furthermore, although the computer storage medium is not a propagated signal, the computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage media may also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). Thus, the computer storage medium may be tangible.

The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer readable storage devices or received from other sources.

The term "client" or "server" includes all kinds of devices, apparatuses, and machines for processing data, such as a programmable processor, a computer, a system-on-a-chip, or a combination or plurality of the foregoing. The device may comprise dedicated logic circuitry, for example an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the device may include code that creates an execution environment for the computer program under discussion, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. Devices and execution environments may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

A computer program (also known as a program, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. The computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Processors suitable for the execution of a computer program include both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Key elements of the computer are: a processor for performing actions in accordance with instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or a computer can be operatively coupled to receive data from or transfer data to the mass storage devices, or both. However, the computer need not have such a device. In addition, the computer may be embedded in another device, such as a mobile phone, a Personal Digital Assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., universal Serial Bus (USB) flash drive), to name a few. Means suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disk; CD-ROM discs and DVD-ROM discs. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, e.g., a CRT (cathode ray tube), LCD (liquid crystal display), OLED (organic light emitting diode), TFT (thin film transistor), plasma, other flexible configuration, or any other monitor, as well as a keyboard, a pointing device (e.g., a mouse, a trackball, etc.), or a touch screen, a touchpad, etc., by which the user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic input, speech input, or tactile input. Further, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, by sending web pages to a web browser on the user's user device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. The communication network may include: local area networks ("LANs") and wide area networks ("WANs"), the internet (e.g., the internet), and point-to-point networks (e.g., ad hoc point-to-point networks).

The features disclosed herein may be implemented on a smart television module (or a networked television module, a hybrid television module, etc.) that may include processing circuitry configured to integrate internet connectivity with more traditional television program sources (e.g., received via cable, satellite, wireless, or other signals). The smart television module may be physically incorporated into a television or the smart television module may include a separate device such as a set-top box, a blue-ray or other digital media player, a gaming machine, a hotel television system, and other companion devices. The smart television module may be configured to allow a viewer to search and find video, movies, photos, and other content on the web, on a local cable television channel, on a satellite television channel, or stored on a local hard disk drive. A Set Top Box (STB) or Set Top Unit (STU) may include an information application device that may contain a tuner and that is connected to a television set and an external signal source, converts the signal into content, which is then displayed on a television screen or other display device. The smart television module may be configured to provide a home screen or top screen that includes icons for a plurality of different applications, such as web browsers and a variety of streaming media services, networked cable or satellite media sources, other network "channels," and the like. The intelligent television module may be further configured to provide the electronic program guide to the user. The companion application of the smart television module may operate on the mobile computing device to provide additional information to the user regarding available programs to allow the user to control the smart television module, etc. In alternative implementations, the features may be implemented on a laptop or other personal computer, smart phone, other mobile phone, handheld computer, tablet PC, or other computing device.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Also, although operations are illustrated in the figures in a particular order, it should not be understood that such operations need to be performed in the particular order illustrated or in sequential order, or that all illustrated operations need be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Thus, particular embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method of estimating user attention to content presented by a client device, comprising:

transmitting, by the audience measurement system, the first content item to each of a plurality of client devices;

Receiving, by the audience measurement system, a first skip indicator for the first content item from each of a first subset of the plurality of client devices, the first subset of the plurality of client devices being smaller than the plurality of client devices;

determining, by the audience measurement system, a first skip rate for the first content item based on a ratio of a number of the first subset of the plurality of client devices to a number of the plurality of client devices;

Transmitting, by the audience measurement system, a second content item to each of the plurality of client devices, the second content item being associated with a measured skip rate obtained from an evaluation group study;

Receiving, by the audience measurement system, a second skip indicator for the second content item from each of a second subset of the plurality of client devices, the second subset of the plurality of client devices being smaller than the plurality of client devices;

Determining, by the audience measurement system, a second skip rate for the second content item based on a ratio of a number of the second subset of the plurality of client devices to a number of the plurality of client devices;

determining, by the audience measurement system, a ratio difference between the second skip rate and the measured skip rate;

Calculating, by the audience measurement system, an attention rate of the first content item by adding the ratio difference and the first skip rate, the attention rate indicating that a plurality of users are viewing the first content item on corresponding client devices; and

An entry is added by the audience measurement system to a database maintained by the audience measurement system, the entry identifying the first content item and a corresponding attention rate.

2. The method of claim 1, further comprising:

identifying, by the audience measurement system, a first subset of a population of client devices sharing a common characteristic;

selecting, by the audience measurement system, the plurality of client devices from the first subset of the population of client devices sharing the common characteristic; and

The second content item associated with the measured skip rate obtained from the evaluation group study is selected by the audience measurement system in response to determining that participants of the evaluation group study share the common characteristic.

3. The method of claim 1, wherein calculating the attention rate further comprises:

determining that the device type of the plurality of client devices is a mobile device; and

The attention rate is reduced by an adjustment rate based on determining that the device type is the mobile device.

4. The method of claim 1, further comprising:

determining, by the audience measurement system, that the first content item was automatically played; and

The content item associated with the evaluation team study including automatically playing content item is responsively selected as the second content item.

5. The method of claim 1, further comprising:

determining, by the audience measurement system, whether the first content item is a leading content item or a gap content item associated with primary content; and

Content items associated with the evaluation team study that are also pre-or interstitial content items are responsively selected as the second content item.

6. The method of claim 1, further comprising:

an attention rate report is generated by the audience measurement system for the first content item based on the attention rate retrieved from the database.

7. The method of claim 1, further comprising:

Determining, by the audience measurement system, that the attention rate of the first content item is greater than a threshold attention rate; and

The first content item is transmitted by the audience measurement system to each of a second plurality of client devices in response to determining that the attention rate is greater than the threshold attention rate.

8. The method of claim 1, further comprising:

Generating, by the audience measurement system, an invoice for a content provider associated with the first content item based on the attention rate; and

The invoice of the first content item is transmitted by the audience measurement system to the content provider.

9. The method of claim 8, further comprising calculating, by the audience measurement system, a discount per presentation rate on the invoice of the content provider based on the attention rate.

10. A system for estimating user attention to content presented by a client device, comprising:

A processor; and

A memory storing processor-executable instructions that, when executed by the processor, cause the processor to:

transmitting a first content item to each of a plurality of client devices;

Receiving a first skip indicator for the first content item from each of a first subset of the plurality of client devices, the first subset of the plurality of client devices being smaller than the plurality of client devices;

determining a first skip rate for the first content item based on a ratio of a number of the first subset of the plurality of client devices to a number of the plurality of client devices;

transmitting a second content item to each of the plurality of client devices, the second content item being associated with a measured skip rate obtained from an evaluation group study;

Receiving a second skip indicator for the second content item from each of a second subset of the plurality of client devices, the second subset of the plurality of client devices being smaller than the plurality of client devices;

determining a second skip rate for the second content item based on a ratio of a number of the second subset of the plurality of client devices to a number of the plurality of client devices;

determining a ratio difference between the second skip rate and the measured skip rate;

calculating an attention rate of the first content item by adding the ratio difference and the first skip rate, the attention rate indicating that a plurality of users are viewing the first content item on corresponding client devices; and

An entry is added to a database maintained by the system, the entry identifying the first content item and a corresponding attention rate.

11. The system of claim 10, wherein the processor-executable instructions, when executed by the processor, further cause the processor to:

identifying a first subset of a population of client devices sharing a common characteristic;

Selecting the plurality of client devices from the first subset of a population of client devices sharing the common characteristic; and

The second content item associated with the measured skip rate obtained from the evaluation group study is selected in response to determining that participants of the evaluation group study share the common characteristic.

12. The system of claim 10, wherein calculating the attention rate further comprises:

13. The system of claim 10, wherein the processor-executable instructions, when executed by the processor, further cause the processor to:

determining that the first content item is automatically played; and

14. The system of claim 10, wherein the processor-executable instructions, when executed by the processor, further cause the processor to:

determining whether the first content item is a leading content item or a gap content item associated with primary content; and

15. The system of claim 10, wherein the processor-executable instructions, when executed by the processor, further cause the processor to generate an attention rate report for the first content item based on the attention rate retrieved from the database.

16. The system of claim 10, wherein the processor-executable instructions, when executed by the processor, further cause the processor to:

determining that the attention rate of the first content item is greater than a threshold attention rate; and

In response to determining that the attention rate is greater than the threshold attention rate, the first content item is sent to each of a second plurality of client devices.