WO2014078995A1

WO2014078995A1 - System and method for calculating predicted measure of content performance

Info

Publication number: WO2014078995A1
Application number: PCT/CN2012/084956
Authority: WO
Inventors: Ting Liu; Xiao Wu; Ching Law
Original assignee: Google Inc.
Priority date: 2012-11-21
Filing date: 2012-11-21
Publication date: 2014-05-30

Abstract

A predictive measure of content performance is calculated by receiving data relating to impressions of content. Each impression is grouped based on a topic associated with each impression. The impressions within each topic are further grouped based on level of user interest in the topic. A measure of content performance is calculated based on the data. A predictive measure of content performance is calculated based on the data and user information. A calibration factor is calculated for each topic and for each level of user interest within each topic based on the measure of content performance and predictive measure of content performance. At least one calibrated predictive measure of content performance is calculated using the calibration factor for each topic, wherein the calibrated predictive measure of content performance is further based on the level of user interest in the topic.

Description

SYSTEM AND METHOD FOR CALCULATING PREDICTED MEASURE OF CONTENT PERFORMANCE

BACKGROUND

[0001] In network-based content distribution systems (e.g., the Internet or other computer network), content providers may provide content (e.g., video content, audio content, text content, graphical content, etc.) for presentation or display on a network resource (e.g., a webpage, intranet resource, local resource, etc.). However, the content displayed on the resource may be of little value to either the content provider or a user viewing the content if the user has no interest in the presented content.

SUMMARY

[0002] One implementation of the present disclosure relates to a method for calculating a predictive measure of content performance. The predictive measure of content performance is calculated by receiving data relating to impressions of content. Each impression is grouped based on a topic associated with each impression. The impressions within each topic are further grouped based on level of user interest in the topic. A measure of content performance is calculated based on the data. A predictive measure of content performance is calculated based on the data and user information. A calibration factor is calculated for each topic and for each level of user interest within each topic based on the measure of content performance and predictive measure of content performance. At least one calibrated predictive measure of content performance is calculated using the calibration factor for each topic, wherein the calibrated predictive measure of content performance is further based on the level of user interest in the topic.

[0003] Another implementation of the present disclosure relates to a method for calculating a predictive measure of content performance, wherein the content is to be displayed to one or more users. The method includes receiving data relating to the number of impressions associated with a user; the impressions relating to content views. The method further includes determining a score for each impression, the score relating to a level of interest between the impression and the user's interest in the impression. The method further includes grouping each impression based on a topic and score associated with each impression. The method further includes calculating a measure of content performance based on the data. The method further includes calculating a predictive measure of content performance based on the data and user information. The method further includes calculating a calibration factor for each grouping based on the measure of content performance and predictive measure of content performance. The method further includes calculating a calibrated predictive measure of content performance using the calibration factor for each grouping.

[0004] Another implementation of the present disclosure relates to a system for calculating a predictive measure of content performance, wherein the content is to be displayed to one or more users. The system includes a processing circuit operable to: receive data relating to the number of impressions associated with a user; the impressions relating to content views; determine a score for each impression, the score relating to a level of interest between the impression and the user's interest in the impression; group each impression based on a topic associated with each impression; calculate a measure of content performance based on the data; calculate a predictive measure of content performance based on the data and user information; calculate a calibration factor for each grouping based on the measure of content performance and predictive measure of content performance; and calculate a calibrated predictive measure of content performance using the calibration factor for each grouping.

[0005] These implementations are mentioned not to limit or define the scope of the disclosure, but to provide an example of an implementation of the disclosure to aid in understanding thereof. Particular implementations may be developed to realize one or more of the following advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0007] FIG. 1 is a block diagram of a computer system in accordance with a described implementation.

[0008] FIG. 2 is a more detailed block diagram of the content management system of FIG. 1 in accordance with a described implementation.

[0009] FIG. 3 is a flow chart of a process for calculating a predictive measure of content performance in accordance with a described implementation.

[0010] FIG. 4 is another flow chart of a process for calculating a predictive measure of content performance in accordance with a described implementation.

DETAILED DESCRIPTION OF THE ILLUSTRATF/E IMPLEMENTATIONS

[0011] Referring generally to the figures, systems and methods for calculating a predicted measure of content performance are shown and described. The systems and methods described herein may be used to calculate a predicted performance of content when the content is displayed client device accessing a resource (e.g., a webpage, intranet resource, local resource, etc.). For example, content may be selected for display on a resource based on the predicted performance of the content, based on a variety of relevant information. In one implementation, the predicted performance may be a predicted click-through rate of the content; in various other implementations, other measures of predicted performance may be used. The predicted performance of the content may be topic and weight-based (e.g., predicted performance calculations may include

considerations of the topic of the content and resource, and the calculations may be weighted based on a predicted level of user interest). [0012] In calculating a predicted performance of content, various factors may be considered, such as interests categories, a previous resource views, etc. For example, it may be desirable to consider impressions associated with previous resource views (e.g., previous instances a particular resource was provided to a client device or the previous resources provided to a client device). Impressions may represent the number of views of a particular resource or content on the resource. Impressions may be analyzed based on the total number of views, a topic or category that the views fall into, etc. The systems and methods of the present disclosure describe a calculation of predicted performance of content that uses information relating to the user interests, previous resource views, impressions, etc. In considering such information, a calibration factor may be calculated which is representative of the user interests.

[0013] For a particular content item, a predicted performance of the content may be calculated. Then, the calibration factor may be calculated as described above and used to adjust the predicted performance measurement. As a result, the new predicted performance of the advertisement may be compared to predicted performances of other content, and one or more content items may be selected for display based on the predicted performance. In other words, user interests may be predicted to determine which content to display.

[0014] Referring now to FIG. 1 , a block diagram of a computer system 100 in accordance with a described implementation is shown. In brief overview, computer system 100 may include a content management system 104, a content source 106, a network 102, and a client device 108. Computer system 100 may provide content to a client device 108 via network 102. Computer system 100 may generally be configured to use information pertaining to the client device 108 as, well as other information, to select content to provide to the client device 108.

[0015] Still referring to FIG. 1 , and in greater detail, network 102 may be any form of computer network that relays information between content sources 106 and client devices 108. For example, network 102 may include the Internet and/or other types of data networks, such as a local area network (LAN), a wide area network (WAN), a cellular network, satellite network, or other types of data networks. Network 102 may also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) that are configured to receive and/or transmit data within network 102. Network 102 may further include any number of hardwired and/or wireless connections. For example, client device 108 may communicate wirelessly (e.g., via WiFi, cellular, radio, etc.) with a transceiver that is hardwired (e.g., via a fiber optic cable, a CAT5 cable, etc.) to other computing devices in network 102.

[0016] Still referring to FIG. 1 , computer system 100 may include one or more client devices 108 which communicate with other computing devices via a network 102. Client device 108 may execute a web browser or other application to retrieve content from other devices via network 102. For example, client device 108 may communicate with any number of content sources 106. Content sources 106 may provide content data (e.g., text documents, PDF files, webpage data, and other forms of electronic documents) to client device 108.

[0017] Client devices 108 may be any number of different types of user electronic devices configured to communicate via network 102 (e.g., a laptop computer, a desktop computer, a mobile phone or other mobile device, a tablet computer, a smartphone, a digital video recorder, a set-top box for a television, a video game console, combinations thereof, etc.).

[0018] Client device 108 may include a processor 110 and memory 1 12, i.e., a processing circuit. Memory 112 may store machine instructions that, when executed by processor 110 to cause processor 110 to perform one or more of the operations described herein.

Processor 110 may include a microprocessor, ASIC, FPGA, etc., or combinations thereof. Memory 112 may include, but is not limited to, electronic, optical, magnetic, or any other storage or transmission device capable of providing processor 110 with program instructions. Memory 112 may include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, EEPROM, EPROM, flash memory, optical media, or any other suitable memory from which processor 110 can read instructions. The instructions may include code from any suitable computer programming language such as, but not limited to, C, C++, C#, Java, JavaScript, Perl, HTML, XML, Python, and Visual Basic.

[0019] Client device 108 may include one or more user interface devices. A user interface device may be any electronic device that conveys data to a user by generating sensory information (e.g., a visualization on a display, one or more sounds, etc.) and/or converts received sensory information from a user into electronic signals (e.g., a keyboard, a mouse, a pointing device, a touch screen display, a microphone, etc.). The one or more user interface devices may be internal to the housing of client device 108 (e.g., a built-in display, microphone, etc.) or external to the housing of client device 108 (e.g., a monitor or speaker connected to client device 108, etc.), according to various implementations. For example, client device 108 may include an electronic display 1 14, which displays content, resources, or other data received from content sources 106 and/or content management system 104.

[0020] Still referring to FIG. 1 , computer system 100 may include one or more content sources 106. Content sources 106 may be one or more electronic devices connected to network 102 that provide content to client devices 108. For example, content sources 106 may be computer servers (e.g., FTP servers, file sharing servers, web servers, etc.) or combinations of servers (e.g., data centers, cloud computing platforms, etc.). Content may include, but is not limited to, webpage data, a text file, a spreadsheet, images, and other forms of electronics documents.

[0021] According to various implementations, content sources 106 may provide content data to client devices 108 that includes one or more content tags. In general, a content tag may be any piece of code associated with including content with a resource. According to various implementations, a content tag may define a slot on a resource for additional content, a slot for "out of page" content (e.g., an interstitial slot), whether content should be loaded asynchronously or synchronously, whether the loading of extraneous content should be disabled on the resource, whether content that loaded unsuccessfully should be refreshed, the network location of a content source that provides the content (e.g., content sources 106, content management system 104, etc.), a network location (e.g., a URL) associated with clicking on the content, how the content is to be rendered on a display, one or more keywords used to retrieve the content, and other functions associated with providing additional content with a resource. For example, content sources 106 may provide resource data that causes client devices 108 to retrieve content from content management system 104. In another implementation, the content may be selected by content management system 104 and provided by a content source 106 as part of the resource data sent to client device 108.

[0022] Still referring to FIG. 1 , computer system 100 may include a content management system 104 configured to receive information from a client device 108, to calculate a predicted measure of performance of content, and to select content to display on client device 108 based on the predicted measure of the content performance. Content management system 104 may be configured to manage content provided to client devices 108 by content sources 106 or another source connected to network 102.

[0023] Similar to content sources 106, content management system 104 may be one or more electronic devices connected to network 102 that provide content to client devices 108. Content management system 104 may be a computer server (e.g., FTP servers, file sharing servers, web servers, etc.) or a combination of servers (e.g., a data center, a cloud computing platform, etc.). The activities and structure of content management system 104 are shown in greater detail in FIG. 2.

[0024] Content selected by content management system 104 may be provided to a client device 108 by content sources 106 or content management system 104. For example, content management system 100 may select content from content sources 106 to be included with a resource served by a content source 106. In another example, content management system 104 may provide the selected content to a client device 108. In some implementations, content management system 104 may select content stored in memory 112 of a client device 108. The content may be selected based on previous resources provided to the client device and a predicted measure of content performance, as generally described in the present disclosure. [0025] The selection of content may further be based on a device identifier associated with client device 108. The device identifier may refer to any form of data that may be used to represent a client device selected to receive relevant content selected by content management system 104. In some implementations, a device identifier may be associated with a user of the client device. In some implementations, multiple device identifiers may be associated with a single user (e.g., a device identifier for a mobile device, a device identifier for a stationary computer, etc.). Device identifiers may include, but are not limited to, device serial numbers, network addresses, or other device identifying data. For example, a device identifier associated with client device 108 may be used to identify client device 108 and the client to content management system 104.

[0026] Content management system 104 may use information associated with a device identifier to select relevant content for the device. For example, content management system 104 may analyze previously retrieved resource data associated with a device identifier to determine one or more potential interest categories (e.g., topics). Previously retrieved resource data may be any data associated with a device identifier that is indicative of an online action (e.g., visiting a webpage, selecting an advertisement, navigating to a webpage, making a purchase, downloading content, etc.). Content providers that have content matching an interest category of a device identifier may select content to be provided to a device. For example, with reference to the present disclosure, the interest categories or topics may be used to calculate a predicted measure of content performance for a given content item, and that measure may be used to select which content to display.

[0027] The content provided by content sources 106 may be advertisements. The advertisements may be image advertisements, flash advertisements, video advertisements, text-based advertisements, or any combination thereof. It should be understood that while the present disclosure is described with reference to advertisements in general, the type of advertisement or other content displayed via a client device 108 may vary according to various implementations.

[0028] Computer system 100 is illustrated as an example environment for use with the systems and methods of the present disclosure; in various implementations, computer system 100 may include more or less systems and modules for use with the systems and methods of the present disclosure.

[0029] In situations in which the systems discussed here collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect user information (e.g, information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), or to control whether and/or how to receive content from the content server that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and used by a content server.

[0030] Referring now to FIG. 2, content management system 104 is shown in greater detail. In brief overview, content management system 104 may include an input/output device 214, and processing electronics 202 comprising a processor 204 and memory 206. Memory 206 may include an impressions module 208, a content performance calculation module 210, and a content selection module 212. Content management system 104 may be configured to select content to display to a client device 108. Content management system 104 may select the content based on information such as a predicted measure of performance of the content to be displayed. For example, a predicted click-through rate (pCTR) may be calculated for one or more content items, and one or more of the content items may be chosen for display based on the pCTR.

[0031] Still referring to FIG. 2, and in greater detail, content management system 104 may include processing electronics 202 including a processor 204 and memory 206. Processor 204 may be implemented as a general purpose processor, an application specific integrated circuit (ASIC), one or more field programmable gate arrays (FPGAs), a group of processing components, or other suitable electronic processing components. Memory 206 is one or more devices (e.g., RAM, ROM, Flash memory, hard disk storage, etc.) for storing data and/or computer code for completing and/or facilitating the various processes described herein. Memory 206 may be or include non-transient volatile memory or non-volatile memory. Memory 206 may include data base components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described herein. Memory 206 may be communicably connected to processor 204 and includes computer code or instructions for executing one or more processes described herein.

[0032] Still referring to FIG. 2, memory 206 may include various modules for completing the methods described herein. For example, referring also to FIG. 3, impressions module 208 may receive impression data for a client device (block 302) and group impressions based on topics associated with the impressions (block 304). Content performance calculation module 210 may be used to calculate a measure of content performance (block 306), calculate a predictive measure of performance (block 308), calculate a calibration factor for each grouping (block 310), and calculate a calibrated predictive measure of performance (312). Content selection module 212 may then be used to select content based on either the predicted measure of performance or calibrated predicted measure of performance. It should be understood that memory 206 may include more or less modules, and that some of the activity described as occurring within memory 206 and processing electronics 202 may be completed by modules located remotely from content management system 104 or processing electronics 202.

[0033] Still referring to FIG. 2, memory 206 may include an impressions module 208. Referring also to FIG. 3, impressions module 208 may be configured to receive device information such as previously viewed resource information and to determine impressions (blocks 302 and 304). An impression may represent the provision of a particular resource or content item to a client device. Impressions module 208 may measure content impressions for a client device (e.g., the number of times the content was provided to the client device). [0034] Impressions may represent the number of views of a particular resource or content on the resource. Impressions may be analyzed for content across a range of topics.

Information relating to previously retrieved resources and other information may be received by impressions module 208. Such information may be used to assess performance of the content for a client device or group of client devices.

[0035] Impressions module 208 may further partition the impressions. In one implementation, the impressions may be partitioned based on the topic (e.g., the vertical) associated with the impression. For example, for an impression related to a vehicle-related content, the impression may be partitioned into a category with other impressions pertaining to vehicle-related content. All impressions received or determined by impressions module 208 may be partitioned into a category. The categories may be determined based on criteria that are different from criteria used to categorize content topics, according to one implementation.

[0036] Additionally, impressions module 208 may further partition the impressions within each topic. For example, within a given topic, the impressions may be partitioned based a predicted level of interest in the topic. For example, impressions of content within the topic that were presented to users having a first level of interest in the topic may be placed into a first bucket, impressions of content within the topic that were presented to users having a second level of interest in the topic may be placed into a second bucket, and so on.

[0037] Still referring to FIG. 2, memory 206 may further include a content performance calculation module 210. Referring also to FIG. 3, content performance calculation module 210 may be configured to calculate a predictive measure of content performance based on device information and impression information (blocks 306 and 308). The predictive measure of content performance may be calculated based on a previous predictive measure of content performance and a calculated calibration factor that considers impression information.

[0038] In some embodiments, content performance calculation module 210 may first calculate or receive a measure of content performance and predictive measure of content performance. For example, content performance may be observed for each impression received by impressions module 208, and a measure of content performance may be calculated based on the observations. The measure of content performance may be based on a single content item or a group of content items (e.g., all content within a particular category). As one example, for all impressions, the number of clicks that occurred on a content item as a result of an impression is analyzed and used to calculate the measure of content performance. Further, a predictive measure of content performance may be calculated for each content item or all content items in the same category, based on various information such as the measure of content performance, previously accessed resource information, historic content performance information, etc. A measure of content performance and predictive measure of content performance may be calculated for each category as determined by impressions module 208, in one implementation.

[0039] In some embodiments, a predictive measure of content performance may be calculated for each content item or all content items within a given bucket in a particular category, based on various information such as the measure of content performance, previously accessed resources, historic content performance information, etc. For example, within a given topic, a first measure of content performance and a first predictive measure of content performance may be calculated for users having a first level of interest in the topic (i.e., users of client devices within the first abovementioned bucket), a second measure of content performance and a second predictive measure of content performance may be calculated for users having a second level of interest in the topic (i.e., users of client devices within the second abovementioned bucket), and so on.

[0040] The measure of content performance may be a content click-through rate (CTR) and the predictive measure of content performance may be a predicted click-through rate (pCTR), according to one implementation. In one implementation, content performance calculation module 210 may calculate the initial CTR and pCTR based on device data, historical data, or other data.

[0041] In other implementations, other measures of content performance may be used such as the number of impressions or views, a cost per action or cost per click (e.g., the number of interactions made by a client device after the content is selected), or revenue generated by the content. While the present disclosure refers to CTR and pCTR as measures of content performance, it should be appreciated that the systems and methods of the present disclosure may be implemented using other measures of content performance.

[0042] Still referring to FIG. 2 and FIG. 3, content performance calculation module 210 may calculate a calibrated predicted measure of content performance (e.g., a calibrated pCTR) for a content item based the CTR and pCTR for each category (blocks 310 and 312). First, a calibration factor may be calculated that represents a difference between a CTR and pCTR. Then, for a content item, a calibrated pCTR may be calculated based on the originally calculated pCTR and the calibration factor. The resulting calibrated pCTR (or other predictive measure of content performance) may then be used instead of the original pCTR as a better predictor of content performance.

[0043] A calibration factor may be calculated for each category. A calibration factor may also be calculated for each bucket within each category. For example, a CTR and pCTR may be determined for a category as described above, and a calibration factor may be calculated for each category using the CTR and pCTR. In one implementation, the calibration factor may be calculated by using a log odd ratio function. For example, the following equation may be used to calculate the calibration factor:

calibration/actor = \ogodd(CTR) - logodd(pCTR) .

[0044] After calculating the calibration factor, another equation may be used to calculate the calibrated measure of content performance. For each content item for a particular impression, the content item may be matched to a category as determined by impressions module 208 above. The calibration factor for the matched category may then be used to calculate a calibrated pCTR for the content. The calculation may include a sigmoid function as shown below:

calibratedPCTR = sigmoid(\ogodd(pCTR) + calibration/actor) . The calibrated pCTR may then be used by content selection module 212 or another module of content management system 104. [0045] Content performance calculation module 210 may further determine whether to calculate the calibrated predictive measure of content performance or to use the original calculated predicted measure of content performance. The number of impressions may differ significantly for different categories or verticals. Some of the categories may have little or no entries due to a lack of impressions for the category, or some of the categories may have a very high observed CTR because a client device interacted extensively with a small number of content items. Content performance calculation module 210 may be configured to detect such situations and to ignore the impressions and categories. For example, if a category has an unusually high or low CTR compared to historical data and there were relatively few impressions, the category may be discarded from the calculations. For the discarded categories, the original pCTR may be used instead of a calibrated pCTR.

[0046] Still referring to FIG. 2, memory 206 may further include a content selection module 212. Content selection module 212 may receive the calibrated pCTR (or other measure of content performance) for each content item as calculated in content performance calculation module 210. Using the calibrated pCTR and other information, content selection module 212 may select which content to provide to a client device. For example, the content with the highest pCTR may be chosen for display. As another example, other factors such as content provider preference, the type of content, a client device profile or previously viewed resource information may be used in addition to pCTR to select a content item for display.

[0047] Still referring to FIG. 2, content management system 104 may include an input/output (I/O) interface 214 configured to receive data from various client devices and to transmit content to the various client devices as described above. I/O interface 214 is configured to facilitate communications, either via a wired connection or wirelessly, with the client device, network, content management system, and other devices as described in the present disclosure.

[0048] Referring again to impressions module 208, impressions may be partitioned into categories as described above. This may be known as "vertical-based pCTR calibration". A calibration factor is calculated for each category, and a pCTR for a given content item may be calibrated using the calibration factor from the corresponding category. This process is shown in greater detail in FIG. 3.

[0049] Referring now to process 300 of FIG. 3, a process of calculating a predictive measure of content performance is shown. Process 300 may include receiving impression data for a client device (block 302) and grouping the impressions based on topics associated with the impressions (block 304). The activities of blocks 302, 304 may be executed by impressions module 208 as described above.

[0050] Process 300 may further include calculating a measure of content performance (block 306) and predictive measure of content performance (block 308). Using the calculations from blocks 306, 308, a calibration factor may be calculated for each grouping established in block 304 (block 310). A calibrated predictive measure of content performance may then be calculated (block 312) as described with reference to content performance calculation module 210.

[0051] Referring again to impressions module 208, in addition to partitioning the impressions into categories as described above, the impressions may be further partitioned based on a predicted level of interest in the topic represented by the category. The predicted level of interest may be assigned a score, and the categories may be further divided based on the scores. For example, for a given category, the category may be divided into two or more sub-categories or buckets, with each bucket representing a different score range. Then, a calibration factor may be calculated for each bucket, and a pCTR for a given content item may be calibrated using the calibration factor from the corresponding bucket, based on the level of interest of the user to whom the content is to be presented. This process is shown in greater detail in FIG. 4.

[0052] Referring now to process 400 of FIG. 4, another process of calculating a predictive measure of content performance is shown. Process 400 may include receiving impression data for a client device (block 402). Process 400 may further include determining a score for each impression (block 404). The score may relate to a level of interest in a topic associated with the impression (e.g., a level of interest in cars if the impressions are associated with a car-related content). Process 400 may further include grouping impressions based on the topics and scores associated with the impressions (block 406).

[0053] In one implementation, the score may range in value from 0 to 1 , with a higher score indicating a higher level of interest. For each category, there may be any number of buckets, each bucket relating to a range of score values. As one example, a category may be divided into 5 buckets, with score ranges from 0 to 0.025, 0.025-0.05, 0.05-0.1 , 0.1-0.2, and 0.2-1.0. The division of the buckets and score ranges may be determined based on the score values typically observed. For example, the divisions may be made such that an approximately equal number of impressions are in each bucket. As another example, the divisions may be made based on mathematical functions (e.g., the score ranges described above approximately mirror an exponential series). It should be appreciated that the scores may range in between any number of values, the number of sub-categories may vary, and the methods of calculating the score and categorizing the score in a sub-category may vary without departing from the scope of the present disclosure, and that the numbers used herein are examples only.

[0054] The scores may be assigned based on various properties associated with the levels of interest. For example, a timestamp associated with the impression may impact the score (e.g., the more recent an impression occurred, the higher the score). The score may decrease for an impression based on a schedule (e.g., the score may "decay" on a daily or weekly basis). As another example, client device interaction based on the impression may impact the score (e.g., the more the client device interacted with the content, the higher the score of the associated impression).

[0055] Still referring to FIG. 4, process 400 may further include calculating a measure of content performance (block 408) and predictive measure of content performance (block 410). Using the calculations from blocks 408, 410, a calibration factor may be calculated for each grouping established in block 406 (block 412). A calibrated predictive measure of content performance may then be calculated (block 414) as described with reference to content performance calculation module 210. The calibrated predictive measure of content performance may consider a predicted level of interest. For example, following the example above, if the content is to be presented to a user having a first level of interest, the calibrated predictive measure of content performance may be calculated using the calibration factor associated with the first bucket of impressions (that is, impressions presented to other users having the same level of interest as the present user).

[0056] Implementations of the subject matter and the operations described in this specification may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on one or more computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions may be encoded on an artificially-generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium may be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium may also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). Accordingly, the computer storage medium is both tangible and non-transitory.

[0057] The operations described in this disclosure may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0058] The term "client or "server" include all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus may include special purpose logic circuitry, e.g., a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The apparatus may also include, in addition to hardware, code that creates an execution environment for the computer program in question (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them). The apparatus and execution environment may realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0059] The systems and methods of the present disclosure may be completed by any computer program. A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0060] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry (e.g., an FPGA or an ASIC).

[0061] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data (e.g., magnetic,

magneto-optical disks, or optical disks). However, a computer need not have such devices. Moreover, a computer may be embedded in another device (e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc.). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks;

magneto-optical disks; and CD-ROM and DVD-ROM disks). The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0062] To provide for interaction with a user, implementations of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display), OLED (organic light emitting diode), TFT (thin-film transistor), or other flexible configuration, or any other monitor for displaying information to the user and a keyboard, a pointing device, e.g., a mouse, trackball, etc., or a touch screen, touch pad, etc.) by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser. [0063] Implementations of the subject matter described in this disclosure may be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer) having a graphical user interface or a web browser through which a user may interact with an implementation of the subject matter described in this disclosure, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a

communication network). Examples of communication networks include a LAN and a WAN, an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0064] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any disclosures or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular disclosures. Certain features that are described in this disclosure in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0065] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products embodied on one or more tangible media.

[0066] The features disclosed herein may be implemented on a smart television module (or connected television module, hybrid television module, etc.), which may include a processing circuit configured to integrate internet connectivity with more traditional television programming sources (e.g., received via cable, satellite, over-the-air, or other signals). The smart television module may be physically incorporated into a television set or may include a separate device such as a set-top box, Blu-ray or other digital media player, game console, hotel television system, and other companion device. A smart television module may be configured to allow viewers to search and find videos, movies, photos and other content on the web, on a local cable TV channel, on a satellite TV channel, or stored on a local hard drive. A set-top box (STB) or set-top unit (STU) may include an information appliance device that may contain a tuner and connect to a television set and an external source of signal, turning the signal into content which is then displayed on the television screen or other display device. A smart television module may be configured to provide a home screen or top level screen including icons for a plurality of different applications, such as a web browser and a plurality of streaming media services (e.g., Netflix, Vudu, Hulu, etc.), a connected cable or satellite media source, other web

"channels", etc. The smart television module may further be configured to provide an electronic programming guide to the user. A companion application to the smart television module may be operable on a mobile computing device to provide additional information about available programs to a user, to allow the user to control the smart television module, etc. In alternate embodiments, the features may be implemented on a laptop computer or other personal computer, a smartphone, other mobile phone, handheld computer, a tablet PC, or other computing device.

[0067] Thus, particular implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

WHAT IS CLAIMED IS:

1. A method for calculating a predictive measure of content performance, comprising:

receiving data relating to impressions of content;

grouping each impression based on a topic associated with each impression, including further grouping the impressions within each topic based on level of user interest in the topic;

calculating a measure of content performance based on the data;

calculating a predictive measure of content performance based on the data and user information;

calculating a calibration factor for each topic and for each level of user interest within each topic based on the measure of content performance and predictive measure of content performance; and

calculating at least one calibrated predictive measure of content performance using the calibration factor for each topic, wherein the calibrated predictive measure of content performance is further based on the level of user interest in the topic.

2. The method of Claim 1 , wherein the calibrated predictive measure of content performance for each content is used to determine which content to display to the one or more users.

3. The method of Claim 1 , wherein the content is an advertisement; and wherein the advertisement is displayed on a webpage.

4. The method of Claim 3, wherein the measure of content performance is a click-through rate (CTR) and the predictive measure of content performance is a predicted click-through rate (pCTR).

5. The method of Claim 1 , wherein calculating the calibration factor for each topic comprises comparing the difference between the measure of content performance and predictive measure of content performance.

6. The method of Claim 5, wherein the calculation comprises calculating a difference between the log odd ratios of the CTR and pCTR.

7. The method of Claim 5, wherein the calibrated predictive measure of content performance is calculated using a sigmoid function.

8. The method of Claim 7, wherein the calculation comprises summing the log odd ratio of the pCTR and the calibration factor, and taking the sigmoid function of the summed value.

9. A method for calculating a predictive measure of content performance, wherein the content is to be displayed to one or more users, comprising:

receiving data relating to the number of impressions associated with a user; the impressions relating to content views;

determining a score for each impression, the score relating to user's level of interest in a topic associated with the impression;

grouping each impression based on a topic and score associated with each impression;

calculating a measure of content performance based on the data;

calculating a calibration factor for each grouping based on the measure of content performance and predictive measure of content performance; and

calculating a calibrated predictive measure of content performance using the calibration factor for each grouping.

10. The method of Claim 9, wherein the calibrated predictive measure of content performance for each content is used to determine which content to display to the one or more users.

11. The method of Claim 9, wherein the content is an advertisement; and

wherein the advertisement is displayed on a webpage.

12. The method of Claim 1 1 , wherein the measure of content performance is a click-through rate (CTR) and the predictive measure of content performance is a predicted click-through rate (pCTR).

13. The method of Claim 9, wherein calculating the calibration factor for each topic comprises comparing the difference between the measure of content performance and predictive measure of content performance.

14. The method of Claim 13, wherein the calculation comprises calculating a difference between the log odd ratios of the CTR and pCTR.

15. The method of Claim 13 , wherein the calibrated predictive measure of content performance is calculated using a sigmoid function.

16. The method of Claim 15, wherein the calculation comprises summing the log odd ratio of the pCTR and the calibration factor, and taking the sigmoid function of the summed value.

17. A system for calculating a predictive measure of content performance, wherein the content is to be displayed to one or more users, comprising a processing circuit operable to:

receive data relating to impressions of content; group each impression based on a topic associated with each impression, including further grouping the impressions within each topic based on level of user interest in the topic;

calculate a measure of content performance based on the data; calculate a predictive measure of content performance based on the data and user information;

calculate a calibration factor for each topic and for each level of user interest within each topic based on the measure of content performance and predictive measure of content performance; and

calculate at least one calibrated predictive measure of content performance using the calibration factor for each topic, wherein the calibrated predictive measure of content performance is further based on the level of user interest in the topic.

18. The system of Claim 17, wherein the calibrated predictive measure of content performance for each content is used to determine which content to display to the one or more users.

19. The method of Claim 17, wherein the content is an advertisement; and

wherein the advertisement is displayed on a webpage.

20. The method of Claim 19, wherein the measure of content performance is a click-through rate (CTR) and the predictive measure of content performance is a predicted click-through rate (pCTR).