CN115545734A

CN115545734A - Recommendation information processing method, recommendation information sorting method and device

Info

Publication number: CN115545734A
Application number: CN202110738774.4A
Authority: CN
Inventors: 李少波
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-06-30
Filing date: 2021-06-30
Publication date: 2022-12-30

Abstract

The disclosure provides a recommendation information processing method, a recommendation information sorting method and a recommendation information sorting device. The recommendation information processing method comprises the following steps: determining a plurality of attributes from an attribute set associated with the target recommendation information as a plurality of candidate aggregation dimensions by solving an objective function of the dimension optimization model, and determining a priority of each candidate aggregation dimension; selecting a reference aggregation dimension of the target recommendation information from a plurality of candidate aggregation dimensions based at least on the priority of each candidate aggregation dimension; aggregating the estimated conversion amount and the actual conversion amount of the reference recommendation information in the reference recommendation information set under the reference aggregation dimension, which correspond to the target recommendation information, to obtain the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set before the current moment in the current delivery cycle; determining a calibration parameter based on at least the total estimated conversion and the total actual conversion; and calibrating the estimated conversion rate of the target recommendation information by using the calibration parameters.

Description

Recommendation information processing method, recommendation information sorting method and device

Technical Field

The present disclosure relates to the multimedia field, and more particularly, to a recommendation information processing method, a recommendation information ranking method, and an apparatus.

Background

In the multimedia field, multimedia platforms often recommend information, such as advertisements, news information, etc., to target groups through various means. Taking the advertisement as an example, the estimation of the advertisement conversion rate plays an important role in realizing the accurate advertisement delivery, evaluating the advertisement delivery effect and the like. Although the overall performance of the advertisement estimated Conversion Rate (PCVR) obtained by the current Conversion Rate estimation model is good, the accuracy of the advertisement estimated Conversion Rate (PCVR) for a specific industry, a specific audience, specific content and the like is not high. Therefore, after obtaining the estimated conversion rate of the advertisement, it is often necessary to calibrate the PCVR of the advertisement in combination with various attributes of the industry, audience, content, brand, and the like of the advertisement. For example, the PCVR of an advertisement may be specifically calibrated for a particular industry by aggregating placement data for associated advertisements having common attributes with the advertisement to obtain the most accurate PCVR value. How to select the best attribute from among the attributes of an advertisement as an aggregation dimension to obtain aggregated placement data of an associated advertisement, and how to implement PCVR calibration using the determined aggregation dimension to obtain the most accurate PCVR value remains to be solved.

Disclosure of Invention

In order to solve the above-mentioned problems, the present disclosure provides a recommended information processing method, a recommended information ranking method, a device and apparatus, a computer-readable storage medium, and a computer program product.

According to an aspect of an embodiment of the present disclosure, there is provided a recommendation information processing method including: determining a plurality of attributes from an attribute set associated with target recommendation information as a plurality of candidate aggregation dimensions based on historical delivery data of a predetermined recommendation information set containing the target recommendation information in a historical delivery period, and determining a priority of each candidate aggregation dimension in the plurality of candidate aggregation dimensions; selecting a reference aggregation dimension of the target recommendation information from the plurality of candidate aggregation dimensions based at least on the priority of each of the plurality of candidate aggregation dimensions; aggregating the estimated conversion amount and the actual conversion amount of the reference recommendation information in the reference recommendation information set under the reference aggregation dimension, which correspond to the target recommendation information, to obtain the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set before the current moment in the current delivery cycle; determining calibration parameters based at least on the total predicted conversion and the total actual conversion of the reference recommendation information set; and calibrating the estimated conversion rate corresponding to the click of the target recommendation information at the current moment by using the calibration parameters.

According to an example of the embodiment of the present disclosure, based on a dimension optimization model established for historical delivery data of a predetermined recommendation information set containing target recommendation information in a historical delivery period, by solving an objective function of the dimension optimization model, a plurality of attributes are determined from an attribute set associated with the target recommendation information as a plurality of candidate aggregation dimensions, and a priority of each candidate aggregation dimension in the plurality of candidate aggregation dimensions is determined.

According to an example of the embodiment of the present disclosure, the objective function of the dimension optimization model is a cumulative sum of calibration deviations of historical predicted conversion amounts of the predetermined recommendation information in the predetermined recommendation information set in the historical release period.

According to an example of the embodiment of the present disclosure, determining a plurality of attributes from the attribute set associated with the target recommendation information as a plurality of candidate aggregation dimensions by solving an objective function of the dimension optimization model, and determining a priority of each candidate aggregation dimension of the plurality of candidate aggregation dimensions includes: randomly selecting a plurality of attributes from a set of attributes associated with the target recommendation information as a plurality of initial aggregation dimensions, each of the plurality of initial aggregation dimensions having a set priority; calibrating the historical estimated conversion rate corresponding to each historical click of each piece of preset recommendation information in the preset recommendation information set in a historical release period by using the plurality of initial aggregation dimensions and the set priority; calculating the calibration deviation of the calibrated historical predicted conversion amount and the historical actual conversion amount of each piece of preset recommendation information, and counting the cumulative sum of the calibration deviations of the predicted conversion amounts of the preset recommendation information in the preset recommendation information set in the historical release period as the target function; and determining a plurality of initial aggregation dimensions which minimize the objective function as the plurality of candidate aggregation dimensions, and determining the set priority of each initial aggregation dimension in the plurality of initial aggregation dimensions which minimize the objective function as the priority of each corresponding candidate aggregation dimension.

According to an example of the embodiment of the present disclosure, calibrating, by using the plurality of initial aggregation dimensions and the set priority, a historical predicted conversion rate corresponding to each historical click of each piece of predetermined recommendation information in the set of predetermined recommendation information includes: for each predetermined recommendation information in the set of predetermined recommendation information: selecting an initial reference aggregation dimension from the plurality of initial aggregation dimensions based at least on the set priority of each initial aggregation dimension; determining a historical calibration parameter at least based on historical total estimated conversion and historical total actual conversion of the reference recommendation information set under the initial reference aggregation dimension corresponding to the preset recommendation information in a historical release period; and calibrating the historical estimated conversion rate corresponding to each historical click of the preset recommendation information in the historical release period by using the historical calibration parameters.

According to an example of the embodiment of the present disclosure, wherein selecting the reference aggregation dimension of the target recommendation information from the plurality of candidate aggregation dimensions based on at least the priority of each candidate aggregation dimension of the plurality of candidate aggregation dimensions comprises: acquiring total delivery consumption of a recommendation information set corresponding to the target recommendation information and under each candidate aggregation dimension in the plurality of candidate aggregation dimensions; and selecting the candidate aggregation dimension with the total delivery consumption of the recommendation information sets larger than a first preset threshold value and the highest priority as the reference aggregation dimension of the target recommendation information.

According to an example of the disclosed embodiment, wherein, before determining the calibration parameter based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, the recommendation information processing method further comprises: determining that the target recommendation information is in an initial release stage or a mature release stage of a current release period at the current moment based on a predetermined rule, wherein when the target recommendation information is in the initial release stage, the recommendation information processing method further comprises: acquiring a total estimated click rate and a total actual click rate of the reference recommendation information set before the current moment in the current delivery cycle, wherein when the target recommendation information is in a delivery maturity stage, the recommendation information processing method further comprises: and acquiring the estimated conversion amount, the actual conversion amount, the estimated click rate and the actual click rate of the target recommendation information before the current moment in the current release period.

According to an example of the embodiment of the present disclosure, determining that the target recommendation information is in the initial stage of release or the mature stage of release of the current release cycle at the current time based on a predetermined rule includes: determining current delivery consumption and current conversion amount generated by the target recommendation information before the current moment in the current delivery cycle; and when the current delivery consumption is less than or equal to a second preset threshold and the current conversion amount is less than or equal to a third preset threshold, determining that the target recommendation information is in the initial delivery stage, otherwise, determining that the target recommendation information is in the mature delivery stage.

According to an example of the embodiment of the present disclosure, determining a calibration parameter based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set when the target recommendation information is in an initial delivery stage comprises: determining the calibration parameter based on at least the total predicted conversion amount, the total actual conversion amount, the total predicted click amount and the total actual click amount of the reference recommendation information set.

According to an example of the embodiment of the present disclosure, wherein determining the calibration parameter based on at least the total predicted conversion amount, the total actual conversion amount, the total predicted click amount and the total actual click amount of the reference recommendation information set comprises: acquiring current release consumption of the target recommendation information generated before the current moment in the current release period, and determining an expansion coefficient based on the current release consumption; and determining the calibration parameters based on the expansion coefficient and the total predicted conversion amount, the total actual conversion amount, the total predicted click amount and the total actual click amount of the reference recommendation information set.

According to an example of the embodiment of the present disclosure, determining a calibration parameter based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set when the target recommendation information is in a release maturity stage comprises: and determining the calibration parameters at least based on the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set, and the estimated conversion amount, the actual conversion amount, the estimated click rate and the actual click rate of the target recommendation information before the current moment in the current putting cycle.

According to an example of the embodiment of the present disclosure, wherein determining the calibration parameter based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, and the predicted conversion amount, the actual conversion amount, the predicted click rate and the actual click rate of the target recommendation information before the current time in the current delivery cycle comprises: acquiring historical actual conversion amount of the target recommendation information in a historical putting period, and determining a smoothing coefficient based on the historical actual conversion amount of the target recommendation information and the time length from the current time to the current putting period; and determining the calibration parameters based on the smoothing coefficient, the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set, and the estimated conversion amount, the actual conversion amount, the estimated click rate and the actual click rate of the target recommendation information before the current moment in the current putting cycle.

According to an example of the embodiment of the present disclosure, the attribute set at least includes a home attribute subset, an audience attribute subset, and a content attribute subset of the target recommendation information, and determining a plurality of attributes from the attribute set associated with the target recommendation information as a plurality of candidate aggregation dimensions includes: determining at least one attribute from the home attribute subset, the audience attribute subset, and the content attribute subset, respectively, as a candidate aggregation dimension of the plurality of candidate aggregation dimensions.

According to another aspect of the embodiments of the present disclosure, a method for ranking recommendation information is provided, including: acquiring a calibrated estimated conversion rate of each piece of recommendation information to be released in a plurality of pieces of recommendation information to be released; calculating the estimated profit of each recommendation information to be released in the plurality of recommendation information to be released according to the calibrated estimated conversion rate of each recommendation information to be released in the plurality of recommendation information to be released; and ranking the plurality of pieces of recommendation information to be released based on the estimated profit of each piece of recommendation information to be released in the plurality of pieces of recommendation information to be released, and sequentially releasing the plurality of pieces of recommendation information to be released according to a ranking result, wherein the obtaining of the calibrated estimated conversion rate of each piece of recommendation information to be released in the plurality of recommendation information to be released comprises: the recommendation information processing method in the above aspect is used for calibrating the estimated conversion rate of each piece of recommendation information to be released in the plurality of pieces of recommendation information to be released, so as to obtain the calibrated estimated conversion rate of each piece of recommendation information to be released in the plurality of pieces of recommendation information to be released.

According to another aspect of the embodiments of the present disclosure, there is provided a recommended information processing apparatus including: a dimension determination unit configured to determine a plurality of attributes as a plurality of candidate aggregation dimensions from a set of attributes associated with target recommendation information based on historical placement data of a predetermined recommendation information set containing the target recommendation information over a historical placement period, and determine a priority of each candidate aggregation dimension of the plurality of candidate aggregation dimensions; a selection unit configured to select a reference aggregation dimension from the plurality of candidate aggregation dimensions based at least on the priority of each candidate aggregation dimension of the plurality of candidate aggregation dimensions; the aggregation unit is configured to aggregate the estimated conversion amount and the actual conversion amount of the reference recommendation information in the reference recommendation information set under the reference aggregation dimension, which correspond to the target recommendation information, to obtain a total estimated conversion amount and a total actual conversion amount of the reference recommendation information set before the current moment in the current delivery cycle; a calibration parameter determination unit configured to determine calibration parameters based on at least said total predicted conversion amount and said total actual conversion amount of said reference recommendation information set; and the calibration unit is configured to calibrate the estimated conversion rate corresponding to the click of the target recommendation information at the current moment by using the calibration parameter.

According to an example of the embodiment of the present disclosure, the dimension determining unit is configured to: randomly selecting a plurality of attributes from a set of attributes associated with the target recommendation information as a plurality of initial aggregation dimensions, each of the plurality of initial aggregation dimensions having a set priority; calibrating the historical estimated conversion rate corresponding to each historical click of each piece of preset recommendation information in the preset recommendation information set in a historical release period by using the plurality of initial aggregation dimensions and the set priority; calculating the calibration deviation of the calibrated historical predicted conversion amount and the historical actual conversion amount of each piece of preset recommendation information, and counting the cumulative sum of the calibration deviations of the predicted conversion amounts of the preset recommendation information in the preset recommendation information set in the historical release period as the target function; and determining a plurality of initial aggregation dimensions which minimize the objective function as the plurality of candidate aggregation dimensions, and determining the set priority of each initial aggregation dimension in the plurality of initial aggregation dimensions which minimize the objective function as the priority of each corresponding candidate aggregation dimension.

According to an example of the embodiment of the present disclosure, the dimension determining unit is further configured to: : for each predetermined recommendation information in the set of predetermined recommendation information: selecting an initial reference aggregation dimension from the plurality of initial aggregation dimensions based at least on the set priority of each initial aggregation dimension; determining a historical calibration parameter at least based on historical total estimated conversion and historical total actual conversion of the reference recommendation information set under the initial reference aggregation dimension corresponding to the preset recommendation information in a historical release period; and calibrating the historical estimated conversion rate corresponding to each historical click of the preset recommendation information in the historical release period by using the historical calibration parameters.

According to an example of the embodiment of the present disclosure, the selecting unit is configured to: acquiring total delivery consumption of a recommendation information set corresponding to the target recommendation information and under each candidate aggregation dimension in the plurality of candidate aggregation dimensions; and selecting the candidate aggregation dimension with the total delivery consumption of the recommendation information set larger than a first preset threshold value and the highest priority as the reference aggregation dimension of the target recommendation information.

According to an example of the embodiment of the present disclosure, the recommendation information processing apparatus further includes a delivery stage determination unit configured to: determining that the target recommendation information is in an initial release stage or a mature release stage of a current release period at the current moment based on a predetermined rule, wherein when the target recommendation information is in the initial release stage, the aggregation unit is further configured to: obtaining a total estimated click rate and a total actual click rate of the reference recommendation information set before the current time in the current delivery cycle, and wherein, when the target recommendation information is in a delivery maturity stage, the aggregating unit is further configured to: and acquiring the estimated conversion amount, the actual conversion amount, the estimated click amount and the actual click amount of the target recommendation information before the current moment in the current putting period.

According to an example of the embodiment of the present disclosure, the delivery phase determining unit is further configured to: determining current delivery consumption and current conversion amount generated by the target recommendation information before the current moment in the current delivery cycle; and when the current delivery consumption is less than or equal to a second preset threshold and the current conversion amount is less than or equal to a third preset threshold, determining that the target recommendation information is in the initial delivery stage, otherwise, determining that the target recommendation information is in the mature delivery stage.

According to an example of the embodiment of the present disclosure, when the target recommendation information is in an initial stage of delivery, the calibration parameter determining unit is configured to: determining the calibration parameter based on at least the total predicted conversion amount, the total actual conversion amount, the total predicted click amount and the total actual click amount of the reference recommendation information set.

According to an example of the embodiment of the present disclosure, when the target recommendation information is in an initial stage of delivery, the calibration parameter determining unit is further configured to: acquiring current release consumption of the target recommendation information generated before the current moment in the current release period, and determining an expansion coefficient based on the current release consumption; and determining the calibration parameters based on the expansion coefficient and the total predicted conversion amount, the total actual conversion amount, the total predicted click amount and the total actual click amount of the reference recommendation information set.

According to an example of the embodiment of the present disclosure, when the target recommendation information is in a release maturity stage, the calibration parameter determining unit is configured to: and determining the calibration parameters at least based on the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set, and the estimated conversion amount, the actual conversion amount, the estimated click rate and the actual click rate of the target recommendation information before the current moment in the current putting cycle.

According to an example of the embodiment of the present disclosure, when the target recommendation information is in a release maturity stage, the calibration parameter determining unit is further configured to: acquiring historical actual conversion amount of the target recommendation information in a historical putting period, and determining a smoothing coefficient based on the historical actual conversion amount of the target recommendation information and the time length from the current moment in the current putting period; and determining the calibration parameters based on the smoothing coefficient, the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set, and the estimated conversion amount, the actual conversion amount, the estimated click rate and the actual click rate of the target recommendation information before the current moment in the current putting cycle.

According to an example of the embodiment of the present disclosure, the attribute set at least includes a home attribute subset, an audience attribute subset and a content attribute subset of the target recommendation information, and the dimension determination unit is further configured to: determining at least one attribute from the home attribute subset, the audience attribute subset, and the content attribute subset, respectively, as a candidate aggregation dimension of the plurality of candidate aggregation dimensions.

According to another aspect of the embodiments of the present disclosure, there is provided a recommendation information ranking apparatus including: the system comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is configured to acquire a calibrated estimated conversion rate of each piece of recommendation information to be released in a plurality of pieces of recommendation information to be released; the profit prediction unit is configured to calculate the prediction profit of each recommendation information to be released in the plurality of recommendation information to be released according to the calibrated prediction conversion rate of each recommendation information to be released in the plurality of recommendation information to be released; and a sorting unit configured to sort the plurality of pieces of recommendation information to be released based on the estimated profit of each piece of recommendation information to be released in the plurality of pieces of recommendation information to be released, and sequentially release the plurality of pieces of recommendation information to be released according to a sorting result, wherein the obtaining unit is further configured to: the recommendation information processing method in the above aspect is used for calibrating the estimated conversion rate of each piece of recommendation information to be released in the plurality of pieces of recommendation information to be released, so as to obtain the calibrated estimated conversion rate of each piece of recommendation information to be released in the plurality of pieces of recommendation information to be released.

According to another aspect of the embodiments of the present disclosure, there is provided a recommendation information processing apparatus including: one or more processors; and one or more memories, wherein the memories have computer-readable code stored therein, which, when executed by the one or more processors, causes the one or more processors to perform a method as set forth in any of the above aspects of the disclosure.

According to another aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer-readable instructions, which, when executed by a processor, cause the processor to perform the method according to any one of the above aspects of the present disclosure.

According to another aspect of embodiments of the present disclosure, there is provided a computer program product comprising computer readable instructions which, when executed by a processor, cause the processor to perform the method according to any one of the above aspects of the present disclosure.

By using the recommendation information processing method, the recommendation information sorting device, the computer readable storage medium and the computer program product according to the aspects of the disclosure, based on a dimension optimization model established for historical delivery data of a predetermined recommendation information set containing target recommendation information in a historical delivery period, a plurality of candidate aggregation dimensions capable of realizing optimal PCVR calibration and the priority of each candidate aggregation dimension can be determined by solving an objective function of the dimension optimization model, so that accurate PCVR calibration of the target recommendation information can be realized by using the plurality of candidate aggregation dimensions and the priority thereof, and the accuracy of the PCVR calibration is greatly improved; in addition, calibration can be performed in a targeted manner by analyzing the characteristics of the estimated conversion rate of the recommendation information of a specific industry, a specific audience, specific content and the like, so that recommendation information release, release cost control, release effect prediction and the like can be performed more accurately; in addition, the release profit of the recommendation information can be estimated based on the estimated conversion rate of the calibration, and the recommendation information to be released is sequenced based on the estimated profit, so that the recommendation information with high estimated profit can be released preferentially, and the profit maximization is realized.

Drawings

The above and other objects, features and advantages of the embodiments of the present disclosure will become more apparent by describing in more detail the embodiments of the present disclosure with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the disclosure, and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the principles of the disclosure and not to limit the disclosure. In the drawings, like reference numbers generally represent like parts or steps.

FIG. 1 shows a schematic diagram of an application scenario according to an embodiment of the present disclosure;

FIG. 2 shows a flow diagram of a recommendation information processing method according to an embodiment of the present disclosure;

FIG. 3 shows a flow diagram for solving a dimensional optimization model, according to an example of an embodiment of the present disclosure;

FIG. 4 illustrates a statistical average of calibrated estimated conversion deviation for a predetermined set of ads in an ad dimension, according to an example of the present disclosure;

FIG. 5 illustrates a distribution scale of calibrated predicted conversion bias for a predetermined set of ads in an ad dimension, according to an example of the present disclosure;

FIG. 6 illustrates a statistical average of calibrated predicted conversion bias for a set of predetermined ads in an advertiser dimension, according to an example of an embodiment of the present disclosure;

FIG. 7 illustrates a distribution scale of calibrated estimated conversion bias for a set of predetermined advertisements in an advertiser dimension according to an example of an embodiment of the present disclosure;

FIG. 8 shows a flow diagram of a recommendation information ranking method in accordance with an embodiment of the present disclosure;

fig. 9 shows a schematic configuration diagram of a recommended information processing apparatus according to an embodiment of the present disclosure;

fig. 10 is a schematic structural diagram illustrating a recommendation information ranking apparatus according to an embodiment of the present disclosure;

FIG. 11 shows a schematic diagram of an architecture of an exemplary computing device, in accordance with embodiments of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. It is to be understood that the described embodiments are merely exemplary of some, and not all, of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without any inventive step, are intended to be within the scope of the present disclosure.

In the embodiment of the present disclosure, the recommendation information may be information that is pushed to the terminal device for display in a manner of a picture, a text, a video, or any combination thereof, for example. For example, the recommendation information may be any content such as an advertisement, recommendation information, an article/video promotion, and the like, which is not specifically limited in this disclosure. In the following embodiments of the present disclosure, the advertisement will be mainly described as an example of the recommendation information, but it should be understood by those skilled in the art that the contents of each part described by taking the advertisement as an example are also applicable to other recommendation information.

Various definitions that may be used in embodiments of the present disclosure are described with advertisements as examples of recommendation information. In the embodiment of the present disclosure, an advertiser generally refers to a party who invests in advertising, an advertising platform refers to a party who uses a platform or technology of the advertiser to help the advertiser to perform advertising, analysis, prediction, and the like, and an audience refers to a user group who watches advertisements. After the advertisement is delivered through the advertisement platform, the following processes are usually performed: first, the user sees the advertisement, a process called exposure; after exposure, an interested user may click to browse the goods contained in the advertisement, for example, jump to a page of goods by clicking a link of the goods in the advertisement, which is called clicking; after jumping to the goods page by clicking, the user may purchase the product in the goods page, or download an application installed in the goods page, etc., this action is called conversion. In the present disclosure, a ratio of the advertisement Click Through amount to the advertisement exposure amount may be referred to as an advertisement Click Through Rate (CTR), and a ratio of the advertisement Conversion amount to the advertisement Click Through amount may be referred to as an advertisement Conversion Rate (CVR).

In order to achieve the purposes of accurate delivery of an advertisement, evaluation of delivery effect of an advertisement, and the like, a model is usually used to estimate a conversion rate after delivery of an advertisement, so as to obtain an estimated conversion rate (PCVR) of the advertisement. With the development of Deep learning, most of the current models for predicting advertisement conversion rate are based on neural network, such as Deep Crossing (Deep Crossing) model, commodity-based neural network (PNN), factoring Machine (FM), and the like. For example, each click of the advertisement corresponds to a conversion with a certain probability, that is, corresponds to a certain conversion rate, and the model such as PNN, FM, or the like can be used to predict the estimated conversion rate corresponding to each click of the advertisement. However, although the estimated conversion rate of the advertisement obtained by using these models has good overall performance, the accuracy of the estimated conversion rate of the advertisement for a specific industry, a specific audience, specific content, and the like is not high, and therefore, the estimated conversion rate of the advertisement needs to be calibrated in a targeted manner.

Typically, the generated placement data for the targeted ad to be calibrated may be selected for PCVR calibration, but often during the initial placement phase of the targeted ad, sufficient placement data has not yet been generated for PCVR calibration, at which point useful information may be obtained by aggregating placement data for ads having some of the same attributes as the targeted ad (which may be referred to as correlated ads) for PCVR calibration of the targeted ad. For example, if the placement data of the relevant advertisements having the same advertiser as the target advertisement may have a certain degree of similarity with the target advertisement, the PCVR of the target advertisement may be calibrated by using the placement data of the relevant advertisements, that is, the placement data of all advertisements under the advertiser may be aggregated by using the same advertiser as an aggregation dimension, and the PCVR of the target advertisement may be calibrated by using the aggregated data.

However, the targeted advertisement has a large number of attributes, which may include, for example, attributes of advertisement affiliation such as advertiser, group of advertiser, company of advertiser, industry of advertisement, region of advertisement, etc., attributes of advertisement audience such as gender, age, region, occupation, behavioral interest of advertisement targeted crowd, attributes of advertisement content such as commodity name, commodity type, commodity industry type, advertisement title information, advertisement picture information, etc. of advertisement, and an optimal aggregation dimension for performing PCVR calibration needs to be selected from a large set of these attributes. In addition, after the accurate estimated conversion rate of the advertisements is obtained through calibration, the estimated revenue of each advertisement needs to be calculated to determine the putting sequence of the advertisements. Therefore, the disclosure provides a recommendation information processing method, a recommendation information sorting method, and corresponding devices and equipment.

First, application scenarios of a recommendation information processing method, a recommendation information ranking method, and corresponding apparatuses and the like according to an embodiment of the present disclosure are described with reference to fig. 1. Fig. 1 shows a schematic diagram of an application scenario 100 according to an embodiment of the present disclosure, in which a server 110 and a plurality of terminals 120 are schematically shown. As shown in fig. 1, recommendation information such as advertisements may be placed on a plurality of terminals 120 through a server 110 for display. When users of multiple terminals 120 browse advertisements, various data such as exposure, click through, conversion, etc., may be generated, collectively referred to herein as impression data. The recommendation information processing method, the recommendation information ranking method, and the corresponding devices according to the embodiments of the disclosure may be mounted on the server 110 to calibrate the estimated conversion rates of the recommendation information delivered to the plurality of terminals 120, and determine the order of the next delivery of each recommendation information based on the calibrated estimated conversion rates.

The server 110 may be an independent server for performing analysis processing on the recommendation information, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a positioning service, and a big data and artificial intelligence platform, which is not limited in this disclosure. Each of the plurality of terminals 120 may be a stationary terminal such as a desktop computer, a mobile terminal such as a smart phone, a tablet computer, a portable computer, a handheld device, a personal digital assistant, a smart wearable device, and the like, or any combination thereof, which are not specifically limited by the embodiments of the present disclosure.

A recommendation information processing method according to an embodiment of the present disclosure is described below with reference to fig. 2. FIG. 2 shows a flow diagram of a recommendation information processing method 200 according to an embodiment of the present disclosure. As shown in fig. 2, in step S210, based on a dimension optimization model established for historical placement data of a predetermined recommendation information set containing target recommendation information in a historical placement period, by solving an objective function of the dimension optimization model, a plurality of attributes are determined as a plurality of candidate aggregation dimensions from an attribute set associated with the target recommendation information, and a priority of each candidate aggregation dimension in the plurality of candidate aggregation dimensions is determined. The target recommendation information may be any recommendation information to be subjected to PCVR calibration, such as an advertisement, recommendation information, an article/video promotion, and the like, which is not specifically limited in this disclosure.

According to an example of an embodiment of the present disclosure, the set of attributes associated with the target recommendation information may include at least a home attribute subset, an audience attribute subset, and a content attribute subset. As the name implies, the affiliation attribute subset includes a plurality of attributes related to affiliation of the recommendation information, the audience attribute subset includes a plurality of attributes related to an audience of the recommendation information, and the content attribute subset includes a plurality of attributes related to content of the recommendation information. The following describes each attribute subset by taking an advertisement as an example.

The attribute subset may include attributes such as an advertiser, a group to which the advertiser belongs, a company to which the advertiser belongs, an industry to which the advertisement belongs, a sub-industry to which the advertisement belongs, a category to which the advertisement belongs, a region to which the advertisement belongs, an operation project group to which the advertisement belongs, an operation team to which the advertisement belongs, a scale level of the advertiser, a scale level of the group to which the advertiser belongs, and the like.

Audience attribute subsets may include, for example, ad targeting attributes such as gender, age, geography, occupation, education level, income level, etc., of ad targeting demographic; behavioral interest targeting attributes of advertisements, such as user application installation behavior, advertisement interaction behavior, e-commerce browsing behavior, search click behavior, information browsing behavior, industry categories of interest, and the like; such as community number fans, application installation demographics, commodity purchasing customers, similar crowd expansion, relationship chain expansion, etc.

The content attribute subset may include attributes such as advertisement title information, advertisement picture information, advertisement video information, commodity library ID (Identity), commodity industry type, advertisement creative specification, background commodity ID, customer commodity name, commodity type, commodity core product word, creative ID, commodity custom tag, commodity external ID, and the like, for example.

The content of the attribute subset including the attribute subset of the target recommendation information, the attribute subset of the audience, and the attribute subset of the content has been described above by taking the advertisement as an example, but the embodiments of the present disclosure are not limited thereto, and the attribute set associated with the target recommendation information may also include more other attributes, depending on the specific category of the recommendation information or depending on the predetermined attribute filtering rule.

In step S210, based on historical delivery data of a predetermined recommendation information set including target recommendation information in a historical delivery period, a plurality of attributes are determined as a plurality of candidate aggregation dimensions from the attribute set associated with the target recommendation information, so that information for PCVR calibration of the target recommendation information can be acquired by aggregating delivery data of recommendation information in each candidate aggregation dimension.

According to an example of the embodiment of the disclosure, based on a dimension optimization model established for historical delivery data of a predetermined recommendation information set containing target recommendation information in a historical delivery period, by solving an objective function of the dimension optimization model, the optimal attributes from an attribute set of the target recommendation information can be determined as a plurality of candidate aggregation dimensions.

The predetermined recommendation information set is a set of a plurality of recommendation information to be calibrated (hereinafter may be referred to as predetermined recommendation information), and the target recommendation information may be any recommendation information in the predetermined recommendation information set. The predetermined recommendation information in the predetermined recommendation information set may be determined randomly, for example; or may belong to a common advertiser, that is, the predetermined recommendation information set may be a set of all advertisements under a certain advertiser, that is, the recommendation information processing method 200 according to the embodiment of the present disclosure may perform PCVR calibration for all advertisements under a common advertiser; or may have a common industry orientation, that is, the predetermined recommendation information set may be a set of all advertisements under a certain industry orientation, that is, the recommendation information processing method 200 according to the embodiment of the disclosure may perform PCVR calibration on all advertisements under a common industry orientation, and so on, but the embodiment of the disclosure is not limited thereto, and the predetermined recommendation information set may also be a set of recommendation information to be calibrated selected according to any other rule.

Furthermore, since the aggregated delivery data in different aggregation dimensions has different effects on the accuracy of the PCVR calibration of the target recommendation information, in this step, the priority of each of the multiple candidate aggregation dimensions may also be determined. For example, the priority of the PCVR calibration of the target recommendation information is determined according to the influence of different aggregation dimensions on the accuracy thereof. In embodiments of the present disclosure, a dimension optimization model may be constructed, and a plurality of candidate aggregation dimensions and a priority of each candidate aggregation dimension may be determined by solving an objective function of the dimension optimization model, as will be described in further detail below.

In step S220, a reference aggregation dimension of the target recommendation information is selected from the plurality of candidate aggregation dimensions for PCVR calibration of the target recommendation information based at least on the priority of each of the plurality of candidate aggregation dimensions determined in step S210. For example, the candidate aggregation dimension with the highest priority may be selected as the reference aggregation dimension. However, in some cases, for example, in the initial stage of delivery of recommendation information, the delivery data of recommendation information in some candidate aggregation dimensions may not be sufficient yet, and the data obtained by aggregating the delivery data of recommendation information in the candidate aggregation dimensions does not have a valid reference value, that is, the aggregated data is invalid. If the aggregated data for the highest priority candidate aggregated dimension of the multiple candidate aggregated dimensions is invalid, PCVR calibration of the target recommendation information using the aggregated data for the highest priority candidate aggregated dimension may be inaccurate. Therefore, in selecting the reference aggregation dimension of the target recommendation information, in addition to the priority, the validity of the aggregation data in each candidate aggregation dimension corresponding to the target recommendation information should be considered.

According to an example of the embodiment of the present disclosure, for a recommendation information set in each candidate aggregation dimension, that is, a set formed by all recommendation information in each candidate aggregation dimension, validity of aggregation data in the candidate aggregation dimension may be determined based on total delivery consumption of the recommendation information set. The release consumption refers to the fee required to be charged to the customer for releasing the recommendation information in a certain amount or for a certain time, and can reflect the click rate or the conversion rate of the recommendation information, so that whether the recommendation information is exposed enough can be reflected, and the validity of the aggregated data of the recommendation information set can be judged. According to an example of an embodiment of the present disclosure, when total delivery consumption of a recommendation information set in a certain candidate aggregation dimension is greater than a first predetermined threshold, it may be determined that aggregated data of the recommendation information set is valid; otherwise, determining that the aggregation data of the recommendation information set is invalid. The first predetermined threshold may be set according to practical situations, and is not particularly limited by the embodiments of the present disclosure.

Taking an advertisement as an example, the advertisement delivery consumption refers to the fee that the advertiser needs to be charged for the delivery of an advertisement for a certain amount or a certain time, and the first predetermined threshold may be set according to the target conversion cost (target _ cpa) of the advertiser, wherein the target conversion cost of the advertiser is the fee that is required for each conversion of a single advertisement desired by the advertiser. For example, when the total consumption of ads in a certain ad set is greater than 4 times the target conversion cost (4 × target _cpa), it may be determined that the aggregated data for that ad set is valid; otherwise, determining that the aggregated data for the set of advertisements is invalid.

Thus, according to an example of an embodiment of the present disclosure, selecting a reference aggregation dimension of the target recommendation information from the plurality of candidate aggregation dimensions based at least on the priority of each of the plurality of candidate aggregation dimensions may comprise: acquiring total delivery consumption of a recommendation information set corresponding to the target recommendation information and under each candidate aggregation dimension in the multiple candidate aggregation dimensions; and determining a candidate aggregation dimension with the highest priority and the total delivery consumption of the recommendation information sets larger than a first preset threshold value in the plurality of candidate aggregation dimensions as a reference aggregation dimension. That is, the candidate aggregation dimension with the highest priority among the plurality of candidate aggregation dimensions and the corresponding aggregation data being valid is determined as the reference aggregation dimension to ensure that the PCVR calibration can be performed on the target recommendation information with the most relevant and sufficient aggregation data.

After the reference aggregation dimension is determined, in step S230, a total predicted conversion amount (PCVR _ valid) and a total actual conversion amount (Conv _ valid) of the reference recommendation information set before the current time in the current delivery cycle are obtained by aggregating the predicted conversion amount and the actual conversion amount of the reference recommendation information in the reference recommendation information set under the reference aggregation dimension corresponding to the target recommendation information. Here, a set formed by referring to all recommendation information in the aggregation dimension is referred to as a reference recommendation information set. The release period may be any time period, such as 12h, 24h (i.e. a complete day), a week, etc., which may be set according to actual needs, and the embodiment of the disclosure does not specifically limit this. For example, for an advertisement, one day (24 h) is generally taken as one delivery period. The current time is the time when the PCVR calibration is to be performed on the target recommendation information. For example, for a click of an advertisement at the current time, after a predicted conversion rate corresponding to the click is predicted by using a model such as PNN, FM, or the like, a total predicted conversion amount and a total actual conversion amount generated by the reference recommendation information set before the current time in the current delivery cycle may be obtained by aggregating the predicted conversion amount and the actual conversion amount of all the reference recommendation information in the reference recommendation information set for calibrating the predicted conversion rate.

In step S240, a calibration parameter is determined based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, so as to calibrate the predicted conversion rate corresponding to the click of the target recommendation information at the current time in step S250 by using the calibration parameter.

Generally, in an initial stage of delivering the target recommendation information, delivery data of the target recommendation information itself is not sufficient, and at this time, a quotient of a total actual conversion amount Conv _ valid and a total estimated conversion rate PCVR _ valid of the reference recommendation information set may be directly determined as a calibration parameter, that is, the calibration parameter f is:

f＝Conv_valid/PCVR_valid (1)

with the progress of the delivery, the click quantity and the conversion quantity of the target recommendation information per se are continuously increased, that is, more delivery data are generated, and at the moment, the target recommendation information can be called to enter the delivery mature stage. In the release maturity stage, the release data of the target recommendation information can also be used for carrying out PCVR calibration on the target recommendation information, and due to better correlation, the release data of the target recommendation information plays a more important role in the PCVR calibration process compared with the data of the reference recommendation information set, so that the accuracy of the PCVR calibration is further improved. In addition, in some cases, for example, for CPM (Cost per thousand) advertisements, the Predicted Click Through Rate (PCTR) also has an effect on the Predicted revenue of the advertisement, so when the PCVR calibration is performed on the advertisement, if the Predicted Click Through Rate of the advertisement can be considered at the same time, the accuracy of the Predicted revenue determined by using the calibrated PCVR can be effectively improved.

Therefore, according to an example of the embodiment of the present disclosure, before step S240, the recommendation information processing method 200 may further include determining that the target recommendation information is in an initial stage of delivery or a mature stage of delivery of the current delivery cycle at the current moment based on a predetermined rule. For example, whether the target recommendation information is in the initial stage of delivery or the mature stage of delivery may be determined based on the current delivery consumption and the current conversion amount of the target recommendation information prior to the current time within the current delivery cycle. Specifically, total release consumption and total conversion amount (which may be referred to as current release consumption and current conversion amount, respectively) generated by the target recommendation information before the current time in the current release period may be statistically determined, and when the current release consumption is less than or equal to a second predetermined threshold and the current conversion amount is less than or equal to a third predetermined threshold, the target recommendation information is determined to be in the release initial stage, otherwise, the target recommendation information is determined to be in the release mature stage. The second predetermined threshold and the third predetermined threshold may be determined according to actual situations, which is not specifically limited in the embodiment of the present disclosure. For example, taking an ad as an example, the second predetermined threshold may be 2 times the targeted conversion cost (2 × target _cpa), which, as noted above, is the cost required for each conversion of a single ad desired by the advertiser; the third predetermined threshold may be, for example, 2. That is, when the current delivery consumption is less than or equal to 2 × target _cpaand the current conversion amount is less than or equal to 2, it may be determined that the target recommendation information is in the initial delivery stage, otherwise, it may be determined that the target recommendation information is in the mature delivery stage.

According to an example of the embodiment of the present disclosure, when the target recommendation information is in the initial stage of delivery, the recommendation information processing method 200 may further include obtaining a total predicted conversion amount (PCVR _ valid), a total actual conversion amount (Conv _ valid), a total predicted click amount (PCTR _ valid), and a total actual click amount (ClickNum _ valid) of the reference recommendation information set before the current time within the current delivery cycle, and may determine the calibration parameter based on at least the obtained total predicted conversion amount (PCVR _ valid), total actual conversion amount (Conv _ valid), total predicted click amount (PCTR _ valid), and total actual click amount (ClickNum _ valid) of the reference recommendation information set. At this time, the calibration parameter f may be determined as:

f＝Conv_valid/(PCVR_valid*PCTR_valid/ClickNum_valid) (2)

when the target recommendation information at the initial stage of delivery is subjected to the PCVR calibration by using the calibration parameter of the above formula, in order to avoid the influence of the PCVR calibration process on the current delivery consumption of the target recommendation information, an expansion coefficient (coef) determined based on the delivery consumption of the target recommendation information may be further introduced, and then the calibration parameter is determined based on the expansion coefficient (coef) and a total estimated conversion amount (PCVR _ valid), a total actual conversion amount (Conv _ valid), a total estimated click amount (PCTR _ valid) and a total actual click amount (ClickNum _ valid) of the reference recommendation information set. At this time, the calibration parameter f may be further determined as:

f＝coef*Conv_valid/(PCVR_valid* PCTR_valid/ClickNum_valid) (3)

according to an example of the embodiment of the present disclosure, when the target recommendation information is in the delivery maturity stage, the recommendation information processing method 200 may further include: and acquiring the predicted conversion amount (sum _ pcvr), the actual conversion amount (conversion _ num), the predicted click amount (sum _ pctr) and the actual click amount (clicknum) of the target recommendation information before the current moment in the current release period. At this time, the calibration parameters may be determined based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, and the predicted conversion amount (sum _ pcvr), the actual conversion amount (conversion _ num), the predicted click amount (sum _ pctr) and the actual click amount (clicknum) of the target recommendation information before the current time in the current delivery cycle. As mentioned above, in the calibration of the PCVR at the release maturity stage, the release data of the target recommendation information itself should play a more important role, and the total predicted conversion and the total actual conversion of the reference recommendation information set can be used as auxiliary factors. For example, the factor determined based on the total predicted amount of conversion and the total actual amount of conversion of the reference recommendation information set may be referred to as a historical PCVR calibration factor (history _ PCVR _ bias _ factor). At this time, the calibration parameter f may be determined as:

f＝(history_pcvr_bias_factor*conversion_num)/(sum_pcvr*sum_pctr/clicknum) (4)

in addition, in some cases, the delivery data of the target recommendation information may be relatively small even at the delivery maturity stage, which may result in a calibration parameter determined according to equation (4) above being higher or lower, i.e., a data anomaly may occur. At this time, such data abnormality may be smoothed by introducing a smoothing coefficient (smooth _ base). Specifically, the historical actual conversion amount of the target recommendation information in the historical release period may be obtained, and the smoothing coefficient may be determined based on the historical actual conversion amount of the target recommendation information and the time length from the current time in the current release period. For example, the smoothing coefficient smooth _ base may be determined by:

wherein sum _ conversion _ yesterday represents the actual converted amount of the target recommendation information in the last delivery cycle, for example, the total actual converted amount in yesterday; h represents the time length (the unit is hour) from the beginning of the current putting period to the current moment of the target recommendation information; ceil () function represents rounding up; the min () function represents taking the minimum value. That is, the larger the historical actual conversion amount is, the smaller the time length H is, and the larger the smoothing coefficient smooth _ base is.

After the smoothing coefficient is determined, the calibration parameters may be determined based on the smoothing coefficient, the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, and the predicted conversion amount (sum _ pcvr), the actual conversion amount (conversion _ num), the predicted click amount (sum _ pctr) and the actual click amount (clicknum) of the target recommendation information before the current time in the current delivery cycle. For example, the calibration parameter f may be further determined as:

f＝(history_pcvr_bias_factor*conversion_num+smooth_base)/(sum_pcvr*

sum_pctr/clicknum+smooth_base) (6)

the history _ PCVR _ bias _ factor is a historical PCVR calibration factor determined based on the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, and a specific determination manner of the history _ PCVR calibration factor is described in detail below.

After the calibration parameter is determined, in step S250, the calibration parameter is used to calibrate the estimated conversion rate corresponding to the click of the target recommendation information at the current time. Specifically, at any current time, after the estimated conversion rate corresponding to the click at the current time obtained using a conversion rate estimation model such as PNN, FM, or the like. The estimated conversion may be calibrated by multiplying the estimated conversion by a calibration parameter. Alternatively, the calibration parameters may be used as weighting factors for a conversion estimation model such as PNN, FM, etc. to directly generate a calibrated estimated conversion.

Next, a process of determining a plurality of candidate aggregation dimensions and priorities of the respective candidate aggregations by using the dimension optimization model in the above step S210 is described with reference to fig. 3. Fig. 3 shows a flow diagram 300 for solving a dimensional optimization model according to an example of an embodiment of the present disclosure. As mentioned before, in step S210, a plurality of candidate aggregation dimensions and a priority of each candidate aggregation dimension are determined based on a calibration analysis of historical predicted conversion rates of a predetermined recommendation information set containing target recommendation information during a historical delivery period. Assuming that there are n pieces of recommendation information including the target recommendation information in the predetermined recommendation information set, a dimension optimization model may be established as shown in the following equation (7), for example:

wherein i is an index of recommendation information in the recommendation information set, i =1 \ 8230 \8230; \8230n; sumPcvr _i The sum of calibrated PCVRs (which can be called estimated conversion amount after calibration) corresponding to valid clicks of the ith recommendation information in a historical release period is obtained, wherein the valid clicks refer to clicks after error clicks, cheat clicks and the like are removed; convNum _i The total conversion amount of the ith recommendation information in the historical release period is obtained; j is the index of each effective click of each piece of recommendation information in the historical release period, j =1 \ 8230 \ 8230, m is the total effective click quantity of the recommendation information in the historical release period; pcvr _j The PCVR corresponding to the j-th effective click; x1, x2 and x3 are candidate aggregation dimensions requiring solution; f (x 1, x2, x 3) is a calibration parameter determined based on the candidate aggregate dimensions (x 1, x2, x 3).

The above equations (7) - (8) define the dimension optimization model, i.e.That is, the objective of the dimension optimization model is to solve such that the objective function

The smallest aggregation dimension (x 1, x2, x 3) is taken as a candidate aggregation dimension, and the priority of the obtained candidate aggregation dimension is solved at the same time. It should be understood that although only three candidate aggregation dimensions (x 1, x2, x 3) to be solved are listed in equation (8), this is by way of example only, and in the embodiments of the present disclosure, more or fewer candidate aggregation dimensions to be solved may be provided.

The specific steps for solving the objective functions (7) and (8) are described below with reference to fig. 3.

In step S310, a plurality of attributes are randomly selected from the set of attributes associated with the target recommendation information as a plurality of initial aggregation dimensions, each of the plurality of initial aggregation dimensions having a set priority. According to an example of an embodiment of the present disclosure, at least one initial aggregation dimension may be selected from a home attribute subset, an audience attribute subset, and a content attribute subset, respectively. For example, an initial aggregation dimension x1 may be selected from the home attribute subset, e.g., x1 may be an advertiser; selecting an initial aggregation dimension x2 from the subset of audience attributes, e.g., x2 may be an age of the ad targeting population; an initial aggregation dimension x3 is selected from the subset of content attributes, for example x3 may be an advertisement commodity type. Also, the priority order of x1, x2, and x3 may be set to x1> x2> x3, i.e., x1 has the highest priority, x3 has the lowest priority, and x2 has the middle priority.

In step S320, a historical estimated conversion rate corresponding to each historical click of each piece of predetermined recommendation information in the set of predetermined recommendation information in the historical release period is calibrated by using the plurality of initial aggregation dimensions and the corresponding set priorities.

Specifically, for each predetermined recommendation information, an initial reference aggregation dimension is selected from a plurality of initial aggregation dimensions based at least on the set priority of each initial aggregation dimension. As described above, in selecting the initial reference aggregation dimension of the predetermined recommendation information, in addition to the priority, the validity of the aggregation data in each initial candidate aggregation dimension should be considered, and the validity of the aggregation data thereof may be judged by the total delivery consumption of the recommendation information set. According to an example of the embodiment of the present disclosure, total delivery consumption of a recommendation information set under a plurality of initial aggregation dimensions corresponding to predetermined recommendation information may be first obtained, and an initial candidate aggregation dimension, in which the total delivery consumption is greater than a first predetermined threshold and a priority is highest, is determined as an initial reference aggregation dimension of the predetermined recommendation information. The first predetermined threshold may be set according to an actual situation, and may be, for example, 4 × target _cpa, which is not specifically limited by the embodiment of the present disclosure. For example, assuming that the aggregation data in the initial aggregation dimensions x1, x2, and x3 are all valid for the predetermined recommendation information, the initial aggregation dimension x1 is selected as the reference aggregation dimension of the predetermined recommendation information.

And then, for the preset recommendation information, calibrating the historical predicted conversion rate corresponding to each historical click of the preset recommendation information at least based on the historical total predicted conversion amount and the historical total actual conversion amount of the reference recommendation information set in the initial reference aggregation dimension corresponding to the preset recommendation information in the historical release period. For example, when it has been determined that the reference aggregation dimension of the predetermined recommendation information is x1, the historical total estimated conversion and the historical total actual conversion of the reference recommendation information set at x1 in the historical delivery period may be statistically determined, for example, when the delivery period is one day, that is, the historical total estimated conversion and the historical total actual conversion of the reference recommendation information set yesterday are statistically determined. Then, for example, (2) - (4), (6) above may be employed to calculate calibration parameters to calibrate the historical predicted conversion rate for each historical click of the predetermined recommendation information.

Similarly, the historical predicted conversion rate for each of the predetermined set of recommendation information is calibrated.

Thereafter, in step S330, a deviation of the calibrated historical predicted conversion amount from the historical actual conversion amount (which may be referred to as a calibration deviation) for each predetermined recommendation information is calculated. Wherein each recommendation informationThe calibrated historical predicted conversion amount is the sum of calibrated historical predicted conversion rates corresponding to effective historical clicks of the recommendation information in a historical release period. Then, the cumulative sum of the calibration deviations of each recommendation information in the predetermined recommendation information set, i.e., the objective function in the above formula (7), is counted

In step S340, a plurality of initial aggregation dimensions that minimize the objective function are determined as a plurality of candidate aggregation dimensions, and the set priority of each of the plurality of initial aggregation dimensions is determined as the priority of each corresponding candidate aggregation dimension. That is, after (x 1, x2, x 3) is obtained as a plurality of candidate aggregation dimensions through the solution in the above steps, the set priority of x1, x2, x3 (for example, x1> x2> x 3) is the priority of the plurality of candidate aggregation dimensions.

It can be seen that, in step S320, for each piece of predetermined recommendation information, an initial reference aggregation dimension is determined, and then historical delivery data under the initial reference aggregation dimension is statistically determined for use in the next calibration process. However, in practical situations, parallel computing processing is often employed to greatly increase the computing speed, benefiting from the rapid development of computer technology. Therefore, on one hand, all the delivery data in each initial aggregation dimension can be synchronously and parallelly aggregated and counted, and on the other hand, the total delivery consumption of the recommendation information sets in a plurality of initial aggregation dimensions corresponding to each piece of predetermined recommendation information can be synchronously and parallelly determined so as to determine the initial reference aggregation dimension of each piece of predetermined recommendation information. Then, for any predetermined recommended information, when the initial reference aggregation dimension is determined, the aggregated data in the corresponding initial reference aggregation dimension can be quickly acquired for the next calibration processing.

Therefore, when actually solving the objective function of the dimension optimization model, for the randomly selected initial aggregation dimensions x1, x2, x3, the data of the historical delivery period can be processed according to the following example steps 1-7. In the following example steps 1 to 7, the recommendation information is described as an advertisement, and the delivery cycle is described as 1 day (24 h). That is, for any predefined advertisement to be calibrated within the predefined advertisement set, a plurality of candidate aggregated dimensions and corresponding priorities for PCVR calibration of the predefined advertisement for a new delivery period (e.g. today) may be determined by solving the objective functions (7) - (8) of the dimension optimization model based on historical delivery data of the predefined advertisement set for the historical delivery period (e.g. yesterday).

Specifically, for any historical calibration time within the historical placement period, and for each scheduled advertisement to be calibrated in the scheduled advertisement set, the historical calibration parameters for calibrating the historical PCVR corresponding to the historical click of the scheduled advertisement at the historical calibration time may be calculated through the following steps.

1. The polymerization is performed for an initial polymerization dimension x 1:

respectively counting the total conversion Conv _ x1_ hour of each advertisement in the last 1 hour from any historical calibration moment in the historical release period under different values of x1 _T PCVR sum PCVR _ x1_ hour corresponding to total valid clicks _T Total click count ClickNum _ x1_ hour _T Sum of PCTR corresponding to total effective exposure PCTR _ x1_ hour _T And the total conversion Conv _ x1_ day, the sum PCVR corresponding to the total effective clicks PCVR _ x1_ day, the total click number ClickNum _ x1_ day and the sum PCTR corresponding to the total effective exposure PCTR _ x1_ day all day long.

With x ₁ For the advertiser example, the placement data Conv _ x1_ hour of the latest 1 hour at the historical calibration time of all the advertisements under different advertisers are counted respectively _T 、PCVR_x1_hour _T 、ClickNum_x1_hour _T 、PCTR_x1_hour _T And the day-to-day delivery data Conv _ x1_ day, PCVR _ x1_ day, clickNum _ x1_ day, PCTR _ x1_ day.

The daily delivery data Conv _ x1_ day, PCVR _ x1_ day, clickNum _ x1_ day, PCTR _ x1_ day refer to total delivery data that have been generated at any historical calibration time within a historical delivery period. Since the put data at different times before the historical calibration time have different effects on the PCVR calibration at the historical calibration time, the time decay strategy shown below is adopted when counting the data all day:

wherein, T is any historical calibration time within the historical release period, for example, may be any calibration time of yesterday; t is any time from the beginning of the historical release period to the historical calibration time, for example, when the historical release period is yesterday and the release is started from yesterday zero, T =1 \8230; T; conv _ x1_ hour _t Is the total ad conversion from time t-1 to time t; PCVR _ x1_ hour _t The sum of PCVRs corresponding to the total effective clicks of the advertisements from the time t-1 to the time t; clicknum _ x1_ hour _t Is the total advertisement click rate between time t-1 and time t; PCTR _ x1_ hour _t Is the sum of the PCVRs corresponding to the total effective exposure of the advertisement between the time t-1 and the time t; λ is a time attenuation coefficient, and may be, for example, 0.05.

In each case obtaining x ₁ After the last 1 hour of historical calibration time data and all day data at different values of (e.g., different advertisers), for any x ₁ Value of (2) x11 (e.g. x) ₁ X11 represents advertiser x 11) from which aggregated data at x11 may be determined. For example, aggregated data at x11 may be determined according to the following rules:

that is, if the aggregated data of the last 1 hour of the historical calibration time is sufficient, the aggregated data of the last 1 hour is selected as the aggregated data at x 11; otherwise, select the data all day as aggregated data at x 11. Here, the rule for determining whether the aggregated data of the last 1 hour is sufficient is similar to the principle described above for determining whether the aggregated data is valid, that is, within the last 1 hour of the historical calibration time, if the total impression consumption of the advertisement set at x11 is greater than a first predetermined threshold (e.g., 4 × target _cpa), the aggregated data of the last 1 hour may be considered to be sufficient; otherwise, the polymerization data of the last 1 hour was considered insufficient. The reason why the processing in the above equation (10) is performed is that, at the time of PCVR calibration, theoretically, the correlation of the shot data closer to the calibration time is better, and therefore it is desirable to select the aggregation data closer to the calibration time, for example, the aggregation data of the latest 1 hour; however, if the aggregated data for the last 1 hour is insufficient, it may instead result in a decrease in accuracy of the PCVR calibration, and therefore, more sufficient all-day data is selected as the final aggregated data at this time.

2. The polymerization is performed for an initial polymerization dimension x2 (which can be performed simultaneously with step 1):

similarly to step 1, aggregate data Conv _ x2, PCVR _ x2, clickNum _ x2, and PCTR _ x2 under different values of x2 can be obtained through aggregation statistics. For example, x2 may be the age of the ad-targeted population, and the aggregate data Conv _ x2, PCVR _ x2, clickNum _ x2, PCTR _ x2 of all ads at different ages may be aggregated and counted. As mentioned previously, step 2 may synchronize and advance the process with step 1 to increase the processing speed.

3. The polymerization is performed for the initial polymerization dimension x3 (which may be performed simultaneously with step 1, step 2):

similar to the step 1 and the step 2, aggregation data Conv _ x3, PCVR _ x3, clickNum _ x3 and PCTR _ x3 under different values of x3 can be obtained through aggregation statistics. For example, x3 may be an advertisement commodity type, and aggregate data Conv _ x3, PCVR _ x3, clickNum _ x3, PCTR _ x3 of all advertisements under different advertisement commodity types may be aggregated and counted. As mentioned earlier, step 3 can be processed in parallel with step 1 and step 2 synchronously to increase the processing speed.

4. Determining a reference aggregation dimension for a predetermined advertisement

According to the x1 dimension value, x2 dimension value and x3 dimension value of the predetermined advertisement, for example, x11 (for example, x11 represents that the predetermined advertisement belongs to advertiser a), x21 (for example, x21 represents that the age of targeted crowd of the predetermined advertisement is 18) and x31 (for example, x31 represents that the commodity type of the predetermined advertisement is first) respectively, corresponding aggregated data Conv _ x11, PCVR _ x11, clickNum _ x11 and PCTR _ x11 are obtained by searching the aggregated data under different values of x1 obtained in step 1, corresponding aggregated data Conv _ x21, PCVR _ x21, clickNum _ x21 and PCTR _ x21 are obtained by searching the aggregated data under different values of x2 obtained in step 2, and corresponding aggregated data Conv _ x31, PCVR _ x31, clicknm _ x31 and PCTR _ x31 are obtained by searching the aggregated data under different values of x3 obtained in step 3, thereby obtaining three types of aggregated data of the predetermined advertisement.

5. Determining an initial reference aggregation dimension

In this step, the initial reference aggregation dimension may be determined based on the validity of the aggregation data at x11, x21, and x31 of the predetermined advertisement and the set priorities of the initial aggregation dimensions x1, x2, and x3.

As previously described, the set priority of the initial aggregation dimensions x1, x2, x3 is x1> x2> x3. It is possible to first determine the validity of the aggregated data (Conv _ x11, PCVR _ x11, clickNum _ x11, PCTR _ x 11) corresponding to the x 1-dimensional value x11 of the predetermined advertisement. The validity determination manner is similar to the above-mentioned principle, that is, it is determined whether the total placement consumption corresponding to all the advertisements under the dimension value x11 is greater than a first predetermined threshold (for example, 4 × target _cpa), for example, it is determined whether the total placement consumption corresponding to all the advertisements belonging to the advertiser a is greater than the first predetermined threshold, if yes, it indicates that the aggregation data corresponding to x11 is valid, and at this time, the initial aggregation dimension x1 may be directly determined as the initial reference aggregation dimension of the predetermined advertisement; otherwise, it indicates that the aggregated data corresponding to x11 is invalid, the validity of the aggregated data (Conv _ x21, PCVR _ x21, clickNum _ x21, PCTR _ x 21) corresponding to the x 2-dimensional value x21 of the predetermined advertisement is continuously determined, and so on.

Finally, the aggregation data with the highest priority and high validity is retained, and may be referred to as reference aggregation data (Conv _ valid, PCVR _ valid, clickNum _ valid, PCTR _ valid), and the corresponding aggregation dimension is referred to as an initial reference aggregation dimension. In addition, if all three types of aggregation data are invalid, the reference aggregation data may be taken as a default value, for example, aggregation data (Conv _ x11, PCVR _ x11, clickNum _ x11, PCTR _ x 11) corresponding to the x 1-dimensional value x11 with the highest priority may be defaulted, that is, the default initial reference aggregation dimension is x1.

After determining the reference aggregated data, if the predetermined advertisement is in the initial stage of delivery at the historical calibration time, the historical calibration parameters for calibrating the historical PCVR corresponding to the historical click of the predetermined advertisement at the historical calibration time may be determined by the following step 6.

6. Calculating historical calibration parameters (if in initial stage of putting)

If the scheduled advertisement is in the initial stage of delivery at the historical calibration time, a historical calibration parameter f may be calculated according to the following formula:

f＝coef*Conv_valid/(PCVR_valid*PCTR_valid/ClickNum_valid) (11)

coef is an expansion coefficient used for balancing the influence of PCVR calibration on the release consumption. The expansion coefficient coef may be determined based on the impression consumption of the scheduled advertisement. Specifically, the impression consumption of an advertisement can be divided into 3 phases: [0,8 × target _cpa); (8 × target _cpa,25 × target _cpa; [25 × target _cpa, ∞.) in these three stages, the expansion coefficients can be determined as follows, respectively:

①[0，8*target_cpa)

at this time, since the predetermined advertisement is less consumed and is easily disturbed, the expansion coefficient coef may be 1.

②(8*target_cpa，25*target_cpa]

At this stage, the expansion coefficient coef can be calculated by:

③[25*target_cpa，∞)

at this stage, the expansion coefficient coef can be calculated by:

in the above equations (12) - (13), target _ cpa is the target conversion cost of the advertiser, and current _ cpa _ bias is the cpa deviation of the predetermined advertisement at the historical calibration time, which can be expressed as: current _ cpa _ bias = cost/(conv _ num × target _ cpa) -1, where cost is the delivered consumption of the scheduled advertisement at the historical alignment time, and conv _ num is the actual conversion amount of the scheduled advertisement at the historical alignment time.

In addition, if the scheduled advertisement is in the delivery maturity stage at the historical calibration time, the historical calibration parameters for calibrating the historical PCVR corresponding to the historical click of the scheduled advertisement at the historical calibration time may be determined through the following step 7.

7. Calculating historical calibration parameters (if at the mature stage of delivery)

If the predetermined advertisement is in a delivery maturity stage at the historical calibration time, a historical calibration parameter f may be calculated according to the following equation:

f＝(history_pcvr_bias_factor*conversion_num+smooth_base)/(sum_pcvr*sum_pctr/clicknum+smooth_base)

wherein the history _ PCVR _ bias _ factor is a historical PCVR calibration factor; sum _ pcvr is the sum of estimated conversion amounts corresponding to total effective clicks of the preset advertisement before the historical calibration time in the historical release period; conversion _ num is the sum of the actual conversion amount of the preset advertisement before the historical calibration time in the historical release period, and sum _ pctr is the estimated click amount corresponding to the total effective exposure of the preset advertisement before the historical calibration time in the historical release period; clicknum is the actual click rate of the preset advertisement before the historical calibration time in the historical release period; smooth _ base is a smoothing coefficient for smoothing out data anomalies due to possible data insufficiency; sum _ conversion _ yesterday represents the actual conversion amount of the scheduled advertisement in the last delivery period (i.e., the last delivery period before the historical delivery), for example, if the historical delivery period is yesterday, sum _ conversion _ yesterday represents the actual conversion amount of the scheduled advertisement in the previous day; h represents a time length (in hours) from the beginning of the historical placement period to the historical calibration time of the scheduled advertisement; ceil () function represents rounding up; the min () function represents taking the minimum value.

The historical PCVR calibration factor may be calculated based on the reference aggregated data (Conv _ valid, PCVR _ valid) determined in step 5. Further, the historical PCVR calibration factor may be calculated by:

history_pcvr_bias_factor＝Conv_valid_p/PCVR_valid_p)/(Conv_valid/PCVR_valid) (15)

wherein, conv _ valid _ p and PCVR _ valid _ p are reference aggregated data corresponding to the predetermined advertisement and in the consumption level p, respectively. In particular, since the impression consumption of an advertisement is large at the impression maturity stage and the impression consumption may also be largely different between different advertisements, the impression data of advertisements having different impression consumptions may also have a large difference and thus have different effects on the PCVR calibration of a predetermined advertisement.

Therefore, when aggregation statistics is performed for the initial aggregation dimensions x1, x2, and x3, the consumption levels of the advertisements in the respective aggregation dimensions may be further divided, and aggregation data of the advertisements in different consumption levels in each initial aggregation dimension may be counted. For example, the consumption of the advertisement may be divided into 5 steps as shown in column 2 of table 1 below. For example, for the initial aggregation dimension x1, the sum of the total conversion amount of all the advertisements whose serving consumption falls within each consumption profile respectively and the total PCVR corresponding to the valid clicks under the initial aggregation dimension x1 can be counted, for example, the aggregation data obtained in the profile 1 under the initial aggregation dimension x1 can be represented as (Conv _ x1_1, PCVR_x 1 \, 1), and so on, as shown in table 1.

TABLE 1 polymerization dimension at initial referenceDegree x ₁ Aggregated data in various consumption levels

Grade of	Releasing consumption Cost	Counting the amount of conversion and PCVR
			1	Cost<＝4*target_cpa	Conv_x1_1，PCVR_x1_1
2	4target_cpa<Cost<＝10target_cpa	Conv_x1_2，PCVR_x1_2
			3	10target_cpa<Cost<＝20target_cpa	Conv_x1_3，PCVR_x1_3
4	20target_cpa<Cost<＝40target_cpa	Conv_x1_4，PCVR_x1_4
			5	Cost>40*target_cpa	Conv_x1_5，PCVR_x1_5

Similarly, the aggregation data in each consumption level can be obtained by statistics aiming at the initial reference aggregation dimension x 2; and counting the aggregation data in each consumption grade aiming at the initial reference aggregation dimension x3.

And then, determining the consumption grade of the scheduled advertisement according to the delivery consumption of the scheduled advertisement at the historical calibration time. Assuming that the predetermined advertisement is in a consumption level p, and the x 1-dimensional value, the x 2-dimensional value and the x 3-dimensional value of the predetermined advertisement are x11, x21 and x31, respectively, three types of aggregated data of the consumption level corresponding to the predetermined advertisement can be obtained from the aggregated data in the initial aggregation dimensions x1, x2 and x 3: (Conv _ x11_ p, PCVR _ x11_ p), (Conv _ x21_ p, PCVR _ x21_ p), (Conv _ x31_ p, PCVR _ x31_ p).

Thereafter, similarly to the above-described step 5, the initial reference aggregation dimension is determined based on the validity of the aggregation data under x11, x21, and x31 of the predetermined advertisement and the set priorities of the initial aggregation dimensions x1, x2, and x3. For example, assuming that the determined initial reference aggregation dimension is x1, the reference aggregation data (Conv _ valid, PCVR _ valid) corresponding to the scheduled advertisement at this time is (Conv _ x11, PCVR _ x 11); and the corresponding aggregate data (Conv _ valid _ p, PCVR _ valid _ p) of the sub-consumption level is (Conv _ x11_ p, PCVR _ x11_ p). At this time, the historical PCVR calibration factor may be calculated according to equation (15) above, and further, the historical calibration parameter f of the scheduled advertisement at the delivery maturity stage may be calculated according to equation (14) above.

The process of calculating the historical calibration parameters for PCVR calibration of any predetermined advertisement in the predetermined advertisement set at any historical calibration time within the historical delivery period is described above through steps 1 to 7. If the scheduled advertisement is in the initial stage of delivery at the historical calibration time, the historical calibration parameters can be calculated by using the step 6; if at the release maturity stage, its historical calibration parameters may be calculated using step 7. As mentioned above, whether the scheduled advertisement is in the initial stage of delivery or the mature stage of delivery may be determined based on the delivery consumption and the conversion amount of the scheduled advertisement before the historical calibration time in the historical delivery period, and the specific determination rule is the same as the principle described above at step S240, and is not described here again.

After determining the historical calibration parameters for the predetermined advertisement at the historical calibration time, the PCVR corresponding to the historical clicks of the predetermined advertisement at the historical calibration time may be calibrated using the historical calibration parameters. For example, the PCVR at the historical calibration time acquired by a conversion prediction model such as PNN, FM, etc. may be calibrated by multiplying the PCVR by historical calibration parameters.

Through the steps 1-7, the calibration of the PCVR of the historical click corresponding to the scheduled advertisement at the historical calibration time is completed. The above process is repeated until the PCVR calibration of all valid historical clicks of the scheduled advertisement in the historical delivery period is completed, and the sum (which may be referred to as historical predicted conversion amount) of the calibrated PCVR corresponding to all valid historical clicks of the scheduled advertisement in the historical delivery period, that is, sumPcvr, is obtained through statistics _i . At this time, the actual historical conversion amount convNum of the scheduled advertisement in the historical delivery period is used _i That is, the deviation between the historical predicted transformation amount and the historical actual transformation amount can be calculated, for example, (sumPcvr) _i -convNum _i ) ² 。

The above process is repeated until the PCVR calibration for all valid historical clicks of all scheduled advertisements (i.e., advertisement 1 \8230;. N) in the scheduled advertisement set in the historical placement period is completed, and the deviation of each scheduled advertisement is calculated, so that the cumulative sum of the deviations of all scheduled advertisements in the scheduled advertisement set, i.e., the objective function, can be obtained

To this end, one calculation cycle is completed for a randomly selected initial aggregation dimension (x 1, x2, x 3). The random selection of new initial polymerization dimensions (x 1', x2', x3 ') continues, and the above calculation cycle is repeated. All attributes in the attribute set are exhausted, and finally, a plurality of initial aggregation dimensions that minimize the objective function are determined as a plurality of candidate aggregation dimensions, and a set priority (for example, x1> x2> x 3) of each of the plurality of initial aggregation dimensions that minimize the objective function is determined as a priority of each corresponding candidate aggregation dimension. At this time, the solution of the objective function of the dimension optimization model is completed.

For example, a particle swarm algorithm may be used to solve the objective function of the dimension optimization model. The particle swarm algorithm is a random search algorithm based on group cooperation, and can quickly search an optimal solution in a solution space. However, the embodiments of the present disclosure are not limited thereto, and any other suitable algorithm may be adopted to perform the solving process of the objective function of the dimension optimization model according to the embodiments of the present disclosure. For example, three candidate aggregation dimensions x1, x2, and x3 which are respectively a commodity core product word, a sub-industry to which an advertisement belongs, and an interested industry category can be solved by using an example group algorithm, and the priority order is commodity core product word > advertisement sub-industry > interested industry category, so that in a new delivery cycle, PCVR calibration can be performed on target recommendation information by using the three candidate aggregation dimensions and the priorities thereof.

In addition, it should be noted that, when aggregation statistics is performed on multiple initial aggregation dimensions (x 1, x2, x 3) in step 1-3, a "site set" sub-dimension, an "old-new" sub-dimension, and the like may be further subdivided, and aggregation is performed under these subdivided sub-dimensions (for example, aggregation similar to the above-mentioned division of consumption levels). Taking an advertisement as an example, the "site set" refers to different delivery platforms of the advertisement, such as a communication platform, an e-commerce platform, and the like, and because delivery data of the advertisement on different site sets have different macroscopic characteristics, the PCVR calibration is affected; "old and new" means whether the ad is an initial placement, for example, an ad that has never been placed before the placement is started in the current placement period (e.g., today, or the last two days, etc.) is a new ad, or an old ad, which also has an effect on PCVR calibration because the placement data of the old and new ads also has different characteristics.

In addition, it should be noted that the specific operations, parameters, formulas and the like in the above steps 1 to 7 are taken as examples and not as limitations to the embodiments of the present disclosure, and various possible modifications based on the above steps 1 to 7 are within the scope of the embodiments of the present disclosure. For example, in step 7, a history _ PCVR calibration factor, that is, historical data information of an associated advertisement of a predetermined advertisement is introduced by adding history _ PCVR _ bias _ factor in formula (14), but the embodiment of the present disclosure is not limited thereto, and the history PCVR calibration factor may be introduced in any other suitable manner, such as weighting and the like; for another example, in step 7, a click rate (CTR) calibration factor is introduced by adding sum _ pctr/clicknum in formula (14), but the embodiments of the present disclosure are not limited thereto, and the CTR calibration factor may also be introduced in any other suitable manner, such as weighting, etc., and will not be illustrated here.

Therefore, aggregation can be further performed under the "site set" sub-dimension, "old and new" sub-dimension, and the like. For example, assuming there are two different sets of sites A and B in common, then in step 1 above, the dimension x is aggregated for the initial reference ₁ Can be further subdivided [ A, new advertisement]And B, new advertisement]And [ A, old advertisement]And B old advertisement]And counting the aggregate data of all advertisements under the 4 sub-dimensions respectively. Similarly, in steps 2 and 3 above, the aggregated data of all advertisements in these 4 sub-dimensions is also counted for the initial aggregation dimensions x2 and x3, respectively. Accordingly, when determining the reference aggregation dimension of the predetermined advertisement in step 4, in addition to the x 1-dimensional value, the x 2-dimensional value, and the x 3-dimensional value of the predetermined advertisement, the aggregation data in the corresponding three types of sub-dimensions may be selected based on which site set the predetermined advertisement belongs to, and the new advertisement and the old advertisement, and the historical calibration parameters may be calculated in the subsequent step by using the aggregation data in the three types of sub-dimensions. Since the process of subdividing the sub-dimensions to calculate the historical calibration parameters is similar to steps 1-7 described above, it is not repeated here for simplicity.

The above describes in detail a process of determining a plurality of candidate aggregation dimensions of target recommendation information and priorities thereof by solving an objective function of a dimension optimization model, with advertisements as an example of recommendation information.

In the recommendation information processing method according to the embodiment of the present disclosure, the specific implementation process of determining the reference aggregation dimension in step S220, acquiring the delivery data of the reference recommendation information set corresponding to the target recommendation information in step S230, and determining the calibration parameter in step S240 is similar to the process described in the above steps 1 to 7, and the difference is only that: (1) the initial aggregation dimensions x1, x2 and x3 in the steps 1 to 3 are respectively set as three candidate aggregation dimensions obtained by solving, such as a commodity core product word, a sub-industry to which the advertisement belongs and an interested industry category, and the priorities of the three candidate aggregation dimensions obtained by solving are adopted, such as a commodity core product word > the sub-industry to which the advertisement belongs > the interested industry category; (2) the historical launch period should be replaced with the current launch period and the historical calibration time should be replaced with the current calibration time. Therefore, a specific example of steps S220 to S240 will not be described repeatedly here.

In one example, a PCVR calibration is performed for each advertisement in a predetermined set of advertisements using a recommendation processing method 200 according to an embodiment of the present disclosure, the calibration results are shown in the tables of FIGS. 4-7. In the recommendation information processing method 200 employed in this example, the PCVR calibration is performed on the predetermined advertisement set by solving three optimal candidate aggregation dimensions by using an objective function of a dimension optimization model, and a statistical mean of PCVR relative calibration deviations of advertisements in the predetermined advertisement set is counted in the advertisement dimension and the advertiser dimension, respectively, as shown in the third column in fig. 4 and 6 (labeled MODE), and a distribution ratio of the PCVR relative calibration deviations falling into each deviation interval is shown in the second row of each time interval in fig. 5 and 7 (labeled MODE). The statistical mean of the PCVR relative calibration deviation may be, for example, an average value, a mean square value, a root mean square value, a weighted value, and the like of the PCVR relative calibration deviation of each advertisement in the predetermined advertisement set, which is not particularly limited by the embodiments of the present disclosure.

Furthermore, to advantage over the highlighting method 200, three candidate aggregation dimensions, i.e., advertiser, advertised item, and advertised item brand, are additionally manually selected and the same process of PCVR calibration is performed on the same predetermined SET of advertisements, with the statistical mean of the PCVR relative calibration offsets for advertisements in the calibrated predetermined SET of advertisements being shown in the second column of fig. 4 and 6 (labeled SET), and the distribution of the PCVR relative calibration offsets falling into the various offset intervals being shown, for example, in the first row of each time slot in fig. 5 and 7 (labeled SET).

As can be seen from fig. 4-7, compared to the artificial selection of candidate aggregation dimensions, with the recommended information processing method 200 according to the embodiment of the present disclosure, by solving the optimal candidate aggregation dimensions using the objective function of the dimension optimization model to perform the PCVR calibration, the statistical mean of the PCVR relative calibration deviation is effectively reduced, and the ratio of the PCVR relative calibration deviation falling within the overestimated deviation interval (e.g., (1.5, ∞)) or the underestimated deviation interval (e.g., (0, 0.5)) is also reduced, i.e., the ratio of the PCVR relative calibration deviation being higher or lower can be improved.

By using the recommendation information processing method according to the embodiment of the disclosure, based on the dimension optimization model established for the historical delivery data of the preset recommendation information set containing the target recommendation information in the historical delivery period, the multiple candidate aggregation dimensions capable of realizing the optimal PCVR calibration and the priority of each candidate aggregation dimension are determined by solving the objective function of the dimension optimization model, so that the accurate PCVR calibration of the target recommendation information can be realized by using the multiple candidate aggregation dimensions and the priorities thereof, and the accuracy of the PCVR calibration is greatly improved; moreover, by using the recommendation information processing method according to the embodiment of the disclosure, calibration can be performed in a targeted manner by analyzing the characteristics of the estimated conversion rate of recommendation information of a specific industry, a specific audience, specific content and the like, so that recommendation information delivery, delivery cost control, delivery effect prediction and the like can be performed more accurately.

In addition, the estimated conversion rate after calibration obtained by the recommendation information processing method according to the embodiment of the disclosure can be used for sequencing recommendation information to be released, so that the recommendation information with higher estimated income is released preferentially, and the income maximization is realized. A recommendation information ranking method according to an embodiment of the present disclosure is described below with reference to fig. 8. FIG. 8 shows a flow diagram of a recommendation information ranking method 800 according to an embodiment of the disclosure.

As shown in fig. 8, in step S810, a calibrated estimated conversion rate of each of the plurality of to-be-delivered recommendation information is obtained. The plurality of pieces of recommendation information to be delivered may be any recommendation information to be delivered, such as a plurality of advertisements to be delivered, which is not limited in this disclosure. In the embodiment of the present disclosure, for example, the estimated conversion rate of each of the plurality of pieces of recommendation information to be delivered may be obtained by calibrating the estimated conversion rate of each of the plurality of pieces of recommendation information to be delivered by using the recommendation information processing method described above with reference to fig. 2, but the embodiment of the present disclosure is not limited thereto, and the estimated conversion rate of each of the plurality of pieces of recommendation information to be delivered may also be obtained in other manners. The estimated conversion rate of each piece of recommendation information to be delivered may be obtained through the conversion rate estimation model such as PNN and FM as described above, or obtained through any other method, which is not specifically limited in the embodiment of the present disclosure. Since the step of calibrating the estimated conversion rate of the recommendation information using the recommendation information processing method shown in fig. 2 has been described in detail above, a repeated description of the same contents is omitted here for the sake of simplicity.

In step S820, an estimated benefit of each piece of recommendation information to be delivered in the plurality of pieces of recommendation information to be delivered is calculated according to the calibrated estimated conversion rate of each piece of recommendation information to be delivered in the plurality of pieces of recommendation information to be delivered. The following description will be made by taking an advertisement as an example. For example, the predicted revenue to be advertised may be measured in effective thousand-time revenue (eCPM), but the disclosed embodiments are not limited thereto and may be measured in any other metric such as a Cost-to-output Ratio (ROI). Generally, the eCPM of an advertisement may depend on a predicted conversion rate (PCVR), a predicted click-through rate (PCTR), and an advertiser's bid (bid). Wherein, the advertiser's bid may refer to a fee that the advertiser is willing to pay for an advertising conversion amount. Thus, the eCPM of an advertisement may be expressed as:

eCPM＝PCVR×PCTR×bid

from the above formula, it can be seen that the higher the estimated conversion rate PCVR of the advertisement is, the higher the estimated profit eCPM thereof is, so that higher revenue can be brought to the advertisement platform. After the calibrated estimated conversion rate of each advertisement to be delivered is obtained in step S810, the estimated revenue eCPM of each advertisement to be delivered, which is more accurate, can be calculated by using the above formula. The estimated click-through rate PCTR can be obtained by an estimation method known in the art, and the advertiser bid depends on the real-time bid of the advertiser.

After obtaining the estimated benefit, such as eCPM, of each recommendation information to be delivered, in step S830, the plurality of recommendations information to be delivered may be ranked based on the estimated benefit of each recommendation information to be delivered. For example, the recommended information to be released with higher estimated profit may be ranked at a former position, and the recommended information to be released with lower estimated profit may be ranked at a later position. After the sorting is completed, the plurality of pieces of recommendation information to be released may be sequentially released in a new release period according to the sorting result, where the new release period may refer to, for example, a new release time period, a new release day, and the like. For example, one or more pieces of recommended information to be released, which are ranked the top, may be selected for releasing; or, the recommended information to be released which is ranked the top may be selected for releasing first, then the recommended information to be released which is ranked the second may be selected for releasing, and so on.

By using the recommendation information ranking method according to the embodiment of the disclosure, the estimated profit of the recommendation information to be released can be more accurately calculated based on the calibrated estimated conversion rate of the recommendation information to be released, so that the plurality of recommendation information to be released can be ranked based on the estimated profit, and thus the recommendation information to be released with higher estimated profit can be preferentially released, and the profit maximization is realized.

A recommended information processing apparatus according to an embodiment of the present disclosure is described below with reference to fig. 9. Fig. 9 shows a schematic configuration diagram of a recommended information processing apparatus 900 according to an embodiment of the present disclosure. As shown in fig. 9, the recommended information processing apparatus 900 may include a dimension determining unit 910, a selecting unit 920, an aggregating unit 930, a calibration parameter determining unit 940, and a calibration unit 950. The calibration device 900 may include other components in addition to the five units, however, since these components are not relevant to the content of the embodiments of the present disclosure, illustration and description thereof are omitted here. In addition, since the functions of the recommendation information processing apparatus 900 are similar to the details of the steps of the recommendation information processing method 200 described above with reference to fig. 2, a repeated description of part of the contents is omitted here for the sake of simplicity.

The dimension determining unit 910 is configured to determine, based on a dimension optimization model established for historical placement data of a predetermined recommendation information set containing target recommendation information over a historical placement period, a plurality of attributes as a plurality of candidate aggregation dimensions from an attribute set associated with the target recommendation information by solving an objective function of the dimension optimization model, and determine a priority of each candidate aggregation dimension of the plurality of candidate aggregation dimensions. The target recommendation information may be any recommendation information to be subjected to PCVR calibration, such as an advertisement, recommendation information, an article/video promotion, and the like, which is not specifically limited in this disclosure.

According to an example of an embodiment of the present disclosure, the set of attributes associated with the target recommendation information may include at least a home attribute subset, an audience attribute subset, and a content attribute subset. As the name implies, the affiliation-attribute subset includes a plurality of attributes related to affiliation of the recommendation information, the audience-attribute subset includes a plurality of attributes related to an audience of the recommendation information, and the content-attribute subset includes a plurality of attributes related to content of the recommendation information.

The dimension determining unit 910 determines a plurality of attributes as a plurality of candidate aggregation dimensions from an attribute set associated with target recommendation information based on historical placement data of a predetermined recommendation information set containing the target recommendation information in a historical placement period, so that information for PCVR calibration of the target recommendation information can be acquired by aggregating placement data of recommendation information in respective candidate aggregation dimensions.

According to the embodiment of the disclosure, based on a dimension optimization model established for historical delivery data of a predetermined recommendation information set containing target recommendation information in a historical delivery period, by solving an objective function of the dimension optimization model, the optimal attributes are determined from an attribute set of the target recommendation information to serve as a plurality of candidate aggregation dimensions.

The predetermined recommendation information set is a set of a plurality of recommendation information to be calibrated (hereinafter may be referred to as predetermined recommendation information), and the target recommendation information may be any recommendation information in the predetermined recommendation information set. The predetermined recommendation information in the predetermined recommendation information set may be determined randomly, for example; or may belong to a common advertiser, that is, the predetermined recommendation information set may be a set of all advertisements under a certain advertiser, that is, the recommendation information processing apparatus 900 according to the embodiment of the present disclosure may perform PCVR calibration for all advertisements under a common advertiser; or may have a common industry orientation, that is, the predetermined recommendation information set may be a set of all advertisements in a certain industry orientation, that is, the recommendation information processing apparatus 900 according to the embodiment of the present disclosure may perform PCVR calibration on all advertisements in a common industry orientation, and so on, but the embodiment of the present disclosure is not limited thereto, and the predetermined recommendation information set may also be a set of recommendation information to be calibrated, which is selected according to any other rule.

Furthermore, since the aggregated delivery data under different aggregation dimensions has different influence on the accuracy of the PCVR calibration of the target recommendation information, the dimension determining unit 910 may also determine the priority of each candidate aggregation dimension of the plurality of candidate aggregation dimensions. For example, the priority of the PCVR calibration of the target recommendation information is determined according to the influence of different aggregation dimensions on the accuracy thereof. In the embodiment of the present disclosure, a dimension optimization model may be constructed, and a plurality of candidate aggregation dimensions and priorities of the candidate aggregation dimensions may be determined by solving an objective function of the dimension optimization model. The dimension optimization model can be defined by the above equations (7) - (8), and the process of solving the objective function of the dimension optimization model can refer to the process described above in connection with steps 1-7, for example, and is not described herein again.

The selecting unit 920 is configured to select a reference aggregation dimension of the target recommendation information from the plurality of candidate aggregation dimensions for PCVR calibration of the target recommendation information based on at least the priority of each candidate aggregation dimension of the plurality of candidate aggregation dimensions determined by the dimension determining unit 910. For example, the candidate aggregation dimension with the highest priority may be selected as the reference aggregation dimension. However, in some cases, for example, in the initial stage of delivery of recommendation information, the delivery data of recommendation information in some candidate aggregation dimensions may not be sufficient yet, and the data obtained by aggregating the delivery data of recommendation information in the candidate aggregation dimensions does not have a valid reference value, that is, the aggregated data is invalid. If the aggregated data of the highest priority candidate aggregated dimension of the plurality of candidate aggregated dimensions is invalid, performing PCVR calibration on the target recommendation information using the aggregated data of the highest priority candidate aggregated dimension may be inaccurate. Therefore, in selecting the reference aggregation dimension of the target recommendation information, in addition to the priority, the validity of the aggregation data in each candidate aggregation dimension corresponding to the target recommendation information should be considered.

According to an example of the embodiment of the present disclosure, for the recommendation information set in each candidate aggregation dimension, that is, the set formed by all recommendation information in each candidate aggregation dimension, the selecting unit 920 may further determine the validity of the aggregation data in each candidate aggregation dimension based on the total delivery consumption of the recommendation information set. According to an example of an embodiment of the present disclosure, when total delivery consumption of a recommendation information set in a certain candidate aggregation dimension is greater than a first predetermined threshold, it may be determined that aggregated data of the recommendation information set is valid; otherwise, determining that the aggregation data of the recommendation information set is invalid. The first predetermined threshold may be set according to practical situations, and is not particularly limited by the embodiments of the present disclosure.

Taking an advertisement as an example, the advertisement placement consumption refers to the fee charged to the advertiser for a certain amount or time of advertisement placement, and the first predetermined threshold may be set according to the target conversion cost (target _ cpa) of the advertiser, where the target conversion cost of the advertiser is the fee required for each conversion of a single advertisement desired by the advertiser. For example, aggregated data for a set of ads may be determined to be valid when the total consumption of ads in the set of ads is greater than 4 times the target conversion cost (4 × target _cpa); otherwise, determining that the aggregated data for the set of advertisements is invalid.

Therefore, according to an example of an embodiment of the present disclosure, the selecting unit 920 is further configured to: acquiring total release consumption of a recommendation information set corresponding to the target recommendation information and under each candidate aggregation dimension in the multiple candidate aggregation dimensions; and determining a candidate aggregation dimension with the highest priority and the total delivery consumption of the recommendation information sets larger than a first preset threshold value in the plurality of candidate aggregation dimensions as a reference aggregation dimension. That is, the candidate aggregation dimension having the highest priority among the plurality of candidate aggregation dimensions and having the corresponding aggregation data valid is determined as the reference aggregation dimension to ensure that the PCVR calibration can be performed on the target recommendation information with the most relevant and sufficient aggregation data.

After the reference aggregation dimension is determined, the aggregation unit 930 obtains a total estimated conversion amount (PCVR _ valid) and a total actual conversion amount (Conv _ valid) of the reference recommendation information set before the current time in the current delivery cycle by aggregating the estimated conversion amount and the actual conversion amount of the reference recommendation information in the reference recommendation information set under the reference aggregation dimension corresponding to the target recommendation information. Here, a set formed by referring to all recommendation information in the aggregation dimension is referred to as a reference recommendation information set. The release period may be any time period, such as 12h, 24h (i.e. a complete day), a week, etc., which may be set according to actual needs, and embodiments of the present disclosure do not specifically limit this. For example, for advertising, it is common to have one day (24 h) as one impression period. The current time is the time when the PCVR calibration is to be performed on the target recommendation information. For example, for a click of an advertisement at a current time, after a predicted conversion rate corresponding to the click is predicted by using a model such as PNN, FM, or the like, the aggregation unit 930 may obtain a total predicted conversion amount and a total actual conversion amount generated by the reference recommendation information set before the current time in the current delivery cycle by aggregating the predicted conversion amount and the actual conversion amount of all the reference recommendation information in the reference recommendation information set for calibrating the predicted conversion rate.

The calibration parameter determination unit 940 is configured to determine a calibration parameter based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, so that the calibration unit 950 calibrates the predicted conversion rate corresponding to the click of the target recommendation information at the current time using the calibration parameter.

Generally, in the initial stage of launching the target recommendation information, the launch data of the target recommendation information itself is not sufficient, and at this time, the quotient of the total actual conversion amount and the total estimated conversion rate of the reference recommendation information set may be directly determined as the calibration parameter, that is, the calibration parameter f may be determined by the above equation (1).

With the progress of the delivery, the click quantity and the conversion quantity of the target recommendation information per se are continuously increased, that is, more delivery data are generated, and at the moment, the target recommendation information can be called to enter the delivery mature stage. In the release maturity stage, the release data of the target recommendation information can also be used for carrying out PCVR calibration on the target recommendation information, and due to better correlation, the release data of the target recommendation information plays a more important role in the PCVR calibration process compared with the data of the reference recommendation information set, so that the accuracy of the PCVR calibration is further improved. In addition, in some cases, for example, for CPM (Cost Per thousand) advertisement, the Predicted Click Through Rate (PCTR) also has an effect on the Predicted profit of the advertisement, so that when the advertisement is calibrated for PCVR, if the Predicted Click Through Rate of the advertisement can be considered at the same time, the accuracy of the Predicted profit determined by using the calibrated PCVR can be effectively improved.

Therefore, according to an example of an embodiment of the present disclosure, the recommendation information processing apparatus 900 may further include a delivery stage determination unit 960 configured to determine that the target recommendation information is at the initial stage of delivery or the mature stage of delivery of the current delivery cycle at the current time based on a predetermined rule. For example, the delivery stage determination unit 960 may determine whether the target recommendation information is in the initial delivery stage or the mature delivery stage based on the current delivery consumption and the current conversion amount of the target recommendation information before the current time within the current delivery period. Specifically, the delivery stage determining unit 960 may statistically determine that the target recommendation information is in a delivery initial stage when the current delivery consumption is less than or equal to a second predetermined threshold and the current conversion amount is less than or equal to a third predetermined threshold, and otherwise, determine that the target recommendation information is in a delivery mature stage. The second predetermined threshold and the third predetermined threshold may be determined according to actual situations, and the embodiment of the present disclosure does not specifically limit this. For example, taking an ad as an example, the second predetermined threshold may be 2 times the targeted conversion cost (2 x target _cpa), which, as described above, is the cost required for each conversion of a single ad desired by the advertiser; the third predetermined threshold may be, for example, 2. That is, when the current delivery consumption is less than or equal to 2 × target _cpaand the current conversion amount is less than or equal to 2, it may be determined that the target recommendation information is in the initial delivery stage, otherwise, it may be determined that the target recommendation information is in the mature delivery stage.

According to an example of an embodiment of the present disclosure, when the target recommendation information is in the initial stage of the delivery, the aggregation unit 930 may further obtain a total predicted click rate (PCTR _ valid) and a total actual click rate (ClickNum _ valid) of the reference recommendation information set before the current time within the current delivery cycle, and the calibration parameter determination unit 940 may determine the calibration parameter based on at least the obtained total predicted conversion rate (PCVR _ valid), total actual conversion rate (Conv _ valid), total predicted click rate (PCTR _ valid), and total actual click rate (ClickNum _ valid) of the reference recommendation information set. At this time, the calibration parameter f can be determined by the above equation (2).

When the PCVR calibration is performed on the target recommendation information at the initial stage of delivery by using the calibration parameter of the above equation (2), in order to avoid the influence of the PCVR calibration process on the current delivery consumption of the target recommendation information, an expansion coefficient (coef) determined based on the delivery consumption of the target recommendation information may be further introduced, and then the calibration parameter is determined based on the expansion coefficient (coef) and a total estimated conversion amount (PCVR _ valid), a total actual conversion amount (Conv _ valid), a total estimated click amount (PCTR _ valid), and a total actual click amount (ClickNum _ valid) of the reference recommendation information set. At this time, the calibration parameter f may be further determined by the above equation (3).

According to an example of the embodiment of the present disclosure, when the target recommendation information is in the delivery maturity stage, the aggregation unit 930 may further obtain a pre-estimated conversion amount (sum _ pcvr), an actual conversion amount (conversion _ num), a pre-estimated click amount (sum _ pctr), and an actual click amount (clicknum) of the target recommendation information before the current time in the current delivery cycle. At this time, the calibration parameter determination unit 940 may determine the calibration parameter at least based on the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, and the predicted conversion amount (sum _ pcvr), the actual conversion amount (conversion _ num), the predicted click amount (sum _ pctr) and the actual click amount (clicknum) of the target recommendation information before the current time in the current delivery cycle. As mentioned above, in the calibration of PCVR at the release maturity stage, the release data of the target recommendation information itself should play a more important role, and the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set can be used as the auxiliary factors. For example, the factor determined based on the total predicted amount of conversion and the total actual amount of conversion of the reference recommendation information set may be referred to as a historical PCVR calibration factor (history _ PCVR _ bias _ factor). At this time, the calibration parameter f can be determined by the above equation (4).

In addition, in some cases, the delivery data of the target recommendation information may be relatively small even at the delivery maturity stage, which may result in a calibration parameter determined according to equation (4) above being higher or lower, i.e., a data anomaly may occur. At this time, such data abnormality may be smoothed by introducing a smoothing coefficient (smooth _ base). Specifically, a historical actual conversion amount of the target recommendation information in the historical release period may be obtained, and the smoothing coefficient may be determined based on the historical actual conversion amount of the target recommendation information and a time length from a current time to a current release period. For example, the smoothing coefficient smooth _ base may be determined by the above equation (5).

After the smoothing coefficient is determined, the calibration parameters may be determined based on the smoothing coefficient, the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set, and the predicted conversion amount (sum _ pcvr), the actual conversion amount (conversion _ num), the predicted click amount (sum _ pctr) and the actual click amount (clicknum) of the target recommendation information before the current time in the current delivery cycle. For example, the calibration parameter f may be further determined by the above equation (6).

After determining the calibration parameters, the calibration unit 950 may calibrate the estimated conversion rate corresponding to the click of the target recommendation information at the current time using the calibration parameters. Specifically, at any current time, after the estimated conversion rate corresponding to the click at the current time obtained using a conversion rate estimation model such as PNN, FM, or the like. The estimated conversion may be calibrated by multiplying the estimated conversion by a calibration parameter. Alternatively, the calibration parameters may be used as weighting factors for a conversion estimation model such as PNN, FM, etc. to directly generate a calibrated estimated conversion.

By using the recommendation information processing device according to the above embodiment of the disclosure, based on the dimension optimization model established for the historical delivery data of the predetermined recommendation information set containing the target recommendation information in the historical delivery period, the multiple candidate aggregation dimensions capable of realizing the optimal PCVR calibration and the priority of each candidate aggregation dimension can be determined by solving the objective function of the dimension optimization model, so that the accurate PCVR calibration on the target recommendation information can be realized by using the multiple candidate aggregation dimensions and the priorities thereof, and the accuracy of the PCVR calibration is greatly improved; moreover, with the recommendation information processing apparatus according to the embodiment of the present disclosure, calibration can be performed in a targeted manner by analyzing characteristics of estimated conversion rates of recommendation information of a specific industry, a specific audience, specific content, and the like, so that recommendation information delivery, delivery cost control, delivery effect prediction, and the like can be performed more accurately.

A recommendation information ranking apparatus according to an embodiment of the present disclosure is described below with reference to fig. 10. Fig. 10 shows a schematic structural diagram of a recommendation information ranking apparatus 1000 according to an embodiment of the present disclosure. As shown in fig. 10, the recommendation information ranking apparatus 1000 includes an obtaining unit 1010, a profit estimation unit 1020, and a ranking unit 1030. In addition to these three units, the recommendation information ranking apparatus 1000 may further include other components, however, since these components are not relevant to the contents of the embodiments of the present disclosure, illustration and description thereof are omitted here. In addition, since the function of the recommendation information ranking apparatus 1000 is similar to the details of the steps of the recommendation information ranking method 800 described above with reference to fig. 8, a repeated description of part of the contents is omitted here for the sake of simplicity.

The obtaining unit 1010 is configured to obtain a calibrated estimated conversion rate of each of the plurality of to-be-delivered recommendation information. The plurality of pieces of recommendation information to be delivered may be any recommendation information to be delivered, such as a plurality of advertisements to be delivered, which is not limited in this disclosure. In the embodiment of the present disclosure, for example, the obtaining unit 1010 may obtain the calibrated estimated conversion rate of each of the plurality of pieces of recommendation information to be delivered by calibrating the estimated conversion rate of each of the plurality of pieces of recommendation information to be delivered by using the recommendation information processing method described above with reference to fig. 2, but the embodiment of the present disclosure is not limited thereto, and may also obtain the calibrated estimated conversion rate of each of the recommendation information to be delivered by other means. The estimated conversion rate of each recommendation information to be delivered may be obtained through a conversion rate estimation model such as PNN and FM as described above, or obtained through any other method, which is not specifically limited in this disclosure. Since the step of calibrating the estimated conversion rate of the recommendation information using the recommendation information processing method shown in fig. 2 has been described in detail above, a repeated description of the same contents is omitted here for the sake of simplicity.

The profit estimation unit 1020 is configured to calculate an estimated profit for each of the plurality of recommendation information to be released according to the calibrated estimated conversion rate of each of the plurality of recommendation information to be released obtained. The following description will take an advertisement as an example. For example, the predicted revenue to be advertised may be measured in effective thousand-time revenue (eCPM), but the disclosed embodiments are not limited thereto and may be measured in any other metric such as a Cost-to-output Ratio (ROI). Generally, the eCPM of an advertisement may depend on a predicted conversion rate (PCVR), a predicted click-through rate (PCTR), and an advertiser's bid (bid). Wherein, the advertiser's bid may refer to a fee that the advertiser is willing to pay for an advertising conversion amount. Thus, an advertised eCPM may be expressed as:

eCPM＝PCVR×PCTR×bid

from the above formula, it can be seen that the higher the estimated conversion rate PCVR of the advertisement is, the higher the estimated profit eCPM thereof is, so that higher revenue can be brought to the advertisement platform. After the obtaining unit 1010 obtains the calibrated estimated conversion rate of each advertisement to be delivered, the estimated revenue eCPM of each advertisement to be delivered can be calculated by using the above formula. The estimated click-through rate PCTR can be obtained by an estimation method known in the art, and the advertiser bid depends on the real-time bid of the advertiser.

After the revenue estimating unit 1020 obtains the estimated revenue, such as eCPM, of each recommendation information to be delivered, the ranking unit 1030 may rank the plurality of recommendations information to be delivered based on the estimated revenue of each recommendation information to be delivered. For example, the recommended information to be released with higher estimated profit may be ranked at a former position, and the recommended information to be released with lower estimated profit may be ranked at a later position. After the sorting is completed, the sorting unit 1030 may sequentially drop the plurality of recommendation information to be dropped according to the sorting result in a new drop period, where the new drop period may refer to a new drop time period, a new drop day, and the like. For example, one or more pieces of recommended information to be released, which are ranked the top, can be selected for releasing; or, the top ranked recommendation information to be released may be selected for releasing first, then the second ranked recommendation information to be released may be selected for releasing, and so on.

By utilizing the recommendation information sequencing device according to the embodiment of the disclosure, the prediction profit of the recommendation information to be released can be more accurately calculated based on the calibrated prediction conversion rate of the recommendation information to be released, so that the plurality of recommendation information to be released can be sequenced based on the prediction profit, the recommendation information to be released with higher prediction profit can be preferentially released, and the profit maximization is realized.

Further, devices according to embodiments of the present disclosure (e.g., recommendation information processing devices, recommendation information ranking devices, etc.) may also be implemented by way of the architecture of an exemplary computing device shown in fig. 11. FIG. 11 shows a schematic diagram of an architecture of an exemplary computing device, in accordance with embodiments of the present disclosure. As shown in fig. 11, the computing device 1100 may include a bus 1110, one or more CPUs 1120, a Read Only Memory (ROM) 1130, a Random Access Memory (RAM) 1140, communication ports 1150 for connecting to a network, input/output components 1160, a hard disk 1170, and the like. Storage devices in the computing device 1100, such as the ROM 1130 or the hard disk 1170, may store various data or files used by the computer for processing and/or communication and program instructions for execution by the CPU. The computing device 1100 may also include a user interface 1180. Of course, the architecture shown in FIG. 11 is merely exemplary, and one or more components of the computing device shown in FIG. 11 may be omitted as desired when implementing different devices. The device according to the embodiments of the present disclosure may be configured to execute the recommended information processing method and the recommended information ranking method according to the above-described respective embodiments of the present disclosure, or to implement the recommended information processing apparatus and the recommended information ranking apparatus according to the above-described respective embodiments of the present disclosure.

Embodiments of the present disclosure may also be implemented as a computer-readable storage medium. Computer readable storage media according to embodiments of the present disclosure have computer readable instructions stored thereon. The recommendation information processing method and the recommendation information ranking method according to the embodiments of the present disclosure described with reference to the above drawings may be performed when computer readable instructions are executed by a processor. Computer-readable storage media include, but are not limited to, volatile memory and/or nonvolatile memory, for example. Volatile memory can include, for example, random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, etc.

There is also provided, in accordance with an embodiment of the present disclosure, a computer program product or computer program, including computer readable instructions, the computer readable instructions being stored in a computer readable storage medium. The processor of the computer device may read the computer readable instructions from the computer readable storage medium, and execute the computer readable instructions, so that the computer device executes the recommendation information processing method and the recommendation information ranking method described in the above embodiments.

Those skilled in the art will appreciate that the disclosure of the present disclosure is susceptible to numerous variations and modifications. For example, the various devices or components described above may be implemented in hardware, or may be implemented in software, firmware, or a combination of some or all of the three.

Furthermore, as used in this disclosure and in the claims, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are inclusive in the plural, unless the context clearly dictates otherwise. The use of "first," "second," and similar terms in this disclosure is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. Similarly, the word "comprising" or "comprises", and the like, means that the element or item preceding the word comprises the element or item listed after the word and its equivalent, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

Furthermore, flowcharts are used in this disclosure to illustrate the operations performed by systems according to embodiments of the present disclosure. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or one or more operations may be removed from the processes.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

While the present disclosure has been described in detail above, it will be apparent to those skilled in the art that the present disclosure is not limited to the embodiments described in the present specification. The present disclosure can be implemented as modifications and variations without departing from the spirit and scope of the present disclosure defined by the claims. Accordingly, the description of the present specification is for the purpose of illustration and is not intended to be in any way limiting of the present disclosure.

Claims

1. A recommendation information processing method, comprising:

determining a plurality of attributes from an attribute set associated with target recommendation information as a plurality of candidate aggregation dimensions based on historical delivery data of a predetermined recommendation information set containing the target recommendation information in a historical delivery period, and determining a priority of each candidate aggregation dimension in the plurality of candidate aggregation dimensions;

selecting a reference aggregation dimension of the target recommendation information from the plurality of candidate aggregation dimensions based at least on the priority of each of the plurality of candidate aggregation dimensions;

aggregating the estimated conversion amount and the actual conversion amount of the reference recommendation information in the reference recommendation information set under the reference aggregation dimension corresponding to the target recommendation information to obtain the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set before the current moment in the current delivery cycle;

determining calibration parameters based at least on the total predicted conversion and the total actual conversion of the reference recommendation information set; and

and calibrating the estimated conversion rate corresponding to the click of the target recommendation information at the current moment by using the calibration parameters.

2. The recommendation information processing method of claim 1, wherein determining a plurality of attributes from a set of attributes associated with the target recommendation information as a plurality of candidate aggregation dimensions based on historical placement data for a historical placement period for a predetermined set of recommendation information including the target recommendation information, and determining a priority for each of the plurality of candidate aggregation dimensions comprises:

the method comprises the steps of determining a plurality of attributes from an attribute set associated with target recommendation information as a plurality of candidate aggregation dimensions by solving an objective function of a dimension optimization model based on historical delivery data of a preset recommendation information set containing the target recommendation information in a historical delivery period, and determining the priority of each candidate aggregation dimension in the plurality of candidate aggregation dimensions.

3. The recommendation processing method of claim 2, wherein the objective function of the dimension optimization model is a cumulative sum of calibrated deviations of historical predicted conversions of predetermined recommendations of the predetermined recommendation set over historical delivery cycles,

wherein determining a plurality of attributes from the set of attributes associated with the target recommendation information as a plurality of candidate aggregation dimensions by solving an objective function of the dimension optimization model, and determining a priority for each of the plurality of candidate aggregation dimensions comprises:

randomly selecting a plurality of attributes from a set of attributes associated with the target recommendation information as a plurality of initial aggregation dimensions, each of the plurality of initial aggregation dimensions having a set priority;

calibrating the historical estimated conversion rate corresponding to each historical click of each piece of preset recommendation information in the preset recommendation information set in a historical release period by using the plurality of initial aggregation dimensions and the set priority;

calculating the calibration deviation of the calibrated historical predicted conversion amount and the historical actual conversion amount of each piece of preset recommendation information, and counting the accumulated sum of the calibration deviations of the predicted conversion amounts of the preset recommendation information in the preset recommendation information set in the historical release period as the target function; and

determining a plurality of initial aggregation dimensions that minimize the objective function as the plurality of candidate aggregation dimensions, and determining the set priority of each of the plurality of initial aggregation dimensions that minimize the objective function as the priority of each corresponding candidate aggregation dimension.

4. The recommendation information processing method of claim 3, wherein calibrating the historical predicted conversion rate corresponding to each historical click of each piece of predetermined recommendation information in the set of predetermined recommendation information using the plurality of initial aggregation dimensions and the set priority comprises:

for each predetermined recommendation information in the set of predetermined recommendation information:

selecting an initial reference aggregation dimension from the plurality of initial aggregation dimensions based at least on the set priority of each initial aggregation dimension;

determining a historical calibration parameter at least based on historical total estimated conversion and historical total actual conversion of the reference recommendation information set under the initial reference aggregation dimension corresponding to the preset recommendation information in a historical release period; and

and calibrating the historical estimated conversion rate corresponding to each historical click of the preset recommendation information in the historical release period by using the historical calibration parameters.

5. The recommendation information processing method of claim 1, wherein selecting a reference aggregation dimension of the target recommendation information from the plurality of candidate aggregation dimensions based at least on the priority of each of the plurality of candidate aggregation dimensions comprises:

acquiring total delivery consumption of a recommendation information set corresponding to the target recommendation information and under each candidate aggregation dimension in the plurality of candidate aggregation dimensions; and

and determining a candidate aggregation dimension with a total delivery consumption of the recommendation information sets larger than a first preset threshold and a highest priority as a reference aggregation dimension of the target recommendation information.

6. The recommended information processing method of claim 1, wherein, prior to determining calibration parameters based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommended information set, the recommended information processing method further comprises:

determining that the target recommendation information is in the initial stage or mature stage of release of the current release period at the current moment based on a predetermined rule,

when the target recommendation information is in an initial delivery stage, the recommendation information processing method further comprises the following steps: acquiring total estimated click rate and total actual click rate of the reference recommendation information set before the current moment in the current putting period, and

when the target recommendation information is in the release maturity stage, the recommendation information processing method further comprises the following steps: and acquiring the estimated conversion amount, the actual conversion amount, the estimated click amount and the actual click amount of the target recommendation information before the current moment in the current putting period.

7. The recommendation information processing method of claim 6, wherein determining that the target recommendation information is at an initial stage of delivery or a mature stage of delivery of a current delivery cycle at a current time based on a predetermined rule comprises:

determining current delivery consumption and current conversion amount of the target recommendation information generated before the current moment in the current delivery cycle;

and when the current delivery consumption is less than or equal to a second preset threshold and the current conversion amount is less than or equal to a third preset threshold, determining that the target recommendation information is in the initial delivery stage, otherwise, determining that the target recommendation information is in the mature delivery stage.

8. The recommendation information processing method of claim 6, wherein determining calibration parameters based on at least the total predicted conversion amount and the total actual conversion amount of the reference recommendation information set when the target recommendation information is in an initial stage of delivery comprises:

determining the calibration parameter based on at least a total predicted conversion amount, a total actual conversion amount, a total predicted click amount and a total actual click amount of the reference recommendation information set.

9. The recommendation information processing method of claim 8, wherein determining the calibration parameter based at least on a total predicted conversion amount, a total actual conversion amount, a total predicted click amount, and a total actual click amount of the reference recommendation information set comprises:

acquiring current release consumption of the target recommendation information generated before the current moment in the current release period, and determining an expansion coefficient based on the current release consumption; and

determining the calibration parameters based on the expansion coefficient and the total predicted conversion amount, the total actual conversion amount, the total predicted click amount and the total actual click amount of the reference recommendation information set.

10. The recommendation information processing method of claim 6, wherein determining calibration parameters based on at least the total pre-estimated conversion amount and the total actual conversion amount of the reference recommendation information set when the target recommendation information is at a delivery maturity stage comprises:

and determining the calibration parameters at least based on the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set, and the estimated conversion amount, the actual conversion amount, the estimated click rate and the actual click rate of the target recommendation information before the current moment in the current putting cycle.

11. The recommendation processing method of claim 10, wherein determining the calibration parameters based on at least a total predicted conversion amount and a total actual conversion amount of the reference recommendation information set, and a predicted conversion amount, an actual conversion amount, a predicted click-through amount, and an actual click-through amount of the target recommendation information prior to the current time within the current delivery cycle comprises:

acquiring historical actual conversion amount of the target recommendation information in a historical putting period, and determining a smoothing coefficient based on the historical actual conversion amount of the target recommendation information and the time length from the current time to the current putting period;

and determining the calibration parameters based on the smoothing coefficient, the total estimated conversion amount and the total actual conversion amount of the reference recommendation information set, and the estimated conversion amount, the actual conversion amount, the estimated click rate and the actual click rate of the target recommendation information before the current moment in the current putting cycle.

12. The recommendation information processing method of claim 1, wherein the set of attributes includes at least a home attribute subset, an audience attribute subset, and a content attribute subset of the target recommendation information, and wherein determining a plurality of attributes from the set of attributes associated with the target recommendation information as the plurality of candidate aggregation dimensions comprises:

determining at least one attribute from the attribution attributes subset, the audience attributes subset, and the content attributes subset, respectively, as a candidate aggregation dimension of the plurality of candidate aggregation dimensions.

13. A recommendation information ranking method comprises the following steps:

acquiring a calibrated estimated conversion rate of each piece of recommendation information to be released in a plurality of pieces of recommendation information to be released;

calculating the estimated profit of each recommendation information to be released in the plurality of recommendation information to be released according to the calibrated estimated conversion rate of each recommendation information to be released in the plurality of recommendation information to be released; and

sorting the plurality of pieces of recommendation information to be released based on the estimated profit of each piece of recommendation information to be released, sequentially releasing the plurality of pieces of recommendation information to be released according to the sorting result,

wherein the obtaining of the calibrated pre-estimated conversion rate of each piece of recommendation information to be delivered in the plurality of pieces of recommendation information to be delivered includes:

calibrating the estimated conversion rate of each recommendation information to be released in the plurality of recommendations information to be released by using the method according to any one of claims 1-12 to obtain the calibrated estimated conversion rate of each recommendation information to be released in the plurality of recommendations information to be released.

14. A recommended information processing apparatus comprising:

a dimension determination unit configured to determine a plurality of attributes from an attribute set associated with target recommendation information as a plurality of candidate aggregation dimensions based on historical delivery data of a predetermined recommendation information set containing the target recommendation information in a historical delivery period, and determine a priority of each candidate aggregation dimension in the plurality of candidate aggregation dimensions;

a selection unit configured to select a reference aggregation dimension from the plurality of candidate aggregation dimensions based at least on the priority of each of the plurality of candidate aggregation dimensions;

the aggregation unit is configured to aggregate the estimated conversion amount and the actual conversion amount of the reference recommendation information in the reference recommendation information set under the reference aggregation dimension, which correspond to the target recommendation information, to obtain a total estimated conversion amount and a total actual conversion amount of the reference recommendation information set before the current moment in the current delivery cycle;

a calibration parameter determination unit configured to determine calibration parameters based on at least said total predicted conversion amount and said total actual conversion amount of said reference recommendation information set; and

the calibration unit is configured to calibrate the estimated conversion rate corresponding to the click of the target recommendation information at the current moment by using the calibration parameter.

15. A recommendation information processing apparatus comprising:

one or more processors; and

one or more memories having computer-readable code stored therein, which when executed by the one or more processors, causes the one or more processors to perform the method of any one of claims 1-12.