CN112070530A

CN112070530A - Online evaluation method and related device of advertisement prediction model

Info

Publication number: CN112070530A
Application number: CN202010872195.4A
Authority: CN
Inventors: 胡乐
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-08-26
Filing date: 2020-08-26
Publication date: 2020-12-11

Abstract

The application provides an online evaluation method and a related device of an advertisement prediction model, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a characteristic data set of an advertisement, and dividing the advertisement from which special data in the characteristic data set comes into a training group and an evaluation group, wherein the characteristic data set comprises a training data set and a prediction data set; obtaining a sub-training dataset corresponding to the training group from the training dataset and a sub-prediction dataset corresponding to the evaluation group from the prediction dataset; training an online advertisement prediction model and an online advertisement prediction model by using the sub-training data set, and predicting by using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model; and comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model. The embodiment of the application effectively improves the accuracy of online evaluation of the advertisement prediction model.

Description

Online evaluation method and related device of advertisement prediction model

Technical Field

The application relates to the technical field of artificial intelligence, in particular to an online evaluation method and a related device of an advertisement pre-estimation model.

Background

The estimation of the click rate and the conversion rate of the advertisement is carried out by an advertisement estimation model such as machine learning or deep learning. The to-be-online advertisement estimation model optimized by the online advertisement estimation model and the relevant characteristics for evaluating the advertisement need to be evaluated offline before the model is online, so that whether the advertisement estimation model can be normally used online is determined by the offline evaluation result.

A large amount of Identification (ID) features are unavoidably added to feature data used in the estimation of the advertisement estimation model, and can generate better influence on the model when the data is sufficient, but the features easily cause the model to be over-fitted and interfere the effect of the model in the research and estimation of other features, so that the online estimation result of the advertisement estimation model is inaccurate.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The embodiment of the application aims to provide an online evaluation method and device of an advertisement prediction model, which can effectively improve the accuracy of online evaluation of the advertisement prediction model.

According to an embodiment of the application, an online evaluation method of an advertisement pre-estimation model can comprise the following steps: acquiring a feature data set of an advertisement, and dividing the advertisement from which the feature data in the feature data set comes into a training group and an evaluation group, wherein the feature data set comprises a training data set and a prediction data set; obtaining a sub-training dataset corresponding to the training set from the training dataset and a sub-prediction dataset corresponding to the evaluation set from the prediction dataset; training an online advertisement prediction model and an online advertisement prediction model by using the sub-training data set, and predicting by using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model; and comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

According to an embodiment of the present application, an online evaluation apparatus for an advertisement prediction model may include: the system comprises a dividing module, a judging module and a judging module, wherein the dividing module is used for acquiring a characteristic data set of an advertisement and dividing the advertisement from which the characteristic data in the characteristic data set comes into a training group and an evaluation group, and the characteristic data set comprises a training data set and a prediction data set; an obtaining module configured to obtain a sub-training dataset corresponding to the training group from the training dataset and obtain a sub-prediction dataset corresponding to the evaluation group from the prediction dataset; the prediction module is used for training an online advertisement prediction model and an online advertisement prediction model by using the sub-training data set and predicting by using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model; and the first evaluation module is used for comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

In some embodiments of the present application, the partitioning module is configured to: clustering the advertisements by using the characteristic data contained in the characteristic data set to obtain a plurality of advertisement clustering clusters; and dividing the advertisement cluster into two groups to obtain the training group and the evaluation group.

In some embodiments of the present application, the first evaluation module is configured to: calculating a first AUC result corresponding to the online advertisement prediction model according to the first prediction result; calculating a second AUC result of the to-be-online advertisement prediction model according to the second prediction result; and comparing the first AUC result with the second AUC result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

In some embodiments of the present application, the first evaluation module is configured to: acquiring the proportion of each advertisement in the sub-prediction data set in a source sorting stage, and determining a weighting coefficient corresponding to each advertisement according to the proportion, wherein the sorting stage indicates a stage of evaluating and sorting the advertisements by applying an advertisement pre-estimation model in an advertisement display life cycle; calculating a first sub AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the sub prediction data set in the first prediction result; and calculating the weighted sum of the first sub-AUC results corresponding to all the advertisements in the sub-prediction data set based on the weighting coefficient corresponding to each advertisement to obtain the first AUC result.

In some embodiments of the present application, the first evaluation module is configured to: calculating a second sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the sub-prediction data set in the second prediction result; and calculating the weighted sum of the second sub-AUC results corresponding to all the advertisements in the sub-prediction data set based on the weighting coefficient corresponding to each advertisement to obtain the second AUC result.

In some embodiments of the present application, further comprising a second evaluation module configured to: training an online advertisement prediction model and an online advertisement prediction model by using the training data set, and predicting by using the prediction data set to obtain a third prediction result of the online advertisement prediction model and a fourth prediction result of the online advertisement prediction model; obtaining the proportion of each advertisement in the prediction data set in a source sorting stage, and determining a weighting coefficient corresponding to each advertisement according to the proportion, wherein the sorting stage indicates a stage of applying an advertisement pre-estimation model to evaluate and sort the advertisements in an advertisement display life cycle; and calculating a third AUC result according to the third prediction result and the weighting coefficient, and calculating a fourth AUC result according to the fourth prediction result and the weighting coefficient, so as to obtain a second online evaluation result of the to-be-online advertisement prediction model by comparing the third AUC result with the fourth AUC result.

In some embodiments of the present application, the second evaluation module is configured to: calculating a third sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the prediction data set in the third prediction result; and calculating the weighted sum of the third sub-AUC results corresponding to all the advertisements in the prediction data set based on the weighting coefficient corresponding to each advertisement to obtain the third AUC result.

In some embodiments of the present application, the second evaluation module is configured to: calculating a fourth sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the fourth prediction result in the prediction data set; and calculating the weighted sum of the fourth sub-AUC results corresponding to all the advertisements in the prediction data set based on the weighting coefficient corresponding to each advertisement to obtain the fourth AUC result.

According to another embodiment of the present application, an electronic device may include: a memory storing computer readable instructions; a processor reading computer readable instructions stored by the memory to perform the method as described above.

According to another embodiment of the present application, a computer program medium having computer readable instructions stored thereon, which, when executed by a processor of a computer, cause the computer to perform the method as described above.

According to another embodiment of the present application, a computer program product or computer program comprises computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various alternative implementations described above.

According to the embodiment of the application, a feature data set of an advertisement is obtained, and the advertisement in the feature data set is divided into a training group and an evaluation group, wherein the feature data set comprises a training data set and a prediction data set; obtaining a sub-training dataset corresponding to the training group from the training dataset and a sub-prediction dataset corresponding to the evaluation group from the prediction dataset; because the training data set and the prediction data set are feature data of the same advertisement, the advertisement is grouped, and then the training group and the evaluation group are respectively obtained from the training data set and the prediction data set, so that the sub-training data set and the sub-prediction data set can be feature data of different advertisements, and the sub-training data set and the sub-prediction data set have different Identification (ID) class features.

Then, training an online advertisement prediction model and an online advertisement prediction model by using the sub-training data set, and predicting by using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model; the method can effectively enhance the evaluation of the feature generalization by enabling the Identification (ID) features of the advertisements corresponding to the training and prediction processes of the model to be different, and further comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

Other features and advantages of the present application will be apparent from the following detailed description, taken in conjunction with the accompanying drawings, or may be learned by practice of the application.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

FIG. 1 shows a schematic diagram of a system to which embodiments of the present application may be applied.

FIG. 2 shows a flow diagram of a method for online evaluation of an advertisement forecast model according to an embodiment of the present application.

FIG. 3 illustrates a flow diagram of a method of comparing a first prediction to a second prediction according to one embodiment of the present application.

FIG. 4 illustrates an advertisement presentation lifecycle flow diagram according to one embodiment of the present application.

FIG. 5 shows a flowchart of an online evaluation method of an advertisement prediction model under a scenario according to the application.

FIG. 6 illustrates an advertisement percentage profile according to an embodiment of the present application.

FIG. 7 is a flowchart illustrating an online evaluation method of an advertisement forecast model in another scenario according to the present application.

FIG. 8 shows a block diagram of an online evaluation apparatus of an advertisement forecast model according to an embodiment of the present application.

FIG. 9 shows a block diagram of an electronic device according to an embodiment of the application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the subject matter of the present application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the application.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

FIG. 1 shows a schematic diagram of a system 100 to which embodiments of the present application may be applied.

As shown in fig. 1, the system 100 may include a server 101 and a terminal 102.

The server 101 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, middleware service, a domain name service, a security service, a CDN, a big data and artificial intelligence platform, and the like. The server 101 may perform online evaluation of the advertisement prediction model, and may also deploy the advertisement prediction model, such as a click-through rate prediction model or a conversion rate prediction model.

The terminal 102 may be a mini-server or an edge device such as a smart phone, raspberry pi, camera, smart watch, etc. The terminal 102 may display the ordered advertisements in the server 101, and the terminal 102 may also deploy an advertisement prediction model, such as a click-through rate prediction model or a conversion rate prediction model.

In particular, in the present exemplary embodiment, the server 101 may provide an artificial intelligence cloud service, for example, provide an online advertisement prediction model for some platforms, and perform click through rate prediction of advertisements; clustering services, such as clustering advertisements, may also be provided. The so-called artificial intelligence cloud Service is also generally called AIaaS (AI as a Service, chinese). The method is a service mode of an artificial intelligence platform, and particularly, the AIaaS platform splits several types of common AI services and provides independent or packaged services at a cloud. This service model is similar to the one opened in an AI theme mall: all developers can access one or more artificial intelligence services provided by the platform through an API (application programming interface), and part of the qualified developers can also use an AI framework and an AI infrastructure provided by the platform to deploy and operate and maintain the self-dedicated cloud artificial intelligence services.

The terminal 102 and the server 101 may be directly or indirectly connected through wireless communication, and the application is not limited in this respect.

In one embodiment of this example, as shown in fig. 1, the server 101 may obtain a feature data set of an advertisement, and divide the advertisement in the feature data set into a training group and an evaluation group, where the feature data set includes a training data set and a prediction data set; obtaining a sub-training dataset corresponding to the training group from the training dataset and a sub-prediction dataset corresponding to the evaluation group from the prediction dataset; training an online advertisement prediction model and an online advertisement prediction model by using the sub-training data set, and predicting by using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model; and comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

FIG. 2 schematically shows a flow diagram of a method for online evaluation of an advertisement forecast model according to an embodiment of the present application. The execution subject of the online evaluation method of the advertisement prediction model may be an electronic device having a calculation processing function, such as the server 101 or the terminal 102 shown in fig. 1.

As shown in fig. 2, the online evaluation method of the advertisement forecast model may include steps S210 to S240.

Step S210, acquiring a feature data set of the advertisement, and dividing the advertisement in the feature data set into a training group and an evaluation group, wherein the feature data set comprises a training data set and a prediction data set;

step S220, obtaining a sub-training data set corresponding to the training group from the training data set, and obtaining a sub-prediction data set corresponding to the evaluation group from the prediction data set;

step S230, training an online advertisement prediction model and an online advertisement prediction model by using the sub-training data set, and predicting by using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model;

step S240, comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

The following describes the specific processes of each step performed when online evaluation of the advertisement prediction model is performed.

In step S210, a feature data set of the advertisement is obtained, and the advertisement from which the feature data in the feature data set originates is divided into a training set and an evaluation set, where the feature data set includes a training data set and a prediction data set.

In the embodiment of the present example, the feature data set of the advertisement is a feature data set obtained according to features of the advertisement in service requirements such as an online advertisement deployment requirement, and the like, and is used for evaluating an estimation model of the advertisement to be online, wherein one part (training data set) is used for training the advertisement evaluation model, and the other part (prediction data set) is used for predicting the advertisement evaluation model, that is, the training data set and the prediction data set are different data under the same features of the same advertisement. The feature data sets are related feature data from an exposure phase, a click phase and a conversion phase of the advertisement. The characteristics of the advertisement include Identification (ID) characteristics such as a product name and characteristics such as an advertisement size.

The feature data set for an advertisement includes feature data for many advertisements, which may include, for example, 1000 advertisements. Further, the advertisements in the feature dataset are divided into a training set and an evaluation set, for example, 1000 advertisements are divided into a training set (including 600 advertisements) and an evaluation set (including 400 advertisements).

In one embodiment, the dividing the advertisements from which the feature data in the feature data set originates into a training set and an evaluation set includes:

and randomly dividing advertisements from which the feature data in the feature data set are sourced into two groups to obtain a training group and an evaluation group.

clustering the advertisements by using the characteristic data contained in the characteristic data set to obtain a plurality of advertisement clustering clusters;

and dividing the advertisement cluster into two groups to obtain a training group and an evaluation group.

Clustering advertisements using feature data included in the feature data set may use kmeans clustering to cluster advertisements, for example, clustering 1000 advertisements into 100 advertisement clusters, so that the advertisements in each advertisement cluster have similarity. The clustering method can also be other clustering algorithms. The feature data sets are input into a preset clustering model (such as a kmeans clustering model), and the clustering model can cluster the advertisements according to the feature data contained in the feature data sets.

The advertisement clusters are divided into two groups, for example, 100 advertisement clusters are divided into a training group (including 50 advertisement clusters) and an evaluation group (including 50 advertisement clusters).

In step S220, a sub-training data set corresponding to the training group is obtained from the training data set, and a sub-prediction data set corresponding to the evaluation group is obtained from the prediction data set.

In the embodiment of the present example, a sub-training data set corresponding to the training group is obtained from the training data set, that is, an intersection advertisement between the advertisement in the training group and the advertisement in the training data set is obtained, and then, feature data of the intersection advertisement in the training data set is obtained, so as to obtain the sub-training data set corresponding to the training group.

And acquiring a sub-prediction data set corresponding to the evaluation group from the prediction data set, namely acquiring an intersection advertisement of the advertisement in the evaluation group and the advertisement in the prediction data set, and then acquiring characteristic data of the intersection advertisement in the prediction data set to obtain a sub-prediction data set corresponding to the evaluation group.

The training group and the evaluation group are respectively obtained from the training data set and the prediction data set, so that the sub-training data set and the sub-prediction data set can be feature data of different advertisements, and the sub-training data set and the sub-prediction data set can have different Identification (ID) features.

In step S230, the sub-training data set is used to train the online advertisement prediction model and the to-be-online advertisement prediction model, and the sub-prediction data set is used to perform prediction, so as to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the to-be-online advertisement prediction model.

In the embodiment of the example, the online advertisement prediction model is a deployed advertisement prediction model, and the to-be-online advertisement prediction model is an advertisement prediction model prepared to be deployed to replace the online advertisement prediction model.

The online advertisement estimation model and the online advertisement estimation model to be online are trained by using the sub-training data set, and the model can be trained according to the online evaluation requirement of the model, so that the trained online advertisement estimation model and the online advertisement estimation model to be online are obtained.

Then, the sub-prediction data set is used for predicting by utilizing the trained online advertisement prediction model and the to-be-online advertisement prediction model, so that a first prediction result predicted by the online advertisement prediction model and a second prediction result predicted by the to-be-online advertisement prediction model can be obtained.

The sub-training data set and the sub-prediction data set have different Identification (ID) features, so that the Identification (ID) features of advertisements corresponding to the training and prediction processes of the model are different, and the use effect of other generalized features except the Identification (ID) features is enhanced.

The first prediction result and the second prediction result can be click rate or conversion rate of the advertisement, and the online advertisement prediction model and the to-be-online advertisement prediction model can be a click rate prediction model or a conversion rate prediction model.

In step S240, the first prediction result and the second prediction result are compared to obtain a first online evaluation result of the to-be-online advertisement prediction model.

In the embodiment of the example, the first prediction result and the second prediction result can respectively and accurately reflect the use effects of the online advertisement prediction model and the to-be-online advertisement prediction model under the acquired feature data set, and further, the first online evaluation result of the to-be-online advertisement prediction model can be obtained by comparing the first prediction result with the second prediction result.

And a first online evaluation result of the to-be-online advertisement prediction model, such as the advertisement profit brought by the online advertisement prediction model after the to-be-online advertisement prediction model is online, is improved or is not changed compared with the online advertisement prediction model, and the like.

And when the second prediction result is better than the first prediction result in accuracy, the advertisement revenue brought by the online of the to-be-online advertisement prediction model is determined to be improved compared with the online advertisement prediction model.

In one embodiment, as shown in fig. 3, the step S240 of comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the pre-estimation model of the to-be-online advertisement, includes:

step S310, calculating a first AUC result corresponding to the online advertisement estimation model according to the first prediction result;

step S320, calculating a second AUC result of the pre-estimation model of the advertisement to be online according to the second prediction result;

and S330, comparing the first AUC result with the second AUC result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

The AUC, namely area under curve, has the physical meaning of area under the ROC curve, and the closer to 1, the better the prediction effect of the model is. The abscissa of the ROC curve is the ratio of the positive case to the negative case in the prediction, and the ordinate is the ratio of the positive case to the positive case in the prediction. The AUC results are equivalent to: randomly extracting a positive sample and a negative sample from the samples, wherein the positive sample obtains the probability that the score is larger than the negative sample. Higher AUC results indicate more effective features in the model architecture of the advertisement forecast model or the feature data set used by the model.

The first prediction result or the second prediction result is a positive sample, and the first prediction result or the second prediction result is an accurate prediction result; the first prediction result or the second prediction result being a negative sample may be embodied as the first prediction result or the second prediction result being an erroneous prediction result, specifically, the first prediction result or the second prediction result being accurate, that is, the first prediction result or the second prediction result is consistent with the label corresponding to the feature data in the prediction data set or the sub-prediction data set, and conversely, the first prediction result or the second prediction result is inconsistent with the label corresponding to the feature data in the prediction data set or the sub-prediction data set, that is, the first prediction result or the second prediction result is an error.

Further, a first AUC result corresponding to the online advertisement prediction model can be calculated according to the first prediction result; and calculating a second AUC result of the pre-estimated model of the advertisement to be online according to the second prediction result.

Comparing the first AUC result and the second AUC result may be calculating a difference between the first AUC result and the second AUC result, or calculating a lifting ratio of the second AUC result relative to the first AUC result. And calculating the promotion ratio of the second AUC result to the first AUC result, for example, the first AUC result is 0.672531, the second AUC result is 0.681854, and the promotion ratio of the second AUC result to the first AUC result is + 1.4%, so as to illustrate that the advertisement revenue brought after the online estimation model of the online advertisement is promoted compared with the online estimation model of the online advertisement.

In one embodiment, calculating a first AUC result corresponding to the online advertisement prediction model based on the first prediction result comprises:

acquiring the ratio of each advertisement in the sub-prediction data set in the source sorting stage, determining a weighting coefficient corresponding to each advertisement according to the ratio, and indicating the stage of evaluating and sorting the advertisements by applying an advertisement pre-estimation model in the advertisement display life cycle in the sorting stage;

calculating a first sub AUC result corresponding to each advertisement according to the corresponding prediction result of each advertisement in the first prediction result in the sub-prediction data set;

and calculating the weighted sum of the first sub-AUC results corresponding to all the advertisements in the sub-prediction data set based on the weighting coefficient corresponding to each advertisement to obtain a first AUC result.

Referring to fig. 4, in the embodiment of the present example, the advertisement presentation lifecycle includes: the method comprises a retrieval recall stage (which can be a stage of searching commodities by a user), a rough arrangement stage (which can be a stage of initially sequencing by applying an advertisement estimation model according to shallow feature data such as app downloading and form filling of advertisements), a fine arrangement stage (which can be a stage of precisely sequencing by applying the advertisement estimation model according to deep feature data such as purchased commodities and registered members of advertisements), an exposure stage (which can be a stage of exposing the first sequenced advertisements on an advertisement display page), a node clicking (which can be a stage of clicking the exposed advertisements by the user), and a conversion stage (which can be a stage of converting after clicking the advertisements by the user). In the embodiment of the present example, the advertisement prediction model is a click-through rate prediction model or a conversion rate prediction model applied to the rough ranking stage. It is understood that the advertisement prediction model may also be a click-through rate prediction model or a conversion rate prediction model applied to the refinement stage.

Acquiring the ratio of each advertisement in the sub-prediction data set in the source sorting stage, and determining the weighting coefficient corresponding to each advertisement according to the ratio, wherein the ratio can be acquired in the source sorting stage and then is used as the weighting coefficient of the advertisement; alternatively, a duty ratio range in which the duty ratio is located may be determined in the weighting coefficient table, and then the weighting coefficient to which the duty ratio range is mapped may be determined. For example, the sub-prediction dataset (from the click-through, conversion phase) includes A, B ads, with 80% of the a ads and 20% of the B ads in the coarse phase, with 80% as the weighting factor for the a ads and 20% as the weighting factor for the B ads.

And calculating a first sub-AUC result corresponding to each advertisement according to the corresponding prediction result of each advertisement in the sub-prediction data set in the first prediction result, for example, calculating the first sub-AUC result AUC of 0.7 according to the prediction result of the advertisement a, and calculating the first sub-AUC result AUC of 0.8 according to the prediction result of the advertisement B.

Further, a weighted sum of first sub-AUC results corresponding to all advertisements in the sub-prediction dataset is calculated based on the weighting coefficient corresponding to each advertisement, and a first AUC result is obtained, for example, 0.7 × 80% +0.8 × 20% ═ 0.72 (first AUC result).

The real use scene of the advertisement estimation model is in the sequencing stage, and the estimated probability is click and conversion probability, so that the characteristic data of the advertisement is click and conversion data. In order to measure the evaluation accuracy and the sorting effect of the advertisement prediction model on each advertisement in the sorting stage, the weighting is carried out by using the ratio of the sorting stage in online evaluation, so that the obtained result is more consistent with the actual online situation. For example, in the rough ranking stage, the ad a accounts for 80%, the ad B accounts for 20%, and in the click stage, the ad a and the ad B each account for 50%, which is likely that the ad B is an ad with narrow targeting but high click conversion rate, so that the exposure probability is high, the occupation ratio of each ad in the source ranking stage in the sub-prediction data set is obtained, and the weighting coefficient corresponding to each ad is determined according to the occupation ratio for weighting, so that the situation can be effectively avoided. On the basis of the evaluation of the enhanced feature generalization, the distribution difference between the distribution of the evaluation data (feature data set) and the distribution of the real prediction can be further reduced, so that the model evaluation is optimized, and the accuracy of the online evaluation result of the first model is further ensured.

In one embodiment, calculating a second AUC result corresponding to the pre-estimation model of the online advertisement according to the second prediction result includes:

calculating a second sub AUC result corresponding to each advertisement according to the corresponding prediction result of each advertisement in the sub prediction data set in the second prediction result;

and calculating the weighted sum of the second sub-AUC results corresponding to all the advertisements in the sub-prediction data set based on the weighting coefficient corresponding to each advertisement to obtain a second AUC result.

Calculating a second sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the sub-prediction data set in the second prediction result, for example, the second sub-AUC result AUC is 0.8 calculated according to the prediction result of the advertisement a, and the second sub-AUC result is 0.6 calculated according to the prediction result of the advertisement B; a weighted sum of second sub-AUC results for all ads in the sub-prediction dataset is calculated based on the weighting factor for each ad, resulting in a second AUC result, e.g., 0.8 × 80% +0.6 × 20% ═ 0.76 (second AUC result).

In one embodiment, the method further comprises:

training an online advertisement prediction model and an online advertisement prediction model by using a training data set, and predicting by using a prediction data set to obtain a third prediction result of the online advertisement prediction model and a fourth prediction result of the online advertisement prediction model;

obtaining the ratio of each advertisement in the forecast data set in the source sorting stage, determining the weighting coefficient corresponding to each advertisement according to the ratio, and indicating the stage of evaluating and sorting the advertisements by applying an advertisement pre-estimation model in the advertisement display life cycle in the sorting stage;

and calculating a third AUC result according to the third prediction result and the weighting coefficient, and calculating a fourth AUC result according to the fourth prediction result and the weighting coefficient, so as to obtain a second online evaluation result of the to-be-online advertisement prediction model by comparing the third AUC result with the fourth AUC result.

In the embodiment of the example, the advertisement is not grouped, the model is evaluated by using the data of the complete set, that is, the online advertisement prediction model and the to-be-online advertisement prediction model are trained by directly using the training data set, and the prediction is performed by using the prediction data set, so that the third prediction result of the online advertisement prediction model and the fourth prediction result of the to-be-online advertisement prediction model are obtained.

Then, a third AUC result and a fourth AUC result under the prediction data set are obtained, a second online evaluation result of the to-be-online advertisement prediction model is obtained by comparing the third AUC result and the fourth AUC result, the distribution difference between the distribution of the evaluation data (characteristic data set) and the real prediction can be reduced under the full data, and further the model evaluation is optimized.

And then the online effect of the to-be-online advertisement estimation model can be comprehensively evaluated according to the first online evaluation result and the second online evaluation result.

In one embodiment, calculating a third AUC result according to the third predicted result and the weighting factor includes:

calculating a third sub AUC result corresponding to each advertisement according to the corresponding prediction result of each advertisement in the third prediction result in the prediction data set;

and calculating the weighted sum of the third sub-AUC results corresponding to all the advertisements in the prediction data set based on the weighting coefficient corresponding to each advertisement to obtain a third AUC result.

Calculating a third sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the prediction data set among the third prediction results, for example, the third sub-AUC result AUC calculated according to the prediction result of the advertisement a is 0.8, and the third sub-AUC result calculated according to the prediction result of the advertisement B is 0.6; a weighted sum of the third sub-AUC results for all ads in the prediction dataset is calculated based on the weighting factor for each ad, resulting in a third AUC result, e.g., 0.8 x 80% +0.6 x 20% ═ 0.76 (third AUC result).

In one embodiment, calculating the fourth AUC result according to the fourth predicted result and the weighting factor includes:

calculating a fourth sub AUC result corresponding to each advertisement according to the corresponding prediction result of each advertisement in the fourth prediction result in the prediction data set;

and calculating the weighted sum of the fourth sub-AUC results corresponding to all the advertisements in the prediction data set based on the weighting coefficient corresponding to each advertisement to obtain a fourth AUC result.

Calculating a fourth sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the prediction data set among the fourth prediction results, for example, the third sub-AUC result AUC is 0.8 calculated according to the prediction result of the a advertisement, and the fourth sub-AUC result AUC is 0.6 calculated according to the prediction result of the B advertisement; a weighted sum of the fourth sub-AUC results for all ads in the prediction dataset is calculated based on the weighting factor for each ad, resulting in a fourth AUC result, e.g., 0.8 × 80% +0.6 × 20% ═ 0.76 (fourth AUC result).

In one embodiment, the first top-line evaluation result indicates that the model effect is significantly improved (e.g., the improvement ratio or difference between the second AUC result and the first AUC result is greater than a first predetermined threshold), the second top-line evaluation result indicates that the model effect is not significantly improved (e.g., the improvement ratio or difference between the third AUC result and the fourth AUC result is less than a second predetermined threshold), which indicates that the added advertisement feature in the feature data set has good generalization, the method has the advantages that the characteristic data of the new advertisement (which can be the advertisement on line within a preset time period, such as the advertisement on line within 1 day) is extracted from the prediction data set or the sub-prediction data set, the evaluation is carried out based on the embodiment for obtaining the first on-line evaluation result again, if the newly obtained first on-line evaluation result shows obvious improvement, the result accords with the guess, and the on-line experiment is carried out on the estimation model of the advertisement to be on line.

In one embodiment, the first upper-line evaluation result indicates that the model effect is not significantly improved (e.g., the improvement ratio or the difference between the second AUC result and the first AUC result is smaller than a first predetermined threshold), the second upper-line evaluation result indicates that the model effect is significantly improved (e.g., the improvement ratio or the difference between the third AUC result and the fourth AUC result is greater than a second predetermined threshold), which indicates that the newly added advertisement features in the feature data set are helpful for the exposure of the advertisement, the feature data of the advertisement with less start-up amount (which may be the advertisement with upper line outside the predetermined time period and with the exposure less than the predetermined size, such as the advertisement with the upper line greater than 1 day and the exposure less than 10000) in the prediction data set or the sub-prediction data set can be extracted, the evaluation is performed based on the embodiment of obtaining the second upper-line evaluation result, and if the newly obtained second upper-line evaluation, and (5) carrying out online experiment on the pre-estimated model of the advertisement to be online.

In one embodiment, if the first online evaluation result indicates that the model effect is significantly improved (for example, the improvement ratio or the difference between the second AUC result and the first AUC result is greater than a first predetermined threshold), and the second online evaluation result indicates that the model effect is significantly improved (for example, the improvement ratio or the difference between the third AUC result and the fourth AUC result is greater than a second predetermined threshold), the to-be-online advertisement prediction model is directly online tested.

In one embodiment, the method for obtaining the weighting coefficient corresponding to each advertisement in the prediction data set or the sub-prediction data set further includes: when the advertisement prediction model is a click rate prediction model, the ratio of the advertisements in the source sorting stage is used as a weighting coefficient corresponding to each advertisement; when the advertisement prediction model is a conversion rate prediction model, using advertiser-side data (namely the rate of commodity purchase quantity on an advertiser site) as a weighting coefficient corresponding to each advertisement; this may reduce the impact of individual differences in data from the advertiser's perspective.

In one example, in the first related art scheme, when the model effect is investigated off-line, the existing on-line advertisement estimation model is used as the reference group, the to-be-on-line advertisement estimation model is used as the experimental group, under the same investigation environment, the training data set is used to train the on-line advertisement estimation model and the to-be-on-line advertisement estimation model, then, using the prediction data set to perform prediction to obtain a first prediction result corresponding to the on-line advertisement prediction model and a second prediction result corresponding to the to-be-on-line advertisement prediction model, where the training data set and the prediction data set are different data of the same advertisement under the same characteristics, for example, as shown with reference to fig. 5, in the first related art scheme a, collecting characteristic data sets corresponding to the target advertisement sets as training data sets every day from 1 month and 1 day to 1 month and 4 days, then, the feature data set corresponding to the target advertisement set of day 5/1 month is used as the prediction data set. Then, a first AUC result is obtained by calculation according to the first prediction result, a second AUC result is obtained by calculation according to the second prediction result, and the lifting proportion of the second AUC result relative to the first AUC result is calculated, so that a first upper line evaluation result is determined.

By applying the first scheme of the embodiment of the present application, when the effect of the model is investigated offline, the existing online advertisement estimation model is used as a reference group, the online advertisement estimation model is used as an experimental group, and under the same investigation environment, the advertisements in the feature data set (including the training data set and the prediction data set) are divided into the training group and the evaluation group, then the sub-training data set corresponding to the training group is obtained from the training data set, and the sub-prediction data set corresponding to the evaluation group is obtained from the prediction data set, for example, as shown in fig. 5, in the first scheme B of the embodiment of the present application, the feature data set corresponding to the target advertisement set is collected as the training data set every day from 1 month 1 day to 1 month 4 days, and the feature data set corresponding to the target advertisement set from 1 month 5 days is used as the prediction data set. Then, the target advertisement set is divided into a training group and an evaluation group, a sub-training data set (refer to a shaded area in fig. 5) corresponding to the training group is obtained from the training data set, and a sub-prediction data set (refer to a shaded area in fig. 5) corresponding to the evaluation group is obtained from the prediction data set. Then, using the sub-training data set to train the online advertisement prediction model and the to-be-online advertisement prediction model, and then using the sub-prediction data set to predict to obtain a first prediction result corresponding to the online advertisement prediction model and a second prediction result corresponding to the to-be-online advertisement prediction model; and then, calculating to obtain a first AUC result according to the first prediction result, calculating to obtain a second AUC result according to the second prediction result, and calculating the lifting proportion of the second AUC result relative to the first AUC result, thereby judging a first upper-line evaluation result.

The following table shows the comparative data of the first AUC result and the second AUC result obtained by applying the first related technical solution and the first scheme of the embodiment of the present application in this example.

In the first related technical solution, due to the influence of the Identification (ID) features, the promotion ratio of the second AUC result to the first AUC result is only + 0.05%, which results in a limited effect (the first online evaluation result) in determining the model architecture of the new to-be-online advertisement estimation model or the advertisement features used by the model, and further influences the online evaluation of the model. By applying the first scheme of the embodiment of the application, the promotion proportion of the second AUC result relative to the first AUC result is calculated to be + 1.4% through the evaluation of the reinforced feature generalization, so that the effect of the model architecture of the new estimation model of the advertisement to be online or the advertisement features used by the model is judged to be good (the first online evaluation result), and the actual online experimental verification is matched with the actual online condition.

Referring to FIG. 6, the study shows the distribution of the advertisement occupation ratio in the rough-ranking stage and the click-through stage, wherein the abscissa is the occupation ratio of the advertisement in the click-through stage, the distribution is mainly concentrated in the range of 0-0.4 x 10 (-7), and the ordinate is the occupation ratio of the advertisement in the rough-ranking stage, the distribution is mainly distributed in the range of 0-0.1 x 10 (-6). It can be seen that the advertisements are more evenly distributed during the coarse-row phase and sparser in the click phase.

Generally, the real use scene of the advertisement prediction model is in the rough ranking stage, and the prediction is the probability of conversion after clicking, so the characteristic data set for evaluating the advertisement prediction model is clicking stage data.

Further, in another example, in the second related technical solution, the online advertisement prediction model and the to-be-online advertisement prediction model are applied to the rough ranking stage, and the feature data set of the advertisement is data in the click stage; and when the AUC result is calculated according to the prediction result, the weighting coefficient corresponding to each advertisement adopts the proportion of the advertisement in the click stage. At this time, referring to fig. 7, in the second related art scenario C, the advertisement 410 accounts for 20% and the advertisement 420 accounts for 80% in the rough stage, but each of the advertisement 410 and the advertisement 420 accounts for 50% in the click stage (in this case, it is highly likely that the advertisement 410 is an advertisement with a narrow advertisement target but a high click conversion rate, and is easily converted by exposure to a click). Because the weighting coefficients corresponding to the advertisement 410 and the advertisement 420 adopt the ratio of the advertisement in the click stage, it is assumed that the evaluation result AUC of the advertisement estimation model for the advertisement 410 is 0.8, and the evaluation result AUC for the advertisement 420 is 0.6, and since the advertisement 410 and the advertisement 420 each account for 50% of the click data, the integrated AUC is 0.7.

By applying the second scheme of the embodiment of the present application, when calculating the AUC result according to the prediction result, the weighting coefficient corresponding to each advertisement adopts the percentage of the advertisement in the source ranking stage (specifically, the percentage in the rough ranking stage). Referring to fig. 7, in the second scheme D, since the weighting coefficients corresponding to the advertisement 410 and the advertisement 420 adopt the ratio of the advertisement in the click stage, it is assumed that the evaluation result AUC of the advertisement predictive model for the advertisement 410 is 0.8, the evaluation result AUC for the advertisement 420 is 0.6, and since the advertisement 410 and the advertisement 420 each account for 50% of the click data, the overall AUC is 0.76.

The following table shows comparative data of AUC results obtained by applying the second related art scheme and the second scheme of the example of the present application in this example.

In the second related technical scheme, due to the influence of real data distribution, the improvement ratio of the calculated AUC result is 0.001195, so that the effect of judging the model architecture of the new to-be-online advertisement estimation model or the advertisement characteristics used by the model is limited, and further the online evaluation of the model is influenced. By applying the second scheme of the embodiment of the application, the distribution difference between the distribution of the evaluation data and the distribution of the real prediction is effectively reduced by adopting the advertisement occupation ratio in the rough arrangement stage as the weighting coefficient, and the promotion ratio of the AUC result is calculated to be 0.00466, so that the effect of judging the model architecture of the new estimation model of the advertisement to be online or the advertisement characteristics used by the model is good, and the model architecture is matched with the actual online condition through the verification of the actual online experiment.

As shown in fig. 8, the online evaluation apparatus 500 of the advertisement prediction model may include a partitioning module 510, an obtaining module 520, a predicting module 530, and a first evaluating module 540.

The dividing module 510 may be configured to obtain a feature data set of an advertisement, and divide the advertisement from which the feature data in the feature data set originates into a training group and an evaluation group, where the feature data set includes a training data set and a prediction data set;

the obtaining module 520 may be configured to obtain a sub-training data set corresponding to the training group from the training data set, and obtain a sub-prediction data set corresponding to the evaluation group from the prediction data set;

the prediction module 530 may be configured to train an online advertisement prediction model and an online advertisement prediction model using the sub-training data set, and perform prediction using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model;

the first evaluation module 540 may be configured to compare the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the application. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

FIG. 9 schematically shows a block diagram of an electronic device according to an embodiment of the application.

It should be noted that the electronic device 600 shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 9, the electronic apparatus 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data necessary for system operation are also stored. The CPU601, ROM602, and RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 505 is also connected to bus 604.

The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN (local area network) card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.

In particular, according to embodiments of the present application, the processes described below with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. When the computer program is executed by a Central Processing Unit (CPU)601, various functions defined in the system of the present application are executed.

It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method described in the above embodiments.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains.

It will be understood that the present application is not limited to the embodiments that have been described above and shown in the drawings, but that various modifications and changes can be made without departing from the scope thereof.

Claims

1. An online evaluation method of an advertisement prediction model is characterized by comprising the following steps:

acquiring a feature data set of an advertisement, and dividing the advertisement from which special data in the feature data set comes into a training group and an evaluation group, wherein the feature data set comprises a training data set and a prediction data set;

obtaining a sub-training dataset corresponding to the training set from the training dataset and a sub-prediction dataset corresponding to the evaluation set from the prediction dataset;

training an online advertisement prediction model and an online advertisement prediction model by using the sub-training data set, and predicting by using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model;

and comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

2. The method of claim 1, wherein the dividing the advertisements from which the feature data in the feature data set originates into a training set and an evaluation set comprises:

and dividing the advertisement cluster into two groups to obtain the training group and the evaluation group.

3. The method of claim 1, wherein comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the pre-estimation model of the to-be-online advertisement, comprises:

calculating a first AUC result corresponding to the online advertisement prediction model according to the first prediction result;

calculating a second AUC result corresponding to the to-be-online advertisement prediction model according to the second prediction result;

and comparing the first AUC result with the second AUC result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

4. The method of claim 3, wherein calculating a first AUC result corresponding to the online advertisement prediction model according to the first prediction result comprises:

acquiring the proportion of each advertisement in the sub-prediction data set in a source sorting stage, and determining a weighting coefficient corresponding to each advertisement according to the proportion, wherein the sorting stage indicates a stage of evaluating and sorting the advertisements by applying an advertisement pre-estimation model in an advertisement display life cycle;

calculating a first sub AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the sub prediction data set in the first prediction result;

and calculating the weighted sum of the first sub-AUC results corresponding to all the advertisements in the sub-prediction data set based on the weighting coefficient corresponding to each advertisement to obtain the first AUC result.

5. The method of claim 4, wherein calculating a second AUC result corresponding to the pre-estimation model of the to-be-online advertisement according to the second prediction result comprises:

calculating a second sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the sub-prediction data set in the second prediction result;

and calculating the weighted sum of the second sub-AUC results corresponding to all the advertisements in the sub-prediction data set based on the weighting coefficient corresponding to each advertisement to obtain the second AUC result.

6. The method of claim 1, further comprising:

training an online advertisement prediction model and an online advertisement prediction model by using the training data set, and predicting by using the prediction data set to obtain a third prediction result of the online advertisement prediction model and a fourth prediction result of the online advertisement prediction model;

obtaining the proportion of each advertisement in the prediction data set in a source sorting stage, and determining a weighting coefficient corresponding to each advertisement according to the proportion, wherein the sorting stage indicates a stage of applying an advertisement pre-estimation model to evaluate and sort the advertisements in an advertisement display life cycle;

7. The method of claim 6, wherein said calculating a third AUC result based on said third predicted result and said weighting factor comprises:

calculating a third sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the prediction data set in the third prediction result;

and calculating the weighted sum of the third sub-AUC results corresponding to all the advertisements in the prediction data set based on the weighting coefficient corresponding to each advertisement to obtain the third AUC result.

8. The method of claim 7, wherein said calculating a fourth AUC result based on said fourth predicted result and said weighting factor comprises:

calculating a fourth sub-AUC result corresponding to each advertisement according to a corresponding prediction result of each advertisement in the fourth prediction result in the prediction data set;

and calculating the weighted sum of the fourth sub-AUC results corresponding to all the advertisements in the prediction data set based on the weighting coefficient corresponding to each advertisement to obtain the fourth AUC result.

9. An online evaluation device of an advertisement prediction model is characterized by comprising:

the system comprises a dividing module, a judging module and a judging module, wherein the dividing module is used for acquiring a characteristic data set of an advertisement and dividing the advertisement from which the characteristic data in the characteristic data set comes into a training group and an evaluation group, and the characteristic data set comprises a training data set and a prediction data set;

an obtaining module configured to obtain a sub-training dataset corresponding to the training group from the training dataset and obtain a sub-prediction dataset corresponding to the evaluation group from the prediction dataset;

the prediction module is used for training an online advertisement prediction model and an online advertisement prediction model by using the sub-training data set and predicting by using the sub-prediction data set to obtain a first prediction result of the online advertisement prediction model and a second prediction result of the online advertisement prediction model;

and the first evaluation module is used for comparing the first prediction result with the second prediction result to obtain a first online evaluation result of the to-be-online advertisement prediction model.

10. An electronic device, comprising: a memory storing computer readable instructions; a processor reading computer readable instructions stored by the memory to perform the method of any of claims 1-8.