WO2022111291A1

WO2022111291A1 - Recommendation information evaluation method, apparatus and device, and computer readable storage medium

Info

Publication number: WO2022111291A1
Application number: PCT/CN2021/130006
Authority: WO
Inventors: 肖小范; 陈龙; 李宥壑
Original assignee: 北京沃东天骏信息技术有限公司
Priority date: 2020-11-27
Filing date: 2021-11-11
Publication date: 2022-06-02
Also published as: CN112435064A

Abstract

Embodiments of the present application provide a recommendation information evaluation method, apparatus and device, and a computer readable storage medium. The method comprises: obtaining, from a recommendation information delivery platform, recommendation information for an object to be recommended and object information of the object; inputting the recommendation information and the object information into a trained document scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension, the dimensions comprising a theme, a compliance degree, an attraction, and smoothness; and determining an evaluation result of the recommendation information on the basis of the scoring result of the recommendation information in each dimension.

Description

Evaluation method, apparatus, device and computer-readable storage medium for recommendation information

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number of 202011362739.9, the application date of November 27, 2020, and the application title of "Recommendation Information Evaluation Method, Apparatus, Equipment and Computer-readable Storage Medium", and requests the Chinese patent application The priority of the Chinese patent application is incorporated herein by reference.

technical field

The present application relates to the technical field of computer applications, and relates to, but is not limited to, a method, apparatus, device, and computer-readable storage medium for evaluating recommendation information.

Background technique

Advertising content is an important element of advertising, which is related to the conversion rate of products and the spread of brands. The importance of advertising copy as the carrier of advertising content is evident. In a comprehensive online shopping mall that sells over tens of thousands of brands and tens of millions of products, and needs to place millions of advertisements every day, how to accurately and objectively evaluate the millions of advertisements in the system, and determine an It is very important to determine whether the advertisement copy is a low-quality advertisement to determine whether the advertisement needs to be filtered out, so as to reduce unnecessary advertisement expenses and reduce the operating cost of the enterprise.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present application provide a method, apparatus, device, and computer-readable storage medium for evaluating recommendation information.

The technical solutions of the embodiments of the present application are implemented as follows:

The embodiment of the present application provides a method for evaluating recommendation information, and the method includes:

Obtain the recommendation information of the object to be recommended and the object information of the object to be recommended from the recommendation information delivery platform;

Inputting the recommendation information and the object information into the trained copywriting scoring model for evaluation, and obtaining the scoring results of the recommendation information in each dimension, the dimensions including subjectivity, compliance, attractiveness, and smoothness;

The evaluation result of the recommendation information is determined based on the scoring results of the recommendation information in each dimension.

An embodiment of the present application provides a device for evaluating recommendation information, and the device includes:

a first obtaining module, configured to obtain recommendation information of the object to be recommended and object information of the object to be recommended from the recommendation information delivery platform;

The evaluation module is configured to input the recommendation information and the object information into the trained copywriting scoring model for evaluation, and obtain the scoring results of the recommendation information in each dimension, and the dimensions include subjectivity, compliance, attractiveness strength and smoothness;

A determination module configured to determine an evaluation result of the recommendation information based on the scoring results of the recommendation information in each dimension.

The embodiment of the present application provides an evaluation device for recommended information, including:

processor; and

a memory configured to store a computer program executable on the processor;

Wherein, when the computer program is executed by the processor, the steps of the above-mentioned evaluation method for recommendation information are implemented.

Embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions, where the computer-executable instructions are configured to execute the steps of the foregoing method for evaluating recommendation information.

Embodiments of the present application provide a method, apparatus, device, and computer-readable storage medium for evaluating recommendation information, wherein the method includes: acquiring recommendation information of an object to be recommended and an object of the object to be recommended from a recommendation information delivery platform information; input the recommendation information and the object information into the trained copywriting scoring model for evaluation, and obtain the scoring results of the recommendation information in each dimension, the dimensions include subjectivity, compliance, attractiveness and smoothness The evaluation result of the recommendation information is determined based on the scoring results of the recommendation information in each dimension. In this way, an objective and multi-dimensional quantitative evaluation of the recommended information can be realized, which can improve evaluation efficiency and evaluation accuracy, shorten evaluation time, reduce evaluation cost, and reduce evaluation risk.

Description of drawings

In the drawings, which are not necessarily to scale, like reference numerals may describe like parts in the different views. The accompanying drawings generally illustrate, by way of example and not limitation, the various embodiments discussed herein.

FIG. 1 is a schematic flowchart of a realization of a method for evaluating recommendation information provided by an embodiment of the present application;

2 is a schematic diagram of a dictionary tree constructed by an evaluation method for recommendation information provided by an embodiment of the present application;

3 is a schematic diagram of a dictionary tree-based search model constructed by the method for evaluating recommendation information provided by an embodiment of the present application;

FIG. 4 is a schematic flowchart of another implementation of the method for evaluating recommendation information provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of the implementation principle of the evaluation method for advertising creative copy provided by the embodiment of the present application;

FIG. 6 is a schematic diagram of training of an advertisement copy theme scoring model provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of training of an advertisement copy theme compliance degree model provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of training of an advertising copy topic attractiveness model provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of training of an advertising copy subject naturalness model provided by an embodiment of the present application;

Figure 10 is a schematic diagram of the scores of three copywriting in each dimension under the ICAN model;

Figure 11 is a radar chart of the scores of the three copywriting in each dimension under the ICAN model;

FIG. 12 is a schematic diagram of the composition and structure of a device for evaluating recommendation information provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of the composition and structure of an evaluation device for recommendation information provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings. All other embodiments obtained under the premise of creative work fall within the scope of protection of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same or a different subset of all possible embodiments, and Can be combined with each other without conflict.

In the following description, the term "first\second\third" is only used to distinguish similar objects, and does not represent a specific ordering of objects. It is understood that "first\second\third" is used in Where permitted, the specific order or sequence may be interchanged to enable the embodiments of the application described herein to be practiced in sequences other than those illustrated or described herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

In order to better understand the embodiments of the present application, an evaluation method of recommendation information in the related art is first described.

The evaluation methods of recommended information in related technologies are as follows: Method 1: Evaluation method based on evaluators, this method mainly relies on evaluators' professional knowledge background, personal experience, etc. to evaluate recommended information, such as selecting experts who can represent consumers' attitudes Evaluate ad copy. Method 2: Based on the evaluation method of questionnaire survey, a questionnaire is designed in combination with the content of the object to be recommended, and the appropriate interviewees are screened according to the audience attributes of the object to be recommended. The interviewee evaluates the form, style, appeal point, and understanding of the recommended information. and so on, and select recommended information that may have an ideal effect for actual delivery. Method 3: Evaluation method based on the actual delivery effect. This method requires the actual delivery of the recommended information, based on the monitored clicks, impressions, costs and other related indicators to evaluate the recommended information, and iteratively revise to optimize the recommended information. , to improve the recommendation effect.

This embodiment of the present application provides a method for evaluating recommendation information. The methods provided by the embodiments of the present application may be implemented by a computer program, and when the computer program is executed, each step in the method for evaluating the recommendation information provided by the embodiments of the present application is completed. In some embodiments, the computer program may be executed by a processor in an evaluation device for recommendation information. FIG. 1 is a schematic flowchart of an implementation of a method for evaluating recommendation information provided by an embodiment of the present application. As shown in FIG. 1 , the method includes the following steps:

Step S101: Obtain recommendation information of the object to be recommended and object information of the object to be recommended from the recommendation information delivery platform.

The embodiment of the present application takes the recommendation information as an advertisement copy as an example, and the recommendation information delivery platform is described as an advertisement delivery platform. The steps of the method provided in the embodiment of the present application may be implemented by an advertising copy evaluation device. The advertising copy evaluation equipment establishes a connection relationship with the advertising delivery platform. Before an advertisement placement platform places an advertisement, in order to ensure the quality of the advertisement to be placed, the evaluation device of the advertisement copy needs to evaluate the to-be-placed advertisement, so as to determine whether the advertisement copy to be placed is normally placed according to the evaluation result. The advertising copy evaluation device first obtains the advertisement copy of the advertisement to be placed, and the object information corresponding to the advertisement copy. Here, it should be noted that the advertisement copy and the object information corresponding to the advertisement copy may be information of the same object or information of different objects. When the ad copy and the object information correspond to the same object, it indicates that the object described by the ad copy and the object described by the object information are the same object, that is, the ad copy matches the object information; when the ad copy and the object information correspond to different objects, it indicates that the object described by the ad copy The object described by the object information is a different object, that is, the ad copy does not match the object information. For example, if the advertisement copy is "a teapot that Dad will like", the object information is "a transparent glass teapot for chrysanthemum tea", and the described objects are all "teapot", in this case, the advertisement copy and the object information corresponding to the advertisement copy are the same The information of the object; if the advertisement copy is "Teapot that Dad will like", the object information is "Premium tea, a special product of Yunnan before the Ming Dynasty", the object described in the advertisement copy is "Teapot", and the object described in the object information is "Tea", at this time , the advertisement copy and the object information corresponding to the advertisement copy are information of different objects.

In step S102, the recommendation information and the object information are input into the trained copywriting scoring model for evaluation, and the scoring results of the recommendation information in each dimension are obtained.

Here, dimensions include topicality, compliance, attractiveness, and fluency. The embodiment of the present application proposes a copywriting scoring model ICAN. The copywriting scoring model ICAN considers at least subjectivity (I, Integrated), compliance (C, Compliance), attractiveness (A, Appeal) and naturalness (N, Natural) (also called fluency) dimensions, pre-train the proposed copywriting scoring model to obtain the trained copywriting scoring model ICAN. Input the information to be evaluated into the trained copywriting scoring model ICAN, and obtain the scoring results of multiple dimensions of subject I, compliance C, attractiveness A, and smoothness N. In the embodiment of the present application, the advertising copy evaluation device performs multi-dimensional quantitative evaluation on the advertisement copy to be placed in the advertisement, which can ensure the objectivity of the scoring result, and can improve the evaluation accuracy by considering the multi-dimensional scoring result. Compared with other evaluation methods, it can improve evaluation efficiency, shorten evaluation time, reduce evaluation cost, and reduce evaluation risk.

Step S103: Determine an evaluation result of the recommendation information based on the scoring results of the recommendation information in each dimension.

In an implementation manner, after obtaining the scoring results of each dimension, the scoring results of the recommended information are determined according to the respective scoring results; it is judged whether the scoring results of the recommended information are greater than the first preset threshold; When the preset threshold is set, the evaluation result of the recommended information is determined to be passed; when the scoring result of the recommended information is less than or equal to the first preset threshold, the evaluation result of the recommended information is determined to be unsuccessful. An implementation method of determining the scoring result of the recommended information according to each scoring result is: when determining the scoring result of the recommended information according to the scoring results of each dimension, it can be determined in combination with the radar chart, and the scoring results of each dimension are formed in the radar chart. The area of the quadrilateral is determined as the scoring result of the recommended information. The scoring result of the recommendation information determined by this implementation is determined based on the integrity of multiple dimensions. The scoring result of the recommended information is determined based on the overallity, which facilitates subsequent overall adjustment and optimization of the recommended information, or direct elimination of the recommended information with a lower overall score.

In another implementation manner, after obtaining the scoring results of each dimension, calculate the variance of the thematic scoring results, the compliance scoring results, the attractiveness scoring results and the naturalness scoring results; and determine whether the variance is less than a second preset threshold; When the variance is smaller than the second preset threshold, it is further judged whether there is at least one scoring result in the thematic scoring result, the compliance scoring result, the attractiveness scoring result and the fluent scoring result that is greater than the third preset threshold; when the thematic scoring result When at least one of the results, the compliance score, the attractiveness score, and the fluent score is greater than the third preset threshold, it is determined that the evaluation result of the recommended information is the evaluation pass; when the variance is greater than or equal to the second preset The threshold value, or the subjectivity score result, compliance score result, attractiveness score result, and smoothness score result are all less than or equal to the third preset threshold, the evaluation result of the recommended information is determined to be an evaluation failure. Through this implementation, it is possible to filter out the recommendation information whose scoring result in a certain dimension is too low or the scoring result in a certain dimension is too high, so as to facilitate the subsequent adjustment and optimization of a certain dimension of the recommended information, or directly eliminate the scoring of each dimension Recommendations that vary widely.

In the evaluation method for recommendation information provided by the embodiment of the present application, the recommendation information of the object to be recommended and the object information of the object to be recommended are obtained from the recommendation information delivery platform; the recommendation information and the object information are input into the trained copywriting scoring model for evaluation, and the The scoring results of the recommended information in each dimension, including subjectivity, compliance, attractiveness, and smoothness; the evaluation results of the recommended information are determined based on the scoring results of the recommended information in each dimension. In this way, an objective and multi-dimensional quantitative evaluation of the recommended information can be realized, which can improve evaluation efficiency and evaluation accuracy, shorten evaluation time, reduce evaluation cost, and reduce evaluation risk.

In some embodiments, before step S102 of the embodiment shown in FIG. 1 , the evaluation method for recommendation information further includes the following steps:

Step S11 , obtaining the thematic sample set, the sensitive word set, the attractiveness sample set and the fluent degree sample set.

It should be noted that the obtained thematic sample set, attractiveness sample set, and commensurate sample set can be the same sample set, and the sample recommendation information and sample object information included in each sample set are the same. The difference lies in the training of different models. , the input information of different models is different. The recommendation information takes ad copy as an example. As the carrier of advertising content, advertising copy should convey the advertising content in a healthy and positive form, and guide consumers to establish correct values. Based on this, sensitive words that do not comply with laws and regulations are formed into a sensitive word set. When it is judged that the ad copy contains sensitive words, it is determined that the ad copy is not compliant.

Step S12, input the thematic sample set, the attractiveness sample set and the fluent degree sample set into the preset thematic network model, the preset attractive network model and the preset fluent network model, respectively, to obtain a trained thematic network model , the trained attractiveness network model and the trained smoothness network model.

Input the thematic sample set into the preset thematic network model to obtain a trained thematic network model. The trained thematic network model is configured to determine the recommendation information of the object to be recommended and the thematic score of the object information of the object to be recommended. In actual implementation, first determine the subject of the recommended information as the first subject, determine the subject of the object information as the second subject, then calculate the matching probability between the first subject and the second subject, and determine the matching probability as the subjectivity of the recommended information score. For example, when the topic matching probability of the subject of the ad copy and the object information is 1, the subject score of the ad copy is 1; when the subject match probability of the subject of the ad copy and the subject information is 0.1, the subject score of the ad copy is 0.1. Input the attractiveness sample set into the preset attractiveness network model to obtain the trained attractiveness network model. The trained attractiveness network model is configured to determine the attractiveness score of the recommended information. The greater the amount of information conveyed by the recommendation information, the more attractive the user is, that is, the greater the attraction. In informatics, a quantitative index for measuring the amount of information is called "information entropy", and in this embodiment of the present application, the information entropy of recommended information may be used to determine its attractiveness. For example, when recommending product advertisements, first preset a feature information set, which includes feature information such as product categories, discounts, and attribute words, and then input the advertisement copy into the trained attractive network model to obtain the advertisement copy. The probability distribution of , calculates the information entropy according to the probability distribution, and determines the information entropy as the attractiveness score of the advertisement copy. Input the smoothness sample set into the preset smoothness network model to obtain the trained smoothness network model. The trained smoothness network model is configured to determine the naturalness (ie, smoothness) score of the recommended information. Language model perplexity (PPL, Perplixity), that is, perplexity, is an indicator to measure the performance of language models. In this embodiment of the present application, the degree of confusion may be used to quantify the degree of smoothness of the recommended information. The lower the degree of confusion of the recommended information, the more natural the semantics of the recommended information, and the higher the degree of smoothness; or there are typos. In actual implementation, the preset PPL calculation formula can be used to calculate the confusion degree, and then based on the Chinese language model N-Gram, the calculated confusion degree of the recommended information is weighted and summed to obtain the smoothness of the recommended information.

Step S13, construct a search model based on dictionary tree according to the sensitive word set.

In some embodiments, step S13 can be implemented by the following steps:

Step S131, construct a dictionary tree according to each sensitive word in the sensitive word set. In actual implementation, first obtain the text data composed of all sensitive words, and divide different sensitive words into different lines; read the current line sensitive words, compare the current characters of the current line sensitive words with the child nodes of the current node, and find the the matched child nodes. If the search succeeds, take the found child node as the current node, and continue with the next character of the current line-sensitive word; if the search fails, insert a new child node into the current node, and use the new child node as the current node, and continue the current line-sensitive word the next character of the word. When all the sensitive words in the current line are searched, the next line of sensitive words is read, and the same operation is continued until the next line of sensitive words is read, and the last character of the last line of sensitive words is searched and stopped. The tree constructed at this time is a dictionary tree, also known as a trie tree. For example, the sensitive word set is {high h, high imitation, usury, simulation gun, real game}, and the constructed trie tree is shown in Figure 2.

Step S132, adding a query failure pointer to each node in the dictionary tree to obtain a lookup model based on the dictionary tree. Although the trie tree can be used for multi-pattern matching, backtracking is required every time the matching fails. If the pattern string is very long, it will be a waste of time. Based on this, after step S131, the embodiment of the present application continues to perform step S132, introducing multiple Modular matching algorithm AC automaton (Aho-Corasick automaton). The AC automaton adds a query failure pointer, that is, the fail pointer, on the basis of the tire tree. If the current node fails to match, the pointer is transferred to the place pointed by the fail pointer, so that the matching can continue without backtracking. In actual implementation, the construction of AC automata can be achieved by the following pseudocode:

1) Point the fail of all child nodes of the root node to the root node, and then list all the child nodes of the root node in sequence; //Here, fail is the query failure pointer.

2) If the queue is not empty:

2.1) Dequeue, dequeue the dequeued node as curr, failTo=curr.fail; //Here, failTo represents the node pointed to by fail of curr.

2.2) a. Determine whether curr.child[i]==failTo.child[i] is established;

Established: curr.child[i].fail = failTo.child[i];

invalid:

Determine whether failTo==null is established;

Established: curr.child[i].fail==root;

Not established: execute failTo=failTo.fail, and continue to step 2.2);

b.curr.child[i] is listed, continue to step 2);

3) If the queue is empty: end.

Still taking the above example to illustrate, adding a sensitive word query failure pointer to the dictionary tree shown in FIG. 2 , a search model based on the dictionary tree is obtained as shown in FIG. 3 . Input the advertisement copy to the search model shown in Figure 3, and get the compliance score result of the advertisement copy, which can be achieved by the following pseudocode: 1) Point the pointer of the current node to the root node of the AC automaton, that is, curr=root; 2) Read (next) a character from the text string of the advertisement copy; 3) Find a node matching the character from all child nodes of the current node; if the search is successful: determine whether the current node and the node pointed to by the current node fail Indicates the end of a character string, if so, record the index start point in the text string in the corresponding character string saving result set (index start point=current index-string length+1). Point curr to the child node, and continue to perform step 2); if the search fails: perform step 4). 4) If fail==null (indicating that no string in the target string is the prefix of the input string, which is equivalent to restarting the state machine), curr=root, continue to step 2); otherwise, point the pointer of the current node to fail node, continue to step 3).

Step S14, constructing a trained copywriting scoring model based on the trained thematic network model, the trained attractiveness network model, the trained smoothness network model and the search model. In this embodiment of the present application, a sample set is obtained, and a pre-proposed copywriting scoring model ICAN is trained to obtain a trained copywriting scoring model ICAN. The trained copywriting scoring model ICAN can perform thematic I, compliance degree C, attractiveness evaluation on the recommended information. The multi-dimensional quantitative evaluation of force A and natural degree N can ensure the objectivity of the scoring results. Considering the multi-dimensional scoring results, it can improve the evaluation accuracy. Compared with the evaluation methods of recommended information in related technologies, it can improve the evaluation efficiency, Shorten evaluation time, reduce evaluation cost, and reduce evaluation risk.

In some embodiments, in the above step S12, "inputting the thematic sample set into a preset thematic network model to obtain a trained thematic network model" can be implemented as the following steps:

Step S121: Obtain sample object information and sample recommendation information of each sample object in the thematic sample set. The sample object information refers to the description information of the sample object, and the sample recommendation information is the recommended content of the sample object. For example, the promoted product is "teapot", the sample object information is "transparent glass teapot for chrysanthemum tea", and the sample recommendation information is "teapot that dad will like".

In step S122, the sample object information and the sample recommendation information of the same sample object are regarded as a group of sample pairs, and the labeling information of the sample pairs is obtained. That is, "a transparent glass teapot for chrysanthemum tea" and "a teapot that dad will like" are used as a set of sample pairs (also called sample sentence pairs). information. During training, the evaluation device for the recommendation information can obtain the label information of the sample pair that is manually pre-labeled and saved from the storage device. Here, the annotation information represents the probability that the sample object information in the sample pair matches the sample recommendation information. For example, if the sample recommendation information in the sample pair describes "teapot", and the sample object information in the sample pair also describes "teapot", the probability that the sample recommendation information matches the sample object information is 1; If the recommendation information describes "teapot", and the sample object information in the sample pair describes "mobile phone", the probability that the sample recommendation information matches the sample object information is 0.

Step S123 , input each sample pair corresponding to each sample object in the thematic sample set and the labeling information of each sample pair into a preset thematic network model for training and learning, and obtain a trained thematic network model. Here, the trained thematic network model is used to determine and output the annotation information of the evaluation pair based on the input evaluation pair, so as to obtain the thematic scoring result of the recommended information. During implementation, the sample object information and sample recommendation information in the thematic sample set can be input to the preset thematic network model as sample pairs, and the annotation information of the sample pairs can be used as the annotation data of the preset thematic network model for transfer learning. Train to get the trained topic network model. Here, the preset topical network model can be a natural language processing BERT (Bidirectional Encoder Representations from Transformers) model. After K rounds of training on the thematic sample set (for example, set K=10), a trained thematic network model based on BERT is obtained, which is denoted as model F. The recommendation information of the object to be recommended and the object information of the object to be recommended constitute an evaluation pair, which is input into the trained thematic network model, and then the thematic score of the recommendation information can be generated. For example, assuming that the description of the product promoted by the advertisement copy Di is Ai (that is, the object information is Ai), after inputting the sentence pair Di and Ai into the model F, F outputs the probability ri related to the two, and ri can be used as the theme of the advertisement copy Di. score.

In some embodiments, in the above-mentioned step S12, "input the attractiveness sample set into the preset attractiveness network model to obtain a trained attractiveness network model", which can be implemented as the following steps:

Step S124: Obtain sample recommendation information of each sample object in the attractive sample set. In informatics, the quantitative index to measure the amount of information is called "information entropy". After seeing the recommended information, users receive new information, which increases their cognitive information entropy. For example, the user did not know that the teapot was on sale before, and when he saw the advertisement "Teapot that Dad will like, 25% off over 100", he learned that the price of the teapot was on sale. And the advertisement "a good thing that the old people like, I won't tell others", the amount of information provided to users is very small. Users always want to see informative advertisements, which is reflected in the advertisement copy, that is, there is a clear concept. Based on this, the embodiments of the present application use the information entropy of the recommendation information to evaluate its attractiveness.

Step S125: Perform information extraction on the sample recommendation information of each sample object to obtain a feature information set of each sample object. The feature information set includes at least one of the name, category, discount and attribute word of the sample object. For example, set the following feature information: category, discount, attribute word. Category, that is, the category information of the product, such as "mobile phone", "fresh food" and other features; discount, that is, the promotional information of the product, such as "full discount", "gift", "discount" and other features; attribute words, such as "red"","log","import","celebrity","summer" and other characteristics. Assuming N pieces of feature information (C ₁ , C ₂ , C ₃ , ..., C _N ), the probability that the ad copy D _i belongs to each feature is a vector

will be ignored. On the contrary, if the concept of D _i is clear, E _i will be relatively small, and users are more likely to be attracted by such ad copy.

Step S126, input the feature information set of each sample object into the preset attractiveness network model to obtain a trained attractiveness network model. The preset attractive network model may be a Magpie model, which is used to predict the probability that a certain recommendation information belongs to each feature.

A good recommendation message must be fluent and natural, read fluent and natural, concise and clear. Therefore, in the embodiment of the present application, when quantifying and evaluating the recommendation information, the fluency is introduced as a dimension of the recommendation information evaluation. In some embodiments, in the above-mentioned step S12, "inputting the fluidity sample set into the preset fluidity network model to obtain a trained fluidity network model" can be implemented as the following steps:

Step S127: Obtain sample recommendation information of each sample object in the fluent degree sample set.

Step S128: Perform word segmentation processing on the sample recommendation information of each sample object to obtain word segmentation of each sample recommendation information. In this embodiment of the present application, the degree of confusion may be used to quantify the degree of smoothness of the recommendation information. The lower the degree of confusion of the recommendation information, the more natural the semantics of the recommendation information, and the higher the degree of smoothness; otherwise, the recommendation information has semantically incomprehensible Happening. Obtain the word segmentation of each sample recommendation information, for example, perform word segmentation processing on the sample advertisement copy s, and obtain s=(w ₁ , w ₂ , ..., _wn ), where w _i represents the ith participle in the sample advertisement copy s, and n is The number of participles.

In step S129, the word segmentation of each sample recommendation information is input into a preset smoothness network model to obtain a trained smoothness network model. Here, the preset smoothness network model can be the Chinese language model N-Gram, which calculates the confusion degree by the word segmentation of the recommended information of each sample, and then uses the weighted summation to obtain the smoothness of the recommended information. After word segmentation is performed on the recommendation information of the recommended object, the confusion degree of the recommendation information is calculated, and the confusion degree ppl(s) can be calculated by the following formula (1):

The smoothness of the recommendation information is obtained by weighted summation. When the N-Gram language model is used in the embodiment of the present application, the value of N is 2, 3, and 4 as examples. For different N-Gram models, the formula for calculating the fluency f(s) of the recommendation information is shown in the following formula (2):

Among them, α _i is the weight value corresponding to the degree of confusion when N takes different values, so far, the degree of smoothness of the recommendation information of the object to be recommended is obtained.

In some embodiments, the above step S102 "input the recommendation information and object information into the trained copywriting scoring model for evaluation, and obtain the scoring results of the recommendation information in each dimension", can be implemented as the following steps:

Step S1021 , input the recommendation information and the object information as a set of evaluation pairs into the trained thematic network model, and obtain the thematic scoring result of the recommendation information.

In step S1022, the recommendation information is input into the search model, and a compliance score result of the recommendation information is obtained.

In step S1023, the recommendation information is input into the trained attractiveness network model, and the attractiveness score result of the recommendation information is obtained.

In step S1024, the recommendation information is input into the trained network model of smoothness, and the result of the smoothness score of the recommended information is obtained.

After obtaining the recommendation information and object information of the object to be recommended in step S101, the recommendation information and the object information are input into the trained thematic network model as a set of evaluation pairs to obtain the thematic score of the recommended information, and the recommendation information is respectively input From the search model, the trained attractiveness network model and the trained smoothness network model, the compliance score, attractiveness score and naturalness score of the recommendation information are obtained respectively, so as to obtain the scoring results of each dimension.

When the above-mentioned step S103 "determines the evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension", at least the following two implementations are included:

In the first implementation manner, the evaluation result of the recommendation information is determined based on the integrity of multiple dimensions. At this time, the above step S103 "determining the evaluation result of the recommendation information based on the scoring results of the recommendation information in each dimension" can be implemented as the following steps:

Step S103a1: Determine the scoring result of the recommended information according to the subjectivity scoring result, the compliance scoring result, the attractiveness scoring result, and the smoothness scoring result.

Step S103a2, judging whether the scoring result of the recommended information is greater than a first preset threshold. When the scoring result of the recommended information is greater than the first preset threshold, it indicates that the recommended information meets the delivery requirements, and the process goes to step S103a3; when the scoring result of the recommended information is less than or equal to the first preset threshold, it may be that the subject of the recommended information and the object information are related. The subject does not match, it may be that the recommended information contains sensitive words, or the amount of information contained in the recommended information is too small, or it may be that the recommended information sentence is not smooth, there are typos and other defects. At this time, it is determined that the recommended information does not meet the delivery requirements. Proceed to step S103a4.

Step S103a3, it is determined that the evaluation result of the recommended information is an evaluation pass.

Step S103a4, it is determined that the evaluation result of the recommended information is that the evaluation fails.

In the second implementation manner, the evaluation result of the recommendation information is determined based on the stability of multiple dimensions. At this time, the above step S103 "determining the evaluation result of the recommendation information based on the scoring results of the recommendation information in each dimension" can be implemented as the following steps:

Step S103b1: Calculate the variance of the subjectivity score results, the compliance score results, the attractiveness score results, and the naturalness score results.

Step S103b2, judging whether the variance is smaller than a second preset threshold. When the variance is less than the second preset threshold, it indicates that the scoring results of each dimension of the recommended information are relatively average, and then the process goes to step S103b3; Or the scoring result of a certain dimension is too high, in this case, it is determined that the recommended information does not meet the delivery requirements, and the process goes to step S103b5.

Step S103b3: Determine whether there is at least one scoring result greater than a third preset threshold in the thematic scoring result, the compliance scoring result, the attractiveness scoring result, and the smoothness scoring result. When at least one of the thematic scoring results, the compliance scoring results, the attractiveness scoring results, and the smoothness scoring results is greater than the third preset threshold, indicating that the recommended information satisfies the delivery results, step S103b4 is entered; At least one of the scoring results, compliance scoring results, attractiveness scoring results, and fluency scoring results does not exist greater than the third preset threshold, namely thematic scoring results, compliance scoring results, attractiveness scoring results, and smoothing If the degree score results are all less than or equal to the third preset threshold, it indicates that although the scores of the recommended information in each dimension are average, each score result is lower. At this time, it is considered that the recommended information does not meet the delivery results, and the process goes to step S103b5.

Step S103b4, it is determined that the evaluation result of the recommended information is the evaluation pass.

Step S103b5, it is determined that the evaluation result of the recommended information is that the evaluation fails.

In some embodiments, in step S103a4 or step S103b5, when it is determined that the evaluation result of the recommended information is that the evaluation fails, the method may further include:

Step S104: Adjust the recommendation information based on at least one of the subjectivity scoring results, the compliance scoring results, the attractiveness scoring results, and the smoothness scoring results. The recommendation information that fails the evaluation is adjusted and optimized, so that the evaluation result of the adjusted recommendation information is the evaluation pass.

In some embodiments, the method may further include:

In step S105, the evaluation result is sent to the recommendation information delivery platform, so that the recommendation information delivery platform delivers the evaluation result as the recommendation information that has passed the evaluation. The evaluation device of the recommendation information notifies the recommendation information delivery platform that the recommendation information of the objects to be recommended can be directly delivered. For the adjusted recommendation information, the adjusted recommendation information also needs to be sent to the recommendation information delivery platform, so that the recommendation information delivery platform can deliver the adjusted recommendation information.

In some embodiments, the evaluation device for recommendation information sends the evaluation result to the recommendation information delivery platform, and may also cause the recommendation information delivery platform to send prompt information, so that users who recommend objects to be recommended know which recommendation information cannot be delivered normally.

The embodiment of the present application further provides a method for evaluating recommendation information. FIG. 4 is a schematic flowchart of another implementation of the method for evaluating recommendation information provided by the embodiment of the present application. As shown in FIG. 4 , the method includes the following steps:

Step S401 , obtaining the thematic sample set, the sensitive word set, the attractiveness sample set and the fluent degree sample set.

Step S402: Obtain sample object information and sample recommendation information of each sample object in the thematic sample set.

In step S403, the sample object information and the sample recommendation information of the same sample object are regarded as a set of sample pairs, and the labeling information of the sample pairs is obtained. Here, the annotation information represents the probability that the sample object information in the sample pair matches the sample recommendation information.

Step S404 , input each sample pair corresponding to each sample object in the thematic sample set and the labeling information of each sample pair into a preset thematic network model for training and learning, and obtain a trained thematic network model.

Step S405, construct a dictionary tree according to each sensitive word in the sensitive word set.

Step S406, adding a query failure pointer to each node in the dictionary tree to obtain a lookup model based on the dictionary tree.

Step S407: Obtain sample recommendation information of each sample object in the attractive sample set.

Step S408: Perform information extraction on the sample recommendation information of each sample object to obtain a feature information set of each sample object.

Here, the feature information set includes at least one of the name, category, discount, and attribute word of the sample object.

Step S409, input the feature information set of each sample object into the preset attractiveness network model to obtain a trained attractiveness network model.

Step S410: Obtain sample recommendation information of each sample object in the fluent degree sample set.

Step S411: Perform word segmentation processing on the sample recommendation information of each sample object to obtain word segmentation of each sample recommendation information.

Step S412, inputting the word segmentation of each sample recommendation information into a preset smoothness network model to obtain a trained smoothness network model.

In step S413, a trained copywriting scoring model is constructed based on the trained thematic network model, the trained attractiveness network model, the trained smoothness network model and the search model.

In some embodiments, the above steps S401 to S413 may also be performed after step S414.

Step S414: Obtain recommendation information of the object to be recommended and object information of the object to be recommended from the recommendation information delivery platform.

Step S415 , input the recommendation information and the object information as a set of evaluation pairs into the trained thematic network model, and obtain the thematic scoring result of the recommendation information.

In step S416, the recommendation information is input into the search model, and the compliance score result of the recommendation information is obtained.

In step S417, the recommendation information is input into the trained attractiveness network model, and the attractiveness score result of the recommendation information is obtained.

In step S418, the recommendation information is input into the trained network model of smoothness, and the result of the smoothness score of the recommended information is obtained.

Step S419: Determine the scoring result of the recommended information according to the subjectivity scoring result, the compliance scoring result, the attractiveness scoring result, and the smoothness scoring result.

Step S420, judging whether the scoring result of the recommended information is greater than a first preset threshold.

When the scoring result of the recommended information is greater than the first preset threshold, it indicates that the recommended information meets the delivery requirements, and at this time, the process goes to step S421; when the scoring result of the recommended information is less than or equal to the first preset threshold, it indicates that the recommended information does not meet the delivery requirements request, go to step S422.

In some embodiments, the above steps 419 to S420 may be replaced by steps S419' to S421': step S419', calculating the variance of the subjectivity score results, compliance score results, attractiveness score results and naturalness score results . Step S420', judging whether the variance is less than the second preset threshold. When the variance is less than the second preset threshold, it indicates that the scoring results of each dimension of the recommended information are relatively average, and the process goes to step S421'; when the variance is greater than or equal to the second preset threshold, it indicates that the scoring result of the recommended information in a certain dimension is too low , or the scoring result in a certain dimension is too high, in this case, it is determined that the recommended information does not meet the delivery requirements, and the process goes to step S422. Step S421', judging whether there is at least one scoring result in the thematic scoring result, compliance scoring result, attractiveness scoring result, and smoothness scoring result that is greater than the third preset threshold. When at least one of the thematic scoring results, compliance scoring results, attractiveness scoring results, and smoothness scoring results is greater than the third preset threshold, indicating that the recommended information satisfies the delivery results, step S421 is entered; The scoring result, the compliance scoring result, the attractiveness scoring result, and the smoothness scoring result are all less than or equal to the third preset threshold, indicating that the recommendation information does not satisfy the delivery result, and the process proceeds to step S422.

Step S421, it is determined that the evaluation result of the recommended information is an evaluation pass.

Proceed to step S424, and deliver the recommended information that has passed the evaluation.

Step S422, it is determined that the evaluation result of the recommended information is that the evaluation fails.

Step S423, adjusting the recommendation information based on at least one of the subjectivity scoring result, the compliance scoring result, the attractiveness scoring result, and the naturalness scoring result. Adjust the subject, sensitive word, amount of information, or sentence of the recommended information, so that the evaluation result of the adjusted recommended information is the evaluation pass, thereby satisfying the delivery condition, and the process proceeds to step S424.

In step S424, the evaluation result is sent to the recommendation information delivery platform, and the recommendation information delivery platform delivers the evaluation result as the recommendation information that has passed the evaluation.

Below, an exemplary application of the embodiments of the present application in a practical application scenario will be described.

The embodiment of this application proposes a quantitative evaluation model for creative copywriting of e-commerce advertisements: ICAN.

ICAN comprehensively considers the four dimensions of compliance (C, Compliance), attractiveness (A, Appeal), theme (I, Integrated), and naturalness (N, Natural) of advertising creative copywriting. Multi-dimensional quantitative scoring.

FIG. 5 is a schematic diagram of the implementation principle of the evaluation method for advertising creative copy provided by the embodiment of the present application. As shown in FIG. 5 , the ICAN advertisement creative evaluation model proposed in the embodiment of the present application is mainly composed of four scoring sub-models: Degree scoring model, copywriting attractiveness scoring model, copywriting theme scoring model, copywriting naturalness scoring model. Each part is described in detail below.

Part 1, Copy Thematic Scoring Model:

The thematic nature of ad copy is to assess whether it is consistent with the advertised product. For example, an advertisement reads "a teapot that Dad will like", but after clicking into it, the user sees products such as electronic products and tea, which not only fails to achieve the effect of promoting the product, but also loses the user experience.

The embodiment of the present application adopts the sentence pair matching task of the BERT model to achieve the thematic score of the copy. FIG. 6 is a schematic diagram of training of an advertising copy theme scoring model provided by an embodiment of the present application. As shown in FIG. 6 , the training process of the copy theme scoring model is as follows:

1) From the existing products and their advertising copy, manually annotate some positive samples and negative samples to form a sample set S. The positive sample here means that the ad copy is semantically related to the description of the product it promotes; while the negative sample is semantically irrelevant.

2) The product description and advertising creative copy of the sample set S are used as the input of the BERT sentence pair relationship matching task, and whether the two are related is used as the labeling data of the model for transfer learning training.

3) After training S for K rounds (in this example, K=10), a BERT-based classifier is obtained, which is set as F.

4) For all existing advertisement copy and the description of each promoted product, form sentence pairs in pairs, and input them into model F to generate the theme score of each advertisement copy.

Assuming that the description of the product promoted by the advertisement copy _{Di is A i} ₍ that is, the object information is A _i ), after inputting the sentence pair _{Di and A i} _into the model F, F outputs the probability _ri related to the two, and _ri can be used as an advertisement Thematic score for copy D _i .

Part 2, Copywriting Compliance Scoring Model:

As the carrier of advertising content, advertising creativity should convey the advertising content in a healthy form and guide consumers to establish correct values. Therefore, compliance is also a very important and indispensable evaluation dimension when evaluating advertising creative copywriting. In the embodiment of this application, the Aho-Corasick algorithm can be used to score the compliance degree of the copy. Since it is necessary to strictly check and kill the creative copy of the advertisement containing sensitive words, the score of each copy is only 1 or 0. 1 means that no copy is found in the copy. Any sensitive words that do not comply with laws and regulations, that is, the copy is compliant; 0 means that there are sensitive words in the copy, so the copy is not compliant. Aho-Corasick is a classic multi-pattern string matching algorithm, which is widely used in pattern string matching scenarios with large text strings and many target strings, so it is suitable for compliance checking of creative copywriting. The construction of a copy sensitive word automaton to detect sensitive words in a copy includes the following three steps: constructing a sensitive word Trie tree (prefix) tree, adding a sensitive word query mismatch pointer to construct an AC automaton, pattern matching and returning the matching sensitive words.

(1) The algorithm steps for constructing the trie tree are as follows: 1) First, obtain all the text data and divide them into line-by-line form. 2) Read in each line of data, compare the current comparison character value with the child nodes of the current node, and find the matching node; 3) If the corresponding child node is found, take the child node as the current node, and remove the data. this character, continue with step 2). 4) If the corresponding child node is not found, insert the new node into the current node, and use the new node as the current node, and proceed to step 2). 5) The termination condition of the operation is that all characters in the data have been removed and the comparison is completed.

(2) The algorithm flow of constructing an AC automaton is as follows:

1) Point the fail of all child nodes of the root node to the root node, and then list all the child nodes of the root node in sequence. 2) If the queue is not empty: 2.1) Dequeue, dequeue the dequeued node as curr, and failTo represents the node pointed to by fail of curr, that is, failTo=curr.fail; 2.2) a. Judging curr.child[i]= =failTo.child[i] is established, established: curr.child[i].fail=failTo.child[i], not established: judge whether failTo==null is established; established: curr.child[i].fail== root; not established: execute failTo=failTo.fail, and continue to execute 2.2); b.curr.child[i] is listed, execute step 2) again; 3) If the queue is empty: end.

(3) The pattern matching operation process of the AC automaton is as follows: 1) The pointer indicating the current node points to the root node of the AC automaton, that is, curr=root; 2) Read (the next) character from the text string; 3) From the Find a node matching the character among all the child nodes of the current node, if successful: judge whether the current node and the node pointed to by the current node fail indicate the end of a string, if so, record the index starting point in the text string in the corresponding string Save the result set (index starting point = current index - string length + 1). curr points to the child node, and proceeds to step 2). If it fails: go to step 4). 4) If fail==null (indicating that no string in the target string is the prefix of the input string, which is equivalent to restarting the state machine) curr=root, go to step 2, otherwise, point the pointer of the current node to the fail node, execute step 3).

Assuming the existing set of sensitive words black_words={high h, high imitation, loan shark, imitation gun, real game}, the two ideas are: creative1="Still buying high imitation? Click here, the big bag is only one dollar!" , creative2 = "High-quality simulated bonsai, large quantity favors the best. Anti-ultraviolet, anti-wind pressure.", Figure 7 is a schematic diagram of the training of the advertising copy theme compliance model provided by the embodiment of the application, and Figure 7 shows two advertisement creatives The entire process of scoring compliance.

Part 3, Copywriting Attractiveness Scoring Model:

Advertising copy affects users’ minds by conveying information to users. Obviously, copy with a larger amount of information is more attractive to them. For example, "a good thing that old people like" is far less informative than "a teapot that dad will like". In informatics, the quantitative index to measure the amount of information is called "information entropy". After the user sees the copy information, he receives new information, which increases the information entropy of his cognition (that is, the original unclear cognition becomes clear). For example, the user did not know that the teapot was on sale before, and when he saw the advertisement "Teapot that Dad will like, 25% off over 100", he learned that the price of the teapot was on sale. And the advertisement "a good thing that old people like, I won't tell others", it provides very little information to users. Users always want to see informative advertisements, which is reflected in the copywriting, that is, there is a clear concept. Therefore, in the embodiments of the present application, the information entropy of the copy is used to evaluate its attractiveness. For example, set the following "concepts": product word, which is the category of the product, such as "mobile phone", "fresh food" and other concepts; benefit point, which is the promotion of the product, such as "full discount", "gift", "discount" and other concepts; attribute words, such as "red", "log", "import", "celebrity", "summer" and other concepts. Fake

relatively larger. When users see such copywriting, they are likely to be clueless and ignore it. On the contrary, if the concept of D _i is clear, E _i will be relatively small, and users are more likely to be attracted by such copy.

FIG. 8 is a schematic diagram of training the theme attractiveness model of advertisement copy provided by the embodiment of the present application. As shown in FIG. 8 , the training steps of the theme attractiveness model of copywriting are as follows: 1) Extract all products from the product library of the e-commerce website, including Their names, categories/brands, attribute words, and related promotions; 2) Use the name as text, category/brand, attribute words, promotion words, etc. as labels to generate a training text set; 3) Using the above training text set, train one more Label classifier, set to model M; such as using the Magpie model as M. Finally, the M model can be used to predict the probability that a copy belongs to each concept. In this way, when there is a new copy D _i , it is input into the model to obtain the probability distribution P _i , and the information entropy E _i is calculated to be the attractiveness score of the copy.

Part 4, Copy Naturalness Scoring Model:

A good copy must be smooth and natural, so that the audience can read it smoothly and naturally, concisely and clearly. Therefore, in the embodiment of the present application, when quantifying the creative copywriting of an advertisement, a score of the naturalness of the copywriting smoothness (corresponding to the smoothness degree above) is introduced as a dimension of the copywriting evaluation. In this embodiment of the present application, perplexity (ppl, Perplexity) is used to quantify the smoothness and naturalness of the copy. The lower the degree of confusion in the copy, the more smooth and natural the copy is, otherwise, the copy is not smooth. For sentence s=(w ₁ , w ₂ , . . . , _wn ), where _wi represents the ith word in sentence s, and the number of words is n, the calculation formula of the confusion degree is shown in the following formula (3):

An N-Gram language model may be used in the embodiments of the present application, where N takes values of 2, 3, and 4. For different N-Gram models, the confusion degree of the copy is calculated by using the weighted summation to obtain the smooth natural degree f of the copy as shown in the following formula (4):

Among them, α _i is the weight value corresponding to the confusion degree of N at different values.

FIG. 9 is a schematic diagram of the training of the theme naturalness model of the advertising copy provided by the embodiment of the present application. The text to be detected is input into the trained copy text naturalness scoring model shown on the right side of FIG. 9 to obtain the smoothness score of the text to be detected, and then It is normalized to obtain a fluent score.

Part 5, Quantitative Scoring of Ad Copy:

Through the above four sub-models, this application obtains the scores of each dimension of the copy D _i : subject score (set as r), compliance score (set as c), attractiveness (set as a), and smooth score (set as f) . Through these four scores, the quality of each copy can be visually evaluated. Figure 10 is a schematic diagram of the scores of the three copywriting in each dimension under the ICAN model, and for the dimension with a low score, a corresponding description is given. For example, sensitive words are given for non-compliant copywriting, incomprehensible fragments are given for inconsistent copywriting, and mismatched category names are given for creative category mismatches. In this embodiment of the present application, the scores of each dimension of each copywriting idea under the ICAN model can be compared using the radar chart in FIG. 11 . In subsequent creative screening and other decisions, copywriting with low scores can be improved or eliminated. In this embodiment of the present application, four dimensions of subjectivity, compliance, attractiveness and naturalness are introduced to quantify and evaluate the copywriting quality of e-commerce advertisements. Through the ICAN model, it is possible to filter out that a single dimension is obviously too low, or the overall dimensions of multiple dimensions are filtered out. Advertising copy that is not high is convenient for subsequent tuning of these copies, or direct elimination. By introducing information entropy, we can quantitatively evaluate the attractiveness of e-commerce advertising copy; by introducing BERT and sentence pair matching, we can quantitatively evaluate the theme of e-commerce advertising copy; A dimension to achieve comprehensive quantitative evaluation of e-commerce advertising copy.

Based on the foregoing embodiments, the embodiments of the present application provide an apparatus for evaluating recommendation information. Each module included in the apparatus and each unit included in each module can be implemented by a processor in a computer device; of course, it can also be implemented by logic Circuit implementation; in the process of implementation, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Microprocessor Unit), a digital signal processor (DSP, Digital Signal Processing) or a field programmable gate Array (FPGA, Field Programmable Gate Array), etc.

An embodiment of the present application further provides an apparatus for evaluating recommended information. FIG. 12 is a schematic structural diagram of the composition of the apparatus for evaluating recommended information provided by an embodiment of the present application. As shown in FIG. 12 , the apparatus for evaluating recommended information 120 includes: an acquisition module 121, configured to acquire recommendation information of the object to be recommended and object information of the object to be recommended from the recommendation information delivery platform; evaluation module 122, configured to input the recommendation information and the object information into the trained The copywriting scoring model is evaluated, and the scoring results of the recommendation information in each dimension are obtained, and the dimensions include subjectivity, compliance, attractiveness and smoothness; the determining module 123 is configured to be based on the recommendation information in each dimension. The scoring result determines the evaluation result of the recommendation information.

In some embodiments, the evaluation device 120 for recommendation information may further include: a second acquisition module, configured to acquire a topic sample set, a sensitive word set, an attractiveness sample set, and a fluency sample set; a training module, configured to In order to input the thematic sample set, the attractiveness sample set and the fluency sample set respectively into the preset thematic network model, the preset attractive network model and the preset smoothness network model, the trained Thematic network model, the trained attractiveness network model and the trained smoothness network model; a construction module, configured to construct a dictionary tree-based search model according to the sensitive word set; a construction module, configured to be based on the training A good topical network model, a trained attractive network model, a trained smoothness network model, and the search model construct a trained copywriting scoring model.

In some embodiments, the training module is further configured to: obtain sample object information and sample recommendation information of each sample object in the thematic sample set; take the sample object information and sample recommendation information of the same sample object as a group sample pair, obtain the label information of the sample pair, the label information represents the probability that the sample object information in the sample pair matches the sample recommendation information; each sample pair corresponding to each sample object in the thematic sample set is The annotation information of each sample pair is input into a preset thematic network model for training and learning, and a trained thematic network model is obtained.

In some embodiments, the construction module is further configured to: construct a dictionary tree according to each sensitive word in the sensitive word set; add a query failure pointer to each node in the dictionary tree to obtain a lookup model based on the dictionary tree.

In some embodiments, the training module is further configured to: obtain sample recommendation information of each sample object in the attractive sample set; perform information extraction on the sample recommendation information of each sample object to obtain the characteristics of each sample object information set, the feature information set includes at least one of the name, category, discount and attribute word of the sample object; input the feature information set of each sample object into the preset attractiveness network model to obtain the trained attraction force network model.

In some embodiments, the training module is further configured to: obtain sample recommendation information of each sample object in the connectivity sample set; perform word segmentation processing on the sample recommendation information of each sample object to obtain the sample recommendation information of each sample object. word segmentation; input the word segmentation of the recommended information of each sample into the preset fluent degree network model to obtain a trained fluent degree network model.

In some embodiments, the evaluation module is further configured to: input the recommendation information and the object information as a set of evaluation pairs to the trained thematic network model, and obtain the thematic scoring result of the recommendation information ; Input the recommended information into the search model to obtain the compliance score result of the recommended information; Input the recommended information into the trained attractiveness network model to obtain the attractiveness score result of the recommended information ; Input the recommended information into the trained fluent network model, and obtain the fluent score result of the recommended information.

In some embodiments, the determining module is further configured to: determine the recommendation information according to the subjectivity scoring results, the compliance scoring results, the attractiveness scoring results, and the smoothness scoring results When the evaluation result of the recommended information is greater than the first preset threshold, it is determined that the evaluation result of the recommended information is passed; when the evaluation result of the recommended information is less than or equal to the first preset threshold, It is determined that the evaluation result of the recommended information is an evaluation failure.

In some embodiments, the determining module is further configured to: calculate the variance of the subjectivity scoring result, the compliance scoring result, the attractiveness scoring result, and the fluency scoring result; when the When the variance is less than the second preset threshold, and there is at least one scoring result in the subjectivity scoring result, the compliance scoring result, the attractiveness scoring result, and the fluency scoring result that is greater than the third preset threshold , it is determined that the evaluation result of the recommendation information is the evaluation pass; when the variance is greater than or equal to the second preset threshold, or the thematic score result, the compliance score result, the attractiveness score result and all When the fluent score results are all less than or equal to the third preset threshold, it is determined that the evaluation result of the recommendation information is an evaluation failure.

In some embodiments, the evaluation device 120 for the recommended information may further include: an adjustment module configured to, when the evaluation result is that the evaluation fails the evaluation, based on the subjectivity scoring result and the compliance scoring result , at least one of the attractiveness scoring result and the fluency scoring result adjusts the recommendation information. The sending module is configured to send the evaluation result to the recommendation information delivery platform, so that the recommendation information delivery platform delivers the evaluation result as the recommendation information that has passed the evaluation.

It should be pointed out here that the description of the above-mentioned embodiment items of the evaluation apparatus for the recommendation information is similar to the description of the above-mentioned method, and has the same beneficial effects as the method embodiment. Those skilled in the art should refer to the description of the method embodiments of the present application to understand the technical details that are not disclosed in the embodiments of the evaluation device for the recommended information of the present application.

It should be noted that, in the embodiments of the present application, if the above-mentioned advertising copy evaluation method is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or in the parts that make contributions to the prior art. The computer software products are stored in a storage medium and include several instructions for A computer device (which may be a personal computer, a server, or a network device, etc.) is caused to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read Only Memory (ROM, Read Only Memory), magnetic disk or optical disk and other media that can store program codes. As such, the embodiments of the present application are not limited to any specific combination of hardware and software.

Correspondingly, the embodiments of the present application provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the method for evaluating recommendation information provided in the foregoing embodiments.

An embodiment of the present application provides an evaluation device for recommended information. FIG. 13 is a schematic diagram of the composition and structure of the device for evaluation of recommended information provided by the embodiment of the present application. According to the exemplary structure of the evaluation device for recommended information 130 shown in FIG. Other exemplary structures of the evaluation device 130 for recommending information are foreseen, so the structures described here should not be regarded as limiting, for example, some components described below may be omitted, or components not described below may be added to suit certain applications special needs.

The evaluation device 130 for recommendation information shown in FIG. 13 includes: a processor 131 , at least one communication bus 132 , a user interface 133 , at least one external communication interface 134 and a memory 135 . Among them, the communication bus 132 is configured to realize the connection communication between these components. The user interface 133 may include a display screen, and the external communication interface 134 may include a standard wired interface and a wireless interface. Wherein, the processor 131 is configured to execute the program of the method for evaluating the recommendation information stored in the memory, so as to implement the steps in the method for evaluating the recommendation information provided by the above embodiments.

The descriptions of the above embodiments of the evaluation device and the storage medium for the recommended information are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For the technical details that are not disclosed in the embodiments of the evaluation device and the storage medium for the recommended information of the present application, please refer to the description of the method embodiments of the present application for understanding.

It is to be understood that reference throughout the specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic associated with the embodiment is included in at least one embodiment of the present application. Thus, appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation. The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.

The unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit; it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration The unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, the execution includes the above The steps of the method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

Alternatively, if the above-mentioned integrated units of the present application are implemented in the form of software function modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or in the parts that make contributions to the prior art. The computer software products are stored in a storage medium and include several instructions for One device is made to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

The above is only the embodiment of the present application, but the protection scope of the present application is not limited to this. Covered within the scope of protection of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A method for evaluating recommendation information, the method comprising:

Obtain the recommendation information of the object to be recommended and the object information of the object to be recommended from the recommendation information delivery platform;

Inputting the recommendation information and the object information into the trained copywriting scoring model for evaluation, and obtaining the scoring results of the recommendation information in each dimension, the dimensions including subjectivity, compliance, attractiveness, and smoothness;

The evaluation result of the recommendation information is determined based on the scoring results of the recommendation information in each dimension.
The method of claim 1, wherein the method further comprises:

Obtain thematic sample sets, sensitive word sets, attractiveness sample sets and fluency sample sets;

Input the thematic sample set, the attractiveness sample set and the fluency sample set into the preset thematic network model, the preset attractiveness network model and the preset fluency network model respectively to obtain the trained theme Sex network model, trained attractiveness network model and trained smoothness network model;

According to the sensitive word set, construct a dictionary tree-based search model;

A trained copywriting scoring model is constructed based on the trained topicality network model, the trained attractiveness network model, the trained smoothness network model and the search model.
The method according to claim 2, wherein inputting the thematic sample set into a preset thematic network model to obtain a trained thematic network model, comprising:

Obtain sample object information and sample recommendation information of each sample object in the thematic sample set;

Take the sample object information and sample recommendation information of the same sample object as a set of sample pairs, and obtain the label information of the sample pair, where the label information represents the probability that the sample object information in the sample pair matches the sample recommendation information ;

Each sample pair corresponding to each sample object in the thematic sample set and the labeling information of each sample pair are input into a preset thematic network model for training and learning, and a trained thematic network model is obtained.
The method according to claim 2, wherein the constructing a dictionary tree-based search model according to the sensitive word set comprises:

According to each sensitive word in the sensitive word set, construct a dictionary tree;

A query failure pointer is added to each node in the dictionary tree to obtain a lookup model based on the dictionary tree.
The method according to claim 2, wherein inputting the attractiveness sample set into a preset attractiveness network model to obtain a trained attractiveness network model, comprising:

obtaining sample recommendation information of each sample object in the attractive sample set;

Perform information extraction on the sample recommendation information of each sample object to obtain a feature information set of each sample object, where the feature information set includes at least one of the name, category, discount and attribute word of the sample object;

Input the feature information set of each sample object into the preset attractiveness network model to obtain the trained attractiveness network model.
The method according to claim 2, wherein, inputting the fluidity sample set into a preset fluidity network model to obtain a trained fluidity network model, comprising:

obtaining sample recommendation information of each sample object in the fluent degree sample set;

Perform word segmentation processing on the sample recommendation information of each sample object to obtain word segmentation of each sample recommendation information;

Inputting the word segmentation of the recommendation information of each sample into a preset smoothness network model to obtain a trained smoothness network model.
The method according to claim 2, wherein the inputting the recommendation information and the object information into a trained copywriting scoring model for evaluation, and obtaining the scoring results of the recommendation information in each dimension, comprising:

Inputting the recommendation information and the object information as a set of evaluation pairs to the trained thematic network model to obtain the thematic scoring result of the recommendation information;

Inputting the recommendation information into the search model to obtain a compliance score result of the recommendation information;

Inputting the recommendation information into the trained attractiveness network model to obtain the attractiveness score result of the recommendation information;

The recommendation information is input into the trained fluent network model, and the fluent score result of the recommendation information is obtained.
The method according to claim 7, wherein the determining the evaluation result of the recommendation information based on the scoring results of the recommendation information in each dimension comprises:

According to the thematic scoring result, the compliance scoring result, the attractiveness scoring result and the smoothness scoring result, determine the scoring result of the recommendation information;

When the scoring result of the recommendation information is greater than the first preset threshold, determine that the evaluation result of the recommendation information is an evaluation pass;

When the scoring result of the recommendation information is less than or equal to the first preset threshold, it is determined that the evaluation result of the recommendation information is an evaluation failure.
The method according to claim 7, wherein the determining the evaluation result of the recommendation information based on the scoring results of each dimension comprises:

calculating the variance of the subjectivity score results, the compliance score results, the attractiveness score results, and the fluency score results;

When the variance is less than the second preset threshold, and at least one of the subjectivity score results, the compliance score results, the attractiveness score results, and the fluent score results is greater than the third predetermined threshold When setting the threshold, it is determined that the evaluation result of the recommended information is an evaluation pass;

When the variance is greater than or equal to a second preset threshold, or the thematic scoring result, the compliance scoring result, the attractiveness scoring result, and the fluency scoring result are all less than or equal to a third preset When the threshold is set, it is determined that the evaluation result of the recommendation information is that the evaluation fails.
The method according to claim 8 or 9, wherein the method further comprises:

When the evaluation result is that the evaluation fails, the recommendation information is evaluated based on at least one of the subjectivity score results, the compliance score results, the attractiveness score results, and the fluent score results. make adjustments.
The method according to any one of claims 1 to 9, wherein the method further comprises:

The evaluation result is sent to the recommendation information delivery platform, so that the recommendation information delivery platform delivers the evaluation result as the recommendation information that has passed the evaluation.
A device for evaluating recommended information, the device comprising:

a first obtaining module, configured to obtain recommendation information of the object to be recommended and object information of the object to be recommended from the recommendation information delivery platform;

The evaluation module is configured to input the recommendation information and the object information into the trained copywriting scoring model for evaluation, and obtain the scoring results of the recommendation information in each dimension, and the dimensions include subjectivity, compliance, attractiveness strength and smoothness;

A determination module configured to determine an evaluation result of the recommendation information based on the scoring results of the recommendation information in each dimension.
The apparatus of claim 12, wherein the apparatus further comprises:

The second obtaining module is configured to obtain the thematic sample set, the sensitive word set, the attractiveness sample set and the fluency sample set;

a training module, configured to input the thematic sample set, the attractiveness sample set and the fluency sample set respectively into a preset thematic network model, a preset attractiveness network model and a preset smoothness network model, Obtain the trained topic network model, the trained attractiveness network model and the trained smoothness network model;

a construction module, configured to construct a dictionary tree-based search model according to the sensitive word set;

A building module is configured to build a trained copywriting scoring model based on the trained thematic network model, the trained attractiveness network model, the trained smoothness network model and the search model.
The apparatus of claim 13, wherein the training module is further configured to:

Obtain sample object information and sample recommendation information of each sample object in the thematic sample set;

Take the sample object information and sample recommendation information of the same sample object as a set of sample pairs, and obtain the label information of the sample pair, where the label information represents the probability that the sample object information in the sample pair matches the sample recommendation information ;

Each sample pair corresponding to each sample object in the thematic sample set and the labeling information of each sample pair are input into a preset thematic network model for training and learning, and a trained thematic network model is obtained.
The apparatus of claim 13, wherein the building block is further configured to:

According to each sensitive word in the sensitive word set, construct a dictionary tree;

A query failure pointer is added to each node in the dictionary tree to obtain a lookup model based on the dictionary tree.
The apparatus of claim 13, wherein the training module is further configured to:

obtaining sample recommendation information of each sample object in the attractive sample set;

Perform information extraction on the sample recommendation information of each sample object to obtain a feature information set of each sample object, where the feature information set includes at least one of the name, category, discount and attribute word of the sample object;

Input the feature information set of each sample object into the preset attractiveness network model to obtain the trained attractiveness network model.
The apparatus of claim 13, wherein the training module is further configured to:

obtaining sample recommendation information of each sample object in the fluent degree sample set;

Perform word segmentation processing on the sample recommendation information of each sample object to obtain word segmentation of each sample recommendation information;

Inputting the word segmentation of the recommendation information of each sample into a preset smoothness network model to obtain a trained smoothness network model.
The apparatus of claim 13, wherein the evaluation module is further configured to:

Inputting the recommendation information and the object information as a set of evaluation pairs to the trained thematic network model to obtain the thematic scoring result of the recommendation information;

Inputting the recommendation information into the search model to obtain a compliance score result of the recommendation information;

Inputting the recommendation information into the trained attractiveness network model to obtain the attractiveness score result of the recommendation information;

The recommendation information is input into the trained fluent network model, and the fluent score result of the recommendation information is obtained.
The apparatus according to claim 18, wherein the determining module is further configured to:

According to the thematic scoring result, the compliance scoring result, the attractiveness scoring result and the smoothness scoring result, determine the scoring result of the recommendation information;

When the scoring result of the recommendation information is greater than the first preset threshold, determine that the evaluation result of the recommendation information is an evaluation pass;

When the scoring result of the recommendation information is less than or equal to the first preset threshold, it is determined that the evaluation result of the recommendation information is an evaluation failure.
The apparatus according to claim 18, wherein the determining module is further configured to:

calculating the variance of the subjectivity score results, the compliance score results, the attractiveness score results, and the fluency score results;

When the variance is less than the second preset threshold, and at least one of the subjectivity score results, the compliance score results, the attractiveness score results, and the fluent score results is greater than the third predetermined threshold When setting the threshold, it is determined that the evaluation result of the recommended information is an evaluation pass;

When the variance is greater than or equal to a second preset threshold, or the thematic scoring result, the compliance scoring result, the attractiveness scoring result, and the fluency scoring result are all less than or equal to a third preset When the threshold is set, it is determined that the evaluation result of the recommendation information is that the evaluation fails.
The apparatus of claim 19 or 20, wherein the apparatus further comprises:

An adjustment module configured to, when the evaluation result is that the evaluation fails, based on at least one scoring result among the subjectivity scoring result, the compliance scoring result, the attractiveness scoring result and the fluency scoring result The recommended information is adjusted.
The apparatus of any one of claims 12 to 20, wherein the apparatus further comprises:

The sending module is configured to send the evaluation result to the recommendation information delivery platform, so that the recommendation information delivery platform delivers the evaluation result as the recommendation information that has passed the evaluation.
An evaluation device for recommended information, including:

processor; and

a memory configured to store a computer program executable on the processor;

Wherein, when the computer program is executed by the processor, the steps of the method of any one of claims 1 to 11 are implemented.
A computer-readable storage medium storing computer-executable instructions configured to perform the steps of the method of any one of claims 1 to 11.