CN115700692A - Method and device for determining potential recommendation information - Google Patents
Method and device for determining potential recommendation information Download PDFInfo
- Publication number
- CN115700692A CN115700692A CN202110863912.1A CN202110863912A CN115700692A CN 115700692 A CN115700692 A CN 115700692A CN 202110863912 A CN202110863912 A CN 202110863912A CN 115700692 A CN115700692 A CN 115700692A
- Authority
- CN
- China
- Prior art keywords
- recommendation information
- exposure
- cost
- boost
- degree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present disclosure relates to a method for determining potential recommendation information, comprising: acquiring multiple groups of historical sorting data associated with target recommendation information; determining the relationship between the exposure promotion degree and the exposure cost promotion degree of the target recommendation information based on the multiple groups of historical sorting data; determining a relationship between the exposure promotion degree and the cost return rate promotion degree of the target recommendation information based on the determined relationship between the exposure promotion degree and the exposure cost promotion degree; predicting a corresponding exposure boost according to one or more predetermined cost return rate boost according to the determined relationship between the exposure boost and the cost return rate boost; and determining the target recommendation information as potential recommendation information in response to the predicted exposure boost for at least one of the one or more predetermined cost rate of return boosts being greater than or equal to the corresponding boost threshold.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, a computing device, and a computer-readable storage medium for determining potential recommendation information.
Background
With the development of computer technology, people increasingly use terminal devices to process various matters in life, work, learning, and the like through computer networks. Accordingly, more and more organizations or individuals are beginning to deliver recommendation information through web platforms to recommend various objects such as commodities, applications, articles, videos, etc. to end users, thereby attracting end users to perform various actions that may bring benefits to the delivery party, such as purchasing physical or virtual commodities, purchasing members, enjoying articles, pictures, or video content, etc.
Generally speaking, a certain resource cost is consumed for releasing the recommendation information, and the exposure cost expected to be released by the releasing party and the exposure probability of the recommendation information are usually positively correlated. Thus, the presenter can increase the exposure probability of the recommendation information by increasing the exposure cost expected to be paid. However, in practical cases, the exposure boost degrees of different recommendation information are often different for the same exposure cost boost degree. Therefore, the blind improvement of the exposure cost of each recommended information is likely to cause excessive consumption of resource cost, but the profit is not satisfactory, thereby causing waste of resource cost. Furthermore, when the sponsor defines the expected cost profitability, it is difficult to directly adjust the expected exposure cost by adjusting the bid for conversion, click, exposure, etc.
Disclosure of Invention
In view of the above, the present disclosure provides a method, apparatus, computing device and computer-readable storage medium for determining potential recommendation information, which may alleviate, alleviate or even eliminate the above-mentioned problems.
According to an aspect of the present disclosure, there is provided a method for determining potential recommendation information, including: acquiring multiple groups of historical sorting data associated with target recommendation information, wherein the historical sorting data comprises data related to multiple candidate recommendation information and one winning recommendation information, and the winning recommendation information is exposure recommendation information determined from the multiple candidate recommendation information; determining the relationship between the exposure promotion degree and the exposure cost promotion degree of the target recommendation information based on the obtained multiple groups of historical sorting data; determining a relationship between the exposure promotion degree and the cost return rate promotion degree of the target recommendation information based on the determined relationship between the exposure promotion degree and the exposure cost promotion degree; predicting a corresponding exposure boost according to one or more predetermined cost return rate boost according to the determined relationship between the exposure boost and the cost return rate boost; and determining the target recommendation information as potential recommendation information in response to the predicted exposure boost for at least one of the one or more predetermined cost-rate-of-return boosts being greater than or equal to a corresponding boost threshold.
In some embodiments, the boost threshold depends on a cost rate of return boost corresponding to the predicted exposure boost.
In some embodiments, each of the sets of historical ordering data includes at least one of: the exposure cost of the target recommendation information, the exposure cost of the recommendation information winning in the ranking, and the identification indicating that the target recommendation information wins in the ranking, and wherein determining the relationship between the exposure boost level and the exposure cost boost level of the target recommendation information based on the obtained sets of historical ranking data comprises: determining the ratio of the exposure cost of the winning recommendation information to the exposure cost of the target recommendation information based on each group of historical sorting data in the multiple groups of historical sorting data; aiming at the ratio determined based on each group of historical sorting data, determining the exposure promotion degree of the target recommendation information when the exposure cost promotion degree of the target recommendation information is equal to the ratio minus one; and determining a relationship between the exposure boost degree and the exposure cost boost degree of the target recommendation information based on the plurality of sets of exposure cost boost degrees and the exposure boost degrees determined for the plurality of sets of historical sorting data.
In some embodiments, determining the relationship between the exposure boost and the exposure cost boost of the target recommendation information based on the plurality of sets of exposure cost boosts and exposure boosts determined for the plurality of sets of historical sorting data comprises: aiming at the multiple groups of exposure cost promotion degrees and exposure promotion degrees, polynomial fitting is executed to determine parameters corresponding to all terms in the polynomial; and determining the relationship between the exposure promotion degree and the exposure cost promotion degree of the target recommendation information according to the parameters corresponding to the items in the determined polynomial.
In some embodiments, determining the relationship between the exposure boost and the cost return boost of the target recommendation information based on the determined relationship between the exposure boost and the exposure cost boost comprises: acquiring a relation between the exposure cost improvement degree and the cost return rate improvement degree, wherein the relation is determined based on a negative correlation relation between the exposure cost and the cost return rate; and determining the relationship between the exposure promotion degree and the cost return rate promotion degree of the target recommendation information based on the relationship between the exposure promotion degree and the exposure cost promotion degree and the obtained relationship.
In some embodiments, the method further comprises: in response to the target recommendation information being determined to be potential recommendation information, performing classification processing on the target recommendation information so as to classify the target recommendation information into one of final potential recommendation information and non-final potential recommendation information.
In some embodiments, in response to the target recommendation information being determined to be potential recommendation information, performing classification processing on the target recommendation information includes: in response to the target recommendation information being determined to be potential recommendation information, performing a classification process on the target recommendation information using a trained machine learning model, comprising: inputting the characteristics of the target recommendation information into the trained machine learning model, and determining the target recommendation information as final potential recommendation information or non-final potential recommendation information according to a classification result output by the trained machine learning model, wherein the characteristics of the target recommendation information comprise a price raising proportion of the target recommendation information, and the price raising proportion is determined based on a ratio of an initial cost return rate and a raised cost return rate.
In some embodiments, the characteristics of the target recommendation information further include one or more of: the target recommendation information comprises an identification of the target recommendation information, a releasing party of the target recommendation information, the exposure of the target recommendation information in a latest statistical period, a releasing optimization type of the target recommendation information and a releasing site of the target recommendation information.
In some embodiments, the trained machine learning model is trained by: obtaining historical pricing data of a plurality of recommendation information, wherein each piece of historical pricing data comprises a historical pricing proportion and a corresponding exposure boost degree, and the pricing proportion is determined based on one of the following items: the ratio of the initial cost return rate of the recommendation information to the promoted cost return rate, and the ratio of the promoted cost of the recommendation information to the initial cost; marking the recommendation information with the exposure lifting degree larger than or equal to the lifting degree threshold value as final potential recommendation information, otherwise, marking the recommendation information as non-final potential recommendation information; and training the machine learning model using historical pricing data of the labeled plurality of recommendation information such that an error between a classification result output by the machine learning model and a label of each labeled recommendation information is minimized.
In some embodiments, the method further comprises: in response to the target recommendation information being determined to be potential recommendation information, generating a potential tag for the target recommendation information, wherein the potential tag is configured to, when sent to the publisher client, cause the publisher client to generate and present an identification indicating that the target recommendation information is potential.
In some embodiments, the method further comprises: in response to the target recommendation information being determined to be final potential recommendation information, generating a potential tag for the target recommendation information, wherein the potential tag is configured to, when sent to the publisher client, cause the publisher client to generate and present an identification indicating that the target recommendation information is potential.
In some embodiments, the method further comprises: monitoring a request for viewing the identifier from a client of the delivering party; and in response to monitoring the request, sending a corresponding exposure boost predicted for at least one of the one or more predetermined cost rate of return boosts to the publisher client.
According to another aspect of the present disclosure, there is provided an apparatus for determining potential recommendation information, including: an obtaining module configured to obtain a plurality of sets of historical ranking data associated with the target recommendation information, wherein the historical ranking data includes data related to a plurality of candidate recommendation information and one winning recommendation information, and the winning recommendation information is exposure recommendation information determined from the plurality of candidate recommendation information; a first determining module configured to determine a relationship between an exposure boost degree and an exposure cost boost degree of the target recommendation information based on the obtained multiple sets of historical sorting data; a second determining module configured to determine a relationship between the exposure boost degree and the cost return rate boost degree of the target recommendation information based on the determined relationship between the exposure boost degree and the exposure cost boost degree; a prediction module configured to predict, for one or more predetermined cost-return-rate boost degrees, a corresponding exposure boost degree according to the determined relationship between the exposure boost degree and the cost-return-rate boost degree; and a third determination module configured to determine the target recommendation information as potential recommendation information in response to the predicted exposure boost for at least one of the one or more predetermined cost rate of return boosts being greater than or equal to the corresponding boost threshold.
According to yet another aspect of the present disclosure, there is provided a computing device comprising: a memory configured to store computer-executable instructions; a processor configured to perform the method described according to the preceding aspect when the computer-executable instructions are executed by the processor.
According to yet another aspect of the present disclosure, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed, perform the method described in accordance with the preceding aspect.
According to the scheme for determining the potential recommendation information provided by the present disclosure, whether the target recommendation information is potential recommendation information can be determined through historical ranking data associated with the target recommendation information. Specifically, the exposure boost degree corresponding to one or more predetermined cost-return-rate boost degrees can be predicted by determining the relationship between the cost-return-rate boost degree and the exposure boost degree, and then the target recommendation information of which the exposure boost degree predicted for at least one predetermined cost-return-rate boost degree is greater than or equal to the corresponding boost degree threshold value is determined as the potential recommendation information. The determined potential recommendation information can acquire a relatively high exposure promotion degree through relatively low resource cost investment, so that a basis and a reference for adjusting resource cost configuration can be provided for a release party, the resource cost configuration can be optimized, and the resource cost conversion efficiency can be improved. Moreover, the scheme of the disclosure establishes the correlation between the cost benefit rate improvement degree and the exposure improvement degree, and also provides possibility for adjusting the exposure probability by adjusting the cost benefit rate.
These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter.
Drawings
Further details, features and advantages of the disclosure are disclosed in the following description of exemplary embodiments, taken in conjunction with the accompanying drawings, in which:
fig. 1 schematically illustrates an example scenario in which the technical solution provided by the present disclosure may be applied;
FIG. 2A schematically illustrates an example interface diagram in the related art;
FIG. 2B schematically illustrates another example interface diagram in the related art;
FIG. 3 schematically illustrates an example flow diagram of a method for determining potential recommendation information, in accordance with some embodiments of the present disclosure;
FIG. 4A schematically illustrates an example flow diagram of sub-steps in FIG. 3, in accordance with some embodiments of the present disclosure;
FIG. 4B schematically illustrates an example flow diagram of another substep in FIG. 3 according to some embodiments of the present disclosure;
FIG. 5 schematically illustrates another example flow diagram of a method for determining potential recommendation information, in accordance with some embodiments of the present disclosure;
FIG. 6 schematically illustrates an example flow diagram of a method for training a machine learning model, in accordance with some embodiments of the present disclosure;
fig. 7A schematically illustrates an example interface diagram according to some embodiments of the disclosure;
FIG. 7B schematically illustrates another example interface diagram according to some embodiments of the present disclosure;
FIG. 7C schematically illustrates yet another example interface diagram according to some embodiments of the present disclosure;
fig. 8 schematically illustrates an example block diagram of an apparatus for determining potential recommendation information in accordance with some embodiments of this disclosure;
fig. 9 schematically illustrates an example block diagram of a computing device in accordance with some embodiments of this disclosure.
Detailed Description
Before describing embodiments of the present disclosure in detail, some relevant concepts are explained first:
1. recommendation information: information for recommending one or more objects, such as advertisements. The recommendation information can be released by an organization or an individual through one or more network platforms, and the network platforms can present the released recommendation information on terminal devices of one or more users according to certain rules. When the user clicks on the recommendation information presented on the terminal device, the terminal device may display a new page or window to present the recommended object, such as a merchandise purchase page, an application download link, an article browsing page, a video playing page, and so forth.
2. Potential recommendation information: meaning that there may be recommendation information for a large boost in exposure when a certain amount of boost is made to bids for exposure, click, conversion, or cost return. For example, it may be defined as recommending information that predicts that the exposure boost may reach 2 times or more the bid boost for several hours in the future. The potential recommendation information may be, for example, a potential advertisement or the like depending on the application scenario. The determination of the potential recommendation information may help a delivery party of the recommendation information to better optimize resource cost configuration, and to better achieve an initial amount and achieve an increase in exposure or conversion, where the initial amount may refer to the recommendation information reaching a threshold number of conversions within a predetermined time after being successfully delivered.
3. Exposure: the recommendation information is presented on the terminal device of the user, for example, displayed through a display, a touch screen, a projector, etc., played through headphones, a sound, etc., or presented in other user-sensible manners.
4. Exposure cost: the resource cost paid by the publisher to expose the recommendation information may be measured by money, etc., or by points, or other items of the network platform, or by other means approved by both parties. In the present disclosure, the exposure Cost may be expressed by a thousand exposure disbursement (CPM, cost Per mill) or a thousand exposure revenue (eCPM, effective Cost Per mill). The CPM is defined at the perspective of a publisher of the recommendation information and represents the resource cost paid by the publisher for thousands of exposures of the recommendation information; the eCPM is defined from the perspective of the network platform used to deliver the recommendation, representing the revenue of resources that the network platform receives for thousands of exposures of the recommendation. Alternatively, the exposure cost mentioned in the present disclosure may be expressed by other means.
5. The cost return rate is as follows: in the present disclosure, the cost return rate is an index parameter for evaluating the profitability of the recommendation information, and the higher the cost return rate is, the higher the profit that can be brought by the recommendation information at the unit resource cost is. Generally, the Return on cost may be expressed by a Return On Investment (ROI), or may be expressed by a logarithm of the Return on Investment, a product of the Return on Investment and a preset coefficient, or may be expressed by other quantities. The return on investment may be defined as the ratio of the profit gained by the investor through the recommendation information to the cost of the resource paid for putting the recommendation information, i.e. ROI = gain/cost of resource 100%.
In some cases, the sponsor may be allowed to preset a desired return on cost, which may also be understood as bidding on the return on cost. For example, recommendation information for an ROI bid may be referred to as ROI recommendation information, and this type of recommendation information is increasingly appearing in and becoming a dominant form of placement in fields such as games, e-commerce, and the like. The releasing form can take the benefits brought by the recommendation information into consideration, and is beneficial to realizing more controllable releasing effect of the recommendation information and more accurate releasing regulation and control.
6. Total Life cycle Value (LTV, life Time Value): generally, the total profit obtained by a publisher of recommendation information from all interactions of a user with a recommendation object related to the recommendation information is referred to as the total profit sum. In practice, the statistical period of the LTV may be set as desired, such as a full period (i.e., from the beginning of the interaction to the end of the interaction), a day, a week, a month, etc. For example, for recommendation information whose recommendation object is a game, the LTV of each user may refer to an amount paid in the game by the user after entering the game by clicking the recommendation information, such as a total amount paid from login to no-login time period or an amount paid within one day, one week, one month, and the like. LTVs may be similarly defined for recommendation information whose recommended object is another object. Further, the LTV may be used to determine a cost return rate of the recommendation information, and the statistical period may be related to an optimization period of the cost return rate, for example, if the optimization target of the recommendation information is the first day cost return rate, the LTV used may be a one-day LTV of the user.
In practical applications, for example, when determining the cost return rate of the recommendation information, the LTV used by the network platform is generally a Predicted Value of the LTV, that is, pLTV (Predicted Life Time Value), which can be obtained by using various prediction algorithms or models in the related art based on the user portrait of the user and various attributes of the recommendation information.
7. Click Rate (CTR, click Through Rate): clicking is a human-computer interaction activity that may be used to trigger the opening of windows, pages, applications, etc. associated with the clicked recommendation information. Clicking may be accomplished in various possible ways, such as using an input device such as a mouse, touchpad, touch screen, or by gesture control, voice control, motion control, light signals, and so forth. The click rate is used for representing the probability that the recommendation information is clicked after exposure, and can be determined by the ratio of the number of clicks to the number of exposures of the recommendation information.
In practical applications, the CTR used by the network platform is generally a Predicted value of the CTR, that is, pCTR (Predicted Click Through Rate), which can be obtained by using various prediction algorithms or models in the related art based on various attributes of the user profile of the user and the recommendation information.
8. Conversion (CVR, conversion Rate): the conversion means that after clicking the recommendation information, the user completes preset user behaviors for the recommended object, such as a purchase behavior for a commodity purchase page, a download behavior for an application download link, browsing for an article browsing page or a video playing page (such as browsing time exceeding a predetermined threshold), a praise, a forward or a reward behavior, and the like. The conversion rate is used for representing the probability of executing corresponding conversion behaviors after the user clicks the recommendation information.
In practical applications, the CVR used by the network platform is generally a pc vr (Predicted Conversion Rate) which is a Predicted value of the CVR, and may be obtained by using various prediction algorithms or models in the related art based on various attributes of the user profile of the user and the recommendation information.
9. Bidding: generally refers to the expected cost of a single exposure, click, or conversion to the recommendation information, and more specifically, to the amount of resources that an organization or individual delivering the recommendation information would expect for a single exposure, click, or conversion to the recommendation information to be delivered to the network platform. Bids are typically pre-set by an organization or individual that places the recommendation information.
In some implementations, the bids can include a shallow bid and a deep bid. The shallow bid can be a bid for a shallow optimization goal and the deep bid can be a bid for a deep optimization goal. The optimization goal may be set according to the terms that the publisher desires to optimize, for example, the publisher may desire to obtain a higher click rate, a higher download rate, a higher form reservation, etc. A deep optimization goal may be an item that is deeper on the conversion link than a shallow conversion goal, e.g., generally, recommendation information for a commodity link, a purchase may be considered deeper on the conversion link than a click; for referral information by application, the registration may be considered deeper in the conversion link relative to the download; and so on. Optionally, in addition to or in lieu of shallow and/or deep bids, a desired cost return rate can be set for the recommendation information, such as setting a desired ROI. Such recommendation information may be considered as recommendation information (such as ROI recommendation information) that sets a desired rate of return on cost as mentioned in this disclosure.
Moreover, it will be understood that, although the terms first, second, etc. may be used in this disclosure to describe various steps, modules, etc., these steps, modules, etc. should not be limited by these terms. These terms are only used to distinguish one step or module from another step or module. It is also to be understood that reference to "a plurality" or similar terms in this disclosure is to be understood as at least two, i.e., two or more.
Fig. 1 schematically illustrates an example scenario 100 in which the technical solutions provided by the present disclosure may be applied.
As shown in fig. 1, the scenario 100 may include a network platform server 110, a publisher device 130 for publishing recommendation information on a network platform, and a user terminal device 150, and optionally a database device 120 for storing publishing data (e.g., historical ranking data, etc.), which may be in communication via a network 170.
Illustratively, the publisher administrator 140 may publish recommendation information on the network platform via the publisher device 130 and set exposure, click and/or conversion bids, expected cost returns, etc. for the published recommendation information. The network platform server 110 may determine the recommended information to be exposed based on a pre-written exposure algorithm. For example, for a certain presentation location, the network platform server 110 may compare click-through rates, conversion rates, or exposure costs, etc. of several candidate recommendation information, and then may push the recommendation information with the highest exposure click-through rate, conversion rate, or exposure cost (i.e., winning recommendation information) to the terminal device of the user for presentation (i.e., exposure) at the presentation location, so as to achieve a higher interaction probability between the user and the presented recommendation information, which may help achieve a higher platform profit and a higher cost profit rate of the delivering party. Illustratively, a recommendation information delivery system (e.g., an advertisement delivery system) that may be at least partially deployed on the network platform server 110 may rank recommendation information (e.g., advertisements) according to exposure costs determined based on the estimated life cycle total value, the estimated click through rate, and the estimated conversion rate expected return on investment, and select the top ranked recommendation information to expose the recommendation information to the terminal device. Alternatively, the network platform server 110 may obtain data related to the recommendation information from a separate database device 120 or from a local database to implement the above-described process of determining recommendation information for exposure. The user 160 may browse the recommendation information pushed by the web platform server 110 on the terminal device 150 and optionally obtain other services provided by the user. The user 160 may click on the recommendation information of interest on the terminal device 150 and optionally perform the conversion action and the payment action. Generally, such conversion and payment activities may be recorded by the sponsor of the recommendation information.
In the present disclosure, the server 110 may be a single server or a cluster of servers on which an application program for performing the method of determining potential recommendation information may be run and storing related data. Alternatively, server 110 may also run other applications and store other data. For example, the server 110 may include multiple virtual hosts to run different applications and provide different services.
In the present disclosure, the terminal device 150 and the presenter device 130 may be various types of devices, such as a mobile phone, a tablet computer, a notebook computer, a wearable device such as a smart watch, an in-vehicle device, and so on. Terminal device 150 may have a client deployed thereon, which may be used to present recommendation information and optionally provide other services, and may take any of the following forms: locally installed applications, applets accessed via other applications, web programs accessed via a browser, and the like. The user 160 may view the recommendation information presented by the client through the input/output interface of the terminal device 150 as well as perform click actions, implement conversion actions, complete payment actions, and so forth. Similarly, a client for delivering recommendation information may be deployed on the terminal device 130, and the delivery administrator 140 may input data related to the recommendation information and view delivery results, etc. through the input/output interface of the delivery device 130. Alternatively, the publisher device 130 may be integrated with the network platform server 110, in which case the network platform side and the recommendation information publisher may be the same principal.
In the present disclosure, the database device 120 may be regarded as an electronic file cabinet, that is, a place for storing electronic files, and a user may add, query, update, delete, etc. to data in the files. A "database" is a collection of data that is stored together in a manner that can be shared by multiple users, has as little redundancy as possible, and is independent of the application.
Further, in the present disclosure, the network 170 may be a wired network connected via a cable, an optical fiber, or the like, or may be a wireless network such as 2G, 3G, 4G, 5G, wi-Fi, bluetooth, zigBee, li-Fi, or the like.
In the related art, there has been a scheme of determining potential recommendation information for which a desired cost-benefit rate is not set, for example, a scheme of determining potential recommendation information for non-ROI recommendation information. The determined potential recommendation information may be presented through an interface as shown in fig. 2A, 2B. FIGS. 2A and 2B schematically illustrate example interfaces 200A and 200B, respectively, in the related art, which schematically illustrate portions of a presentation interface related to potential advertising, in the context of an advertising scene.
As shown in FIG. 2A, the interface 200 may present a plurality of advertisements, each advertisement relating to several items of attribute information. Specifically, taking the first piece of advertisement information at the top of fig. 2A as an example, the status is in delivery; the promotion target is sales lead collection, namely, the intention client business is collected so as to develop potential clients; the bidding mode is oCPM (Optimized Cost Per Mille, optimized thousand-time display bidding), the shallow optimization target is form reservation, the bidding value (namely shallow bidding value) aiming at the form reservation is 460.00 yuan, namely the resource Cost expected to be paid by a delivery party is 460 yuan aiming at completing one form reservation; the depth optimization goal and the goal bid are null. The attribute information can be preset by the releasing party, and the network platform can optimize the releasing condition of the corresponding recommendation information according to the attribute information. As indicated by reference numeral 201, the user may be informed of which advertisements are potential advertisements by a "potential" word identification.
When the user clicks on the tab 201, a window 202 such as in the interface 200B may be presented. As shown in FIG. 2B, the window 202 can include the system recommended depth target bid, the corresponding depth target bid adjustment scale, and the predicted future 1-hour exposure (including exposure and exposure boost), and can also include a chart showing the predicted future 1-hour exposure and exposure rise/bid rise for multiple depth target bids. Further, window 202 can include options that allow the bidder to one-click optimize respective advertising impressions according to the recommended depth target bids, as well as options that allow the bidder to select whether to parity optimize bids.
However, general bidding schemes such as CPM (Cost Per mill, show bid thousands), opmp, CPA (Cost Per Action), opca (Optimized Cost Per Action), etc. only relate to exposure, clicking and conversion processes of recommendation information, and do not relate to the post-conversion income condition. Therefore, these bidding methods cannot comprehensively and objectively measure the delivery effect of the recommendation information. The applicant finds that by setting the desired cost-benefit ratio for the recommendation information, the converted benefit situation can be taken into account to achieve a better delivery effect. However, for such recommendation information (e.g., ROI recommendation information) with a desired cost benefit rate set, the exposure cost (e.g., eCPM) and the shallow bid often do not have a significant positive correlation or negative correlation, so it is difficult to adjust the exposure cost of the recommendation information by simply adjusting the shallow bid, and thus, the exposure amount. Moreover, no solution exists for determining potential recommendation information specifically for such recommendation information. However, with the widespread use of such recommendation information, the need for a scheme of determining potential recommendation information for such recommendation information is becoming more urgent.
Fig. 3 schematically illustrates an example flow diagram of a method 300 for determining potential recommendation information, in accordance with some embodiments of the present disclosure. Illustratively, the method 300 may be deployed on the server 110 shown in fig. 1 or a virtual host therein in the form of an application program or the like to determine the potential recommendation information.
Specifically, in step 310, multiple sets of historical ranking data associated with the target recommendation information may be obtained, where the historical ranking data may include data related to multiple candidate recommendation information and one winning recommendation information, which is recommendation information of exposure determined from the multiple candidate recommendation information. Illustratively, a server or host computer performing the method 300 may retrieve sets of historical ranking data associated with target recommendation information from a local storage device or an external storage device (such as an external database device). As mentioned above, when the network platform determines the recommendation information to be exposed for a presentation location, the candidate recommendation information is ranked according to a predetermined index, and a winning recommendation information is determined to be exposed at the presentation location. The historical ranking data referred to herein may be data that is relevant to the ranking process. The historical ranking data associated with the target recommendation information may refer to the historical ranking data associated with the ranking process in which the target recommendation information participated.
In some embodiments, when the network platform determines the recommendation information to be exposed for one presentation location, the network platform may perform multiple screening on multiple candidate recommendation information in sequence, for example, one or more of recall, rough ranking, fine ranking, and the like in sequence. Recalls may be used to perform a preliminary filtering of a large amount of recommendation information by a simple filtering condition (e.g., a targeting condition determined by various information of a user) in order to quickly acquire several recommendation information that may be of interest to the user. The bold ranking may be performed with respect to the number of recommendations obtained through, for example, a recall process, which may further sort and/or filter the number of recommendations obtained based on certain characteristics to determine therefrom a number of recommendations to enter the refinement process, and the number of recommendations allowed to enter the refinement process may be predetermined. Refinement may be used to rank the plurality of recommendation information according to a preset index (e.g., exposure cost) and determine therefrom a winning recommendation for the final exposure. In general, the recommendation information with the highest exposure cost may be determined as the winning recommendation information. In such an embodiment, the historical ranking data may be data related to a refinement process. Illustratively, a set of historical ranking data may include names of a plurality of candidate recommendation information entering a refinement process and corresponding exposure costs, an identifier indicating which recommendation information wins the refinement process, and the like, wherein the exposure costs of the candidate recommendation information may be represented by eCPM predicted by the network platform for the candidate recommendation information in the refinement process. Alternatively, the eCPM may be predicted directly from information such as the characteristics of candidate recommendation information (e.g., the category of the recommendation object to which the recommendation information relates, the identity of the sponsor, the price of the recommendation object or the orientation of the recommendation information, etc.) and the user profile of the user to whom the exposed recommendation information determined by this refinement process is to be presented), or may be calculated indirectly from one or more of the bids, pCTR, pCVR, pLTV, etc. In such an example, the multiple sets of historical ranking data associated with the target recommendation information may respectively correspond to multiple refinement processes in which the target recommendation information participates, i.e., each set of historical ranking data may correspond to one refinement process.
Further, alternatively, the acquired plurality of sets of history ranking data may be history ranking data within a certain time range, such as the latest 1 hour, the latest 6 hours, the latest 12 hours, the latest day, the latest three days, 1 hour around the same time of the previous day, a certain hour or several hours directly specified, and the like. The time range can be specifically set according to actual application scenes and requirements. Alternatively, the obtained sets of historical ranking data may be filtered historical ranking data, such as may be filtered according to geographic location characteristics, user characteristics, presence location characteristics, and so forth.
At step 320, a relationship between the exposure boost and the exposure cost boost of the target recommendation information may be determined based on the obtained sets of historical ranking data. In the present disclosure, the exposure cost improvement degree can be understood as an improvement ratio of the exposure cost, i.e., the post-improvement exposure cost/the pre-improvement exposure cost-1. Exposure boost, cost-return boost, and the like can be similarly understood. The relationship between the degree of increase in exposure and the degree of increase in exposure cost may be a relationship expressed by various means such as a table, a line graph, a functional expression, and the like.
In some embodiments, for each set of historical ordering data, a relationship between the exposure boost and the exposure cost boost for the target recommendation information may be determined based on a comparison between the exposure cost of the target recommendation information and the exposure cost of the winning recommendation information of the exposure. For example, for a set of historical ranking data, the exposure boost degree of the target recommendation information when the exposure cost of the target recommendation information is boosted to the exposure cost of the winning recommendation information may be determined based on the exposure cost of the target recommendation information and the exposure cost of the winning recommendation information. Further, by integrating the exposure cost improvement degree and the exposure improvement degree of the target recommendation information determined according to each group of historical sorting data, the relationship, such as a functional relationship, between the exposure improvement degree and the exposure cost improvement degree of the target recommendation information can be determined.
At step 330, a relationship between the exposure boost and the cost return rate boost of the target recommendation information may be determined based on the determined relationship between the exposure boost and the exposure cost boost. Similarly, the relationship between the exposure boost degree and the cost return rate boost degree may also be a relationship expressed by various ways such as a table, a line graph, a functional expression, and the like.
Since cost return relates to the ratio of revenue to cost, in general, there may be a predetermined relationship between cost return and exposure cost, which may be known or may be determinable through data analysis means. For example, the cost return rate and the exposure cost may be inversely related, such as inversely proportional. Alternatively, the relationship between the two may be determined by analyzing the exposure cost and the cost return rate in the historical delivery data of the recommended information, for example, by data fitting. Further, the relationship between the cost return rate promotion and the exposure cost promotion can be determined according to the relationship between the cost return rate and the exposure cost. Alternatively, the relationship between the cost return rate improvement degree and the exposure cost improvement degree may also be obtained by other methods, for example, the relationship between the cost return rate improvement degree and the exposure cost improvement degree in the historical delivery data of the recommendation information may be directly determined by analyzing the cost return rate improvement degree and the exposure cost improvement degree. Thus, the relationship between the exposure improvement degree and the cost return rate improvement degree of the target recommendation information can be determined based on the relationship between the cost return rate improvement degree and the exposure cost improvement degree and the relationship between the exposure improvement degree and the exposure cost improvement degree determined in step 320 by taking the exposure cost improvement degree as an intermediate quantity.
In step 340, a corresponding exposure boost may be predicted for one or more predetermined cost-rate-of-return boosts based on the determined relationship between the exposure boost and the cost-rate-of-return boost. For example, a plurality of cost-rate-of-return (costr) enhancement degrees may be preset, or a plurality of costr enhancement degree determination mechanisms may be preset, so that the plurality of costr enhancement degrees may be adaptively determined according to the preset mechanism. For example, for a cost return such as in the form of ROI, the gradient of reduction of the cost return and the number of degrees of increase of the cost return may be set. For example, for the target recommendation information whose optimization target is the first-day cost rate of return, the decreasing gradient may be set to 0.01, and the number of cost rate of return increase degrees is set to 4, so that the plurality of predetermined cost rate of return increase degrees may be-0.01, -0.02, -0.03, and-0.04; for the target recommendation information with an optimization target of 30-day cost-rate of return, the reduction gradient may be set to 0.1, and the number of cost-rate of return increases may be set to 4, and the plurality of predetermined cost-rate of return increases may be-0.1, -0.2, -0.3, and-0.4.
Further, for each pre-determined or determined cost rate of return enhancement, a corresponding exposure enhancement may be predicted based on the relationship between the exposure enhancement and the cost rate of return enhancement determined at step 330. For example, if the determined relationship between the exposure boost and the cost return rate boost is a relationship described by a functional expression, each cost return rate boost may be substituted into the functional expression to obtain a predicted exposure boost; if the determined relationship between the exposure boost degree and the cost return rate boost degree is the relationship described by the line graph, predicting the corresponding exposure boost degree according to the coordinates of the points corresponding to the cost return rate boost degrees on the line graph; if the determined relationship between the exposure lifting degree and the cost return rate lifting degree is a table description relationship, the exposure lifting degree corresponding to each cost return rate lifting degree can be directly read from the table, or the exposure lifting degree corresponding to each cost return rate lifting degree can be predicted based on data in the table by a method such as linear interpolation; and so on.
It is to be understood that the above description is merely illustrative and exemplary, and that one or more predetermined cost-return-rate boost levels may be flexibly set and corresponding exposure boost levels predicted according to actual needs.
In step 350, the target recommendation information may be determined as potential recommendation information in response to the predicted exposure boost for at least one of the one or more predetermined cost rate of return boosts being greater than or equal to the corresponding boost threshold. Alternatively, the threshold of the degree of improvement may be fixed, or may be variable according to the characteristics of the target recommendation information and/or the predetermined degree of improvement of the cost return rate. In some embodiments, the boost threshold may depend on a cost return boost corresponding to the predicted exposure boost. For example, the boost threshold may comprise a product of a predetermined cost-return rate boost corresponding to the predicted exposure boost and a preset threshold. And optionally, the boost threshold may be equal to a product of a predetermined cost rate of return boost corresponding to the predicted exposure boost and a preset threshold, or equal to an absolute value of a product of a predetermined cost rate of return boost corresponding to the predicted exposure boost and a preset threshold, or other value determined based on the product. The preset threshold may be determined according to actual needs, and may be, for example, 1, 2, 2.5, 2.8, 3, or the like. Alternatively, in some embodiments, the boost threshold may include a sum, a difference, a ratio of a predetermined cost rate of return boost corresponding to the predicted exposure boost to a preset threshold, or may be determined according to a square, a cube, a logarithm, etc. of the predetermined cost rate of return boost corresponding to the predicted exposure boost.
It is to be understood that when the condition in step 350 is not satisfied, that is, when the predicted exposure boost for each of the one or more predetermined cost rate of return boosts is less than the corresponding boost threshold, the target recommendation information is not determined to be potential recommendation information or is determined to be non-potential recommendation information.
The method 300 provides a solution for determining potential recommendation information for which a desired cost-benefit rate is set. The method comprises the steps of establishing a relation between exposure improvement degrees and cost rate of return improvement degrees based on historical sorting data related to target recommendation information, predicting the exposure improvement degrees of the cost rate of return under each preset cost rate of return improvement degree through the relation, and determining whether the target recommendation information is potential recommendation information according to a predicted value. The determined potential recommendation information may achieve a desired exposure boost at a certain boost in cost-benefit rate. Therefore, the potential recommendation information can help the delivering party to reasonably configure the resource cost so as to obtain the exposure promotion degree meeting the expectation, and further possibly obtain better recommendation information delivering effect and final benefit.
In some embodiments, each of the sets of historical ranking data mentioned in step 310 of FIG. 3 may include at least one of: exposure cost of the target recommendation information, exposure cost of recommendation information winning in the ranking, identification indicating that the target recommendation information wins in the ranking. In such an embodiment, step 320 shown in FIG. 3 may be performed according to the flowchart shown in FIG. 4A. However, it should be understood that although the sub-steps of step 320 are shown in fig. 4A, this is merely exemplary, and in fact, step 320 may be executed according to other procedures, such as determining the relationship between the exposure boost degree and the exposure cost boost degree of the target recommendation information according to other items in the historical sorting data, or acquiring the relationship between the exposure boost degree and the exposure cost boost degree of the target recommendation information according to multiple sets of historical sorting data, for example, by means of machine learning, and so on.
Specifically, in fig. 4A, in step 321, a ratio of the exposure cost of the winning recommendation information to the exposure cost of the target recommendation information may be determined based on each of the sets of history ranking data.
Illustratively, the historical ordering data may be data related to the above-mentioned refinement process, and the exposure cost may be characterized by eCPM. Suppose that the target recommendation information enters the refinement process n times within a certain time period (e.g., within 1 hour), wherein m times (m ≦ n) are won and (n-m) times are not won, i.e., m times (n-m) are exposed. For each set of historical ranking data, a ratio of the exposure cost of the winning recommendation information to the exposure cost of the target recommendation information, i.e., a ratio of the eCPM of the winning recommendation information to the eCPM of the target recommendation information, in this example, may be determined. Obviously, in the m sets of historical sorting data with the winning target recommendation information, the ratio is 1; in the remaining (n-m) sets of historical ranking data, the ratio is x, where x ranges from (1, + ∞) to x 1 , ..., x n-m 。
Alternatively, step 321 may be performed based on other types of historical ordering data and/or exposure cost characterization.
At step 322, the exposure boost of the target recommendation information when the exposure cost boost of the target recommendation information is equal to the ratio minus one may be determined for the ratio determined based on each set of historical ranking data.
Illustratively, in the example detailed above with respect to step 321, the determined odds may be further sorted in order from smaller to larger to obtain a sequence of length n, i.e., [1, 1, \ 8230;, 1, x 1 , x 2 , …, x n-m ]Wherein the number of 1 is m. It will be appreciated that, in an ideal case, when the exposure cost of the target recommendation information (in this example, eCPM) reaches x of the original exposure cost 1 Multiple times, i.e. when new _ ecpm = (x) 1 * origi _ ecpm), i.e. when the order is correctThe exposure cost promotion degree of the standard recommendation information is equal to x 1 1, the target recommendation information will be at the ratio x 1 The corresponding refined ranking process becomes the winning recommendation information, that is, the target recommendation information is obtained more
Carrying out multiple exposure; when the exposure cost of the target recommendation information (eCPM in this example) reaches x of the original exposure cost 2 Multiple times, i.e. when new _ ecpm = (x) 2 * original _ ecpm), that is, when the exposure cost promotion degree of the target recommendation information is equal to x 2 1, the target recommendation information will be at the ratio x 1 And x 2 The corresponding refined ranking process becomes the winning recommendation information, that is, the target recommendation information is obtained more
Carrying out multiple exposure; by analogy, can aim at x 3 , …, x n-m Determines the exposure boost of the target recommendation information.
At step 323, a relationship between the exposure boost and the exposure cost boost of the target recommendation information may be determined based on the plurality of sets of exposure cost boosts and exposure boosts determined for the plurality of sets of historical sorting data.
In some embodiments, the relationship between the degree of exposure boost and the degree of exposure cost boost of the target recommendation information may be determined according to a polynomial fit. Specifically, polynomial fitting may be performed on the multiple sets of exposure cost improvement degrees and exposure improvement degrees determined in step 322 to determine parameters corresponding to terms in the polynomial; and determining the relationship between the exposure promotion degree and the exposure cost promotion degree of the target recommendation information according to the parameters corresponding to the items in the determined polynomial.
Illustratively, in the example described in detail above with respect to steps 321 and 322, the exposure boost and the exposure boost may be based on the determined sets of exposure cost boost and exposure boost, i.e., (x) 1 -1，
）、（x 2 -1，
）、...、（x n-m -1，
) A polynomial fitting, such as a third-order polynomial fitting, is performed to determine parameters corresponding to each term in the polynomial, and then a relation y = f (x) between the exposure boost and the exposure cost boost of the target recommendation information is obtained according to the determined parameters, where x is the exposure cost boost and y is the exposure boost.
Alternatively, a different scheme from the above example may be adopted to determine the relationship between the degree of increase in exposure and the degree of increase in exposure cost of the target recommendation information. For example, other types of functions may be employed to fit the determined sets of exposure cost boosters and exposure boosters. Alternatively, the relationship between the exposure cost increase degree and the exposure increase degree may be determined by using a plotting chart, machine learning, or the like.
Through the flowchart shown in fig. 4A, the relationship between the exposure boost degree and the exposure cost boost degree of the target recommendation information can be determined more easily and conveniently based on the obtained multiple sets of historical sorting data, so that the method is beneficial to keeping lower computation complexity, and is further beneficial to improving the determination speed and saving computation resources, storage resources and the like.
Fig. 4B schematically illustrates an example flow diagram of another substep 330 in fig. 3 according to some embodiments of the present disclosure. It should be understood that although the substeps of step 330 are shown in fig. 4B, this is merely exemplary, and step 330 may actually be performed according to other flows.
Specifically, in step 331, a relationship between the degree of increase in the exposure cost and the degree of increase in the cost return rate, which is determined based on a negative correlation between the exposure cost and the cost return rate, may be obtained. Alternatively, the relationship between the exposure cost boost and the cost rate of return boost may be fixed or variable, and the server or host performing step 330 may retrieve the relationship from a local or external storage device, or may determine the relationship itself.
In some embodiments, the exposure cost may be characterized by eCPM and the cost return may be characterized by ROI. In such an embodiment, the ROI may be calculated as a ratio of the total profit to the total cost associated with the target recommendation information, i.e. ROI = total profit/total cost, and the calculation formula may be rewritten as:
the CPM of the recommendation information should ideally be equal to eCPM, and in practical applications, CTR, CVR and LTV can be estimated by their predictors pCTR, pCVR and pLTV, respectively. Thus, the above equation can be further rewritten as:
coefficients are used to refer to adjustment coefficients that may be present, which are not relevant to the disclosed aspects and are not described in detail herein. Therefore, the eCPM is negatively correlated with the ROI, the eCPM can be improved by reducing the ROI, and the exposure probability of the target recommendation information can be increased. According to the relationship between the eCPM and the ROI derived above, the relationship between the eCPM increase and the ROI increase can be described by the following formula:
wherein z is the ROI enhancement, and g (z) is the eCPM enhancement. Thus, in the above embodiment, the relationship between the degree of improvement of the exposure cost and the degree of improvement of the cost return rate is determined.
Alternatively, in some embodiments, the exposure cost and the cost return may be characterized by other quantities, and in such embodiments, the relationship between the degree of increase in exposure cost and the degree of increase in cost return may be determined by a different process than the example process described above.
In step 332, a relationship between the exposure boost degree and the cost return rate boost degree of the target recommendation information may be determined based on the relationship between the exposure boost degree and the exposure cost boost degree and the obtained relationship. For example, the relationship between the exposure boost degree and the cost return rate boost degree may be determined based on the relationship between the exposure boost degree and the exposure cost boost degree and the relationship between the exposure cost boost degree and the cost return rate boost degree obtained in step 331, with the exposure cost boost degree as an intermediate amount.
Illustratively, in the embodiment described in detail above with respect to step 331, the relationship between the degree of increase in exposure cost and the degree of increase in cost return rate may be determined as
Thus, the relational expression can be substituted into the relation between the degree of exposure improvement and the degree of exposure cost improvement, that is, by
The exposure cost boost is replaced to determine the relationship between the exposure boost and the cost return rate boost.
Further, for example, in the embodiment described in detail above with respect to the steps of fig. 4A, assuming that the relationship between the degree of exposure boost and the degree of cost return boost therein can be determined according to the embodiment described in detail with respect to fig. 4B, the relationship between the degree of exposure boost and the degree of exposure cost boost and the relationship between the degree of exposure cost boost and the degree of cost return boost can be based on the target recommendation information y = f (x)
The composite function y = f (g (z)) is obtained, which may be further denoted as y = h (z), where z is the lifting degree of the ROI and y is the exposure lifting degree. In this case, when the corresponding exposure elevating degrees are determined for a plurality of ROIs elevating degrees of-0.01, -0.02, -0.03, and-0.04, it can be determined by calculating h (-0.01), h (-0.02), h (-0.03), and h (-0.04), respectively, andwhen any one or more of h (-0.01), h (-0.02), h (-0.03), and h (-0.04) is greater than or equal to the corresponding threshold value of the lifting degree, the target recommendation information may be considered as potential recommendation information; when the corresponding exposure boost degrees are determined for a plurality of ROI boost degrees-0.1, -0.2, -0.3 and-0.4, it may be determined by calculating h (-0.1), h (-0.2), h (-0.3) and h (-0.4), respectively, and further, when any one or more of h (-0.1), h (-0.2), h (-0.3) and h (-0.4) is greater than or equal to the corresponding boost degree threshold value, the target recommendation information may be considered as potential recommendation information. Further, when the threshold of the lifting degree is the product of the lifting degree of the cost return rate corresponding to the predicted lifting degree of the exposure and the preset threshold, the judgment can be passed
Or
Whether z is the corresponding cost-rate-of-return improvement and t is a predetermined threshold, which may be predetermined as needed as described above.
Through the flowchart shown in fig. 4B, the relationship between the exposure improvement degree and the cost return rate improvement degree can be determined more easily and accurately, which helps to keep the computation complexity low and helps to determine whether the target recommendation information is the potential recommendation information quickly and accurately.
Fig. 5 schematically illustrates an example flow diagram of a method 300' for determining potential recommendation information, in accordance with some embodiments of the present disclosure. Similar to the method 300, the method 300' may be deployed on the server 110 shown in fig. 1 or a virtual host therein in the form of an application or the like to determine the potential recommendation information.
Method 300' also includes steps 310-350, as compared to method 300, except that step 360 is also included. The steps 310-350 included in the method 300' are the same as the steps 310-350 described with respect to fig. 3 and fig. 4A, 4B and are not repeated here.
In step 360, in response to the target recommendation information being determined to be potential recommendation information, the target recommendation information may be classified so as to be one of final potential recommendation information and non-final potential recommendation information.
In some embodiments, when it is determined whether the plurality of pieces of recommendation information are potential recommendation information according to the method 300 in sequence, the number of determined potential recommendation information may be large, and at this time, the potential recommendation information may be secondarily screened through the step 360, so as to obtain more simplified final potential recommendation information. This helps to determine high-potential recommendation information more accurately to better assist the placement party in resource configuration.
In some embodiments, the classification process in step 360 may be implemented by means of a machine learning model, i.e., in response to the target recommendation information being determined to be potential recommendation information, the target recommendation information may be classified using a trained machine learning model. Specifically, the features of the target recommendation information may be input to the trained machine learning model and determined as final potential recommendation information or non-final potential recommendation information according to the classification result output by the trained machine learning model, where the features of the target recommendation information may include a price raising ratio of the target recommendation information, the price raising ratio being determined based on a ratio of the initial cost return rate and the promoted cost return rate. For example, the price-raising ratio may be equal to
Or may be otherwise connected with
The relevant amount. Illustratively, the price-raising proportion may be determined according to at least one predetermined cost-rate-of-return (RTR) raise mentioned in step 350 at which the predicted exposure raise is greater than or equal to a corresponding raise threshold.
Alternatively, the XGBoost model may be used to perform the classification process described above. XGBoost is a binary model and the output values may be classified into 0 and 1, where 0 may correspond to non-final potential recommendation information and 1 may correspond to final potential recommendation information. Alternatively, the output value of the XGBoost model may be compared to a threshold, and if greater than the threshold, it is classified as 1, otherwise it is classified as 0. Optionally, the accuracy of the model and/or the amount of final potential recommendation information can be adjusted by adjusting the threshold. Illustratively, the threshold may be set to 0.7 or other value. Further, other machine learning models may alternatively be used to perform the classification process described above, such as logistic regression models, fully connected neural networks, support vector machines, GBDTs, lightgbms, and the like.
In some embodiments, the features of the target recommendation information input to the trained machine learning model include, in addition to the bid proportion, one or more of: the target recommendation information comprises an identifier of the target recommendation information, a release party of the target recommendation information, an exposure amount of the target recommendation information in a latest statistical period, a release optimization type of the target recommendation information, and a release site of the target recommendation information, wherein the release optimization type can be optimized for conversion behaviors, optimized for cost return rate, and the like. Illustratively, the one or more features may be encoded into a vector along with the bid ratio and optionally other features and input to a trained machine learning model.
In some embodiments, the machine learning model described above may be trained by the method 600 shown in FIG. 6. The training process may be performed on the server or host that performs the method 300 or 300', or may be performed on other servers or hosts as well.
Specifically, at step 610, historical pricing data of a plurality of recommendation information may be obtained, each historical pricing data including a historical pricing proportion and a corresponding exposure boost, wherein the pricing proportion is determined based on one of: a ratio of an initial cost return rate to an enhanced cost return rate of the recommendation information, and a ratio of an enhanced cost to an initial cost of the recommendation information.
In some embodiments, only recommendations that set a desired rate of return on cost may be obtainedHistorical price-raising data of the information, the price-raising proportion can be determined according to the ratio of the initial cost return rate and the raised cost return rate of the recommendation information, for example, the price-raising proportion can be equal to
Or may be otherwise connected with
The amount of correlation. In some embodiments, since the number of pieces of recommendation information for which a desired rate of return on cost is set may be small, the historical pricing data of recommendation information for which a desired rate of return on cost is set and the historical pricing data of recommendation information for which a desired rate of return on cost is not set may be acquired at the same time. In this embodiment, for the recommendation information for which the expected cost return rate is set, the price raising proportion may be determined according to a ratio of the initial cost return rate of the recommendation information to the promoted cost return rate; for recommendation information for which a desired cost return rate is not set, a bid proportion may be determined from a ratio of an elevated cost and an initial cost of the recommendation information, the cost may be represented by a bid set for an exposure, click, or particular conversion behavior, for example, the bid proportion may be determined from a ratio of an elevated bid and an initial bid of the recommendation information. Illustratively, the price-raising ratio may be determined according to the following formula:
at step 620, recommendation information with exposure boost greater than or equal to the boost threshold may be marked as final potential recommendation information, otherwise marked as non-final potential recommendation information.
Illustratively, the recommendation information may be labeled as final potential recommendation information or non-final potential recommendation information depending on whether the exposure boost reaches the corresponding boost threshold within three hours after the boost bid or the boost cost return rate. The threshold for the degree of lift here may be set similarly to the threshold for the degree of lift in step 350. For example, the recommendation information may be marked as final potential recommendation information or non-final potential recommendation information according to whether the exposure boost degree reaches 2 times of the absolute value of the cost boost degree or the cost return rate boost degree. Alternatively, the final potential recommendation information may be labeled with class 1 and the non-final potential recommendation information may be labeled with class 0.
At step 630, the machine learning model may be trained using historical pricing data for the labeled plurality of recommendation information such that the error of the classification result output by the machine learning model and the label of each labeled recommendation information is minimized.
For example, a feature of each of the labeled plurality of recommendation information may be encoded and the encoded feature may be input to a machine learning model. Model parameters in the machine learning model may then be adjusted to reduce the error based on the error between the classification result output by the machine learning model and the label of the corresponding recommendation information. This process may be repeated so that the error of the classification result output by the machine learning model and the labeling of each labeled recommendation information is minimized. It should be understood that the feature types and feature encoding methods used in training the machine learning model should generally be consistent with the feature types and feature encoding methods used in using the trained machine learning model.
The machine learning model trained by the method can quickly and accurately determine whether the target recommendation information is the final potential recommendation information, so that the final potential recommendation information can be screened out from a plurality of potential recommendation information quickly and accurately.
In some embodiments, in response to the target recommendation information being determined to be potential recommendation information, a potential tag may be generated for the target recommendation information; alternatively, in some embodiments, a potential tag may be generated for the target recommendation information in response to the target recommendation information being determined to be potential recommendation information and being determined to be final potential recommendation information. The generated potential mark can be sent to the releasing party client, and when the releasing party client receives the potential mark aiming at the target recommendation information, an identifier indicating that the target recommendation information has potential can be generated and presented.
Illustratively, the publisher client may present an indication of the potential for recommendation information in an interface 700A as shown in FIG. 7A. FIG. 7A illustrates an example of an advertisement scenario, schematically showing an interface 700A associated with an ROI advertisement. Illustratively, the corresponding recommendation information may be indicated to be potential through the "latent" word identifier 701.
By generating a potential mark for the target recommendation information and generating and presenting an identifier indicating that the recommendation information has potential on the client of the releasing party through the potential mark, the releasing party can more conveniently and quickly know which recommendation information is potential recommendation information.
In some embodiments, a request to view the identification from the publisher client may also be intercepted, and in response to intercepting such a request, a corresponding exposure boost predicted for at least one of the one or more predetermined cost rate of return boosts may be sent to the publisher client. Illustratively, the "latent" word identifier 701 presented in the interface 700A may be viewed, e.g., a user may view the identifier by clicking on the identifier 701, hovering a pointer over the identifier 701, or the like. The releasing party client side can send a request for checking the identifier to the server when the checking operation is detected. When the server monitors the request, it may send, to the delivering client, a corresponding exposure boost predicted for at least one predetermined cost rate of return boost in the one or more predetermined cost rate of return boosts, and optionally determine and send other data such as a recommended cost rate of return boost according to the at least one predetermined cost rate of return boost and the corresponding exposure boost. After receiving the data, the client of the delivering party can process the received data and present the data to the user in any sensible manner.
Illustratively, the publisher client may present the received data as shown in interface 700B shown in FIG. 7B. FIG. 7B also illustrates an example of an advertisement scenario, schematically showing an interface 700B associated with an ROI advertisement. Interface 700B includes a potential advertisement window 702. The window 702 may include the system recommended ROI, the corresponding ROI adjustment scale, and the predicted future 1 hour exposure (including exposure and corresponding exposure boost), and may also include a chart showing the predicted future 1 hour exposure and exposure upshifts/ROI downshifts for multiple ROIs. Further, window 702 can include an option that allows the publisher to one-click optimize the corresponding ad placement according to the system recommended ROI. Additionally or alternatively, other information may be presented in the window 702 as desired.
By monitoring the request for viewing the identifier and sending the corresponding predicted exposure improvement degree for at least one predetermined cost return rate improvement degree in the one or more predetermined cost return rate improvement degrees to the client of the launching party, the launching party can acquire more detailed data of the potential recommendation information and/or the final potential recommendation information, and therefore the cost return rate of the recommendation information can be adjusted more conveniently to obtain the expected exposure improvement degree.
Further, in some embodiments, data regarding the amount of potential recommendation information and/or final potential recommendation information may also be sent to the publisher client, or the publisher client may count the data on its own and present in an interface 700C as shown in fig. 7C. FIG. 7C also illustrates an advertisement scenario, schematically showing an advertisement statistics interface 700C. Illustratively, the number of potential ads, and optionally the expected upscaling of exposure that can be obtained by adjusting the ROI, can be presented as indicated by reference numeral 703. This helps the delivery party to macroscopically understand the number of potential recommendations and optional expected exposure rises in the currently delivered recommendations.
Fig. 8 schematically illustrates an apparatus 800 for determining potential recommendation information according to some embodiments of the present disclosure. As shown in fig. 8, the apparatus 800 includes an obtaining module 810, a first determining module 820, a second determining module 830, a predicting module 840, and a third determining module 850.
Specifically, the obtaining module 810 may be configured to obtain multiple sets of historical ranking data associated with the target recommendation information, where the historical ranking data includes data related to multiple candidate recommendation information and one winning recommendation information, and the winning recommendation information is exposure recommendation information determined from the multiple candidate recommendation information; the first determining module 820 may be configured to determine a relationship between the exposure boost degree and the exposure cost boost degree of the target recommendation information based on the obtained sets of historical sorting data; the second determining module 830 may be configured to determine a relationship between the exposure boost degree and the cost return rate boost degree of the target recommendation information based on the determined relationship between the exposure boost degree and the exposure cost boost degree; the prediction module 840 may be configured to predict, for one or more predetermined cost-rate-of-return enhancements, corresponding exposure enhancements according to the determined relationship between exposure enhancements and cost-rate-of-return enhancements; and the third determination module 850 may be configured to determine the target recommendation information as potential recommendation information in response to the predicted exposure boost for at least one of the one or more predetermined cost rate of return boosts being greater than or equal to the corresponding boost threshold.
It should be understood that the apparatus 800 may be implemented in software, hardware, or a combination of software and hardware. Several different modules may be implemented in the same software or hardware configuration, or one module may be implemented by several different software or hardware configurations.
Furthermore, the apparatus 800 may be used to implement the methods 300 or 300' described above, the relevant details of which have been described in detail above and will not be repeated here for the sake of brevity. The apparatus 800 may have the same features and advantages as described with respect to the previous method.
Fig. 9 schematically illustrates an example block diagram of a computing device 900 in accordance with some embodiments of this disclosure. For example, which may represent the server 110 of fig. 1 or other type of computing device that may be used to deploy the apparatus 800 provided by the present disclosure.
As shown, the example computing device 900 includes a processing system 901, one or more computer-readable media 902, and one or more I/O interfaces 903 communicatively coupled to each other. Although not shown, the computing device 900 may also include a system bus or other data and command transfer system that couples the various components to one another. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures, or that also includes data lines, such as control and data lines.
The computer-readable medium 902 is illustrated as including a memory/storage 905. Memory/storage 905 represents memory/storage associated with one or more computer-readable media. The memory/storage 905 may include volatile storage media (such as Random Access Memory (RAM)) and/or nonvolatile storage media (such as Read Only Memory (ROM), flash memory, optical disks, magnetic disks, and so forth). The memory/storage 905 may include fixed media (e.g., RAM, ROM, a fixed hard drive, etc.) as well as removable media (e.g., flash memory, a removable hard drive, an optical disk, and so forth). Illustratively, the memory/storage 905 may be used to store the historical ranking data, the expected rate of return on cost, the determined exposure boost level versus exposure cost boost level, the determined exposure cost boost level versus rate of return on cost boost level, and the like mentioned in the above embodiments. The computer-readable medium 902 may be configured in various other ways, which are further described below.
One or more input/output interfaces 903 represent functionality that allows a user to enter commands and information to computing device 900, and also allows information to be presented to the user and/or sent to other components or devices using various input/output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, touch functionality (e.g., capacitive or other sensors configured to detect physical touch), a camera (e.g., motion that does not involve touch may be detected as gestures using visible or invisible wavelengths such as infrared frequencies), a network card, a receiver, and so forth. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a haptic response device, a network card, a transmitter, and so forth. For example, in the above-described embodiments, the administrator may be allowed to input various preset thresholds and the like through the input device, and the administrator may be allowed to monitor, view and the like the execution process and/or the result of the method of determining the potential recommendation information through the output device.
Various techniques may be described herein in the general context of software, hardware, components, or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The terms "module," "functionality," and the like, as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that the techniques may be implemented on a variety of computing platforms having a variety of processors.
An implementation of the described modules and techniques may be stored on or transmitted across some form of computer readable media. Computer readable media can include a variety of media that can be accessed by computing device 900. By way of example, and not limitation, computer-readable media may comprise "computer-readable storage media" and "computer-readable signal media".
"computer-readable storage medium" refers to a medium and/or device, and/or a tangible storage apparatus, capable of persistently storing information, as opposed to mere signal transmission, carrier wave, or signal per se. Accordingly, computer-readable storage media refers to non-signal bearing media. Computer-readable storage media include hardware such as volatile and nonvolatile, removable and non-removable media and/or storage devices implemented in a method or technology suitable for storage of information such as computer-readable instructions, data structures, program modules, logic elements/circuits, or other data. Examples of computer readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical storage, hard disks, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other storage devices, tangible media, or an article of manufacture suitable for storing the desired information and accessible by a computer.
"computer-readable signal medium" refers to a signal-bearing medium configured to transmit instructions to the hardware of computing device 900, such as via a network. Signal media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave, data signal or other transport mechanism. Signal media also includes any information delivery media. By way of example, and not limitation, signal media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.
As previously described, hardware elements 901 and computer-readable media 902 represent instructions, modules, programmable device logic, and/or fixed device logic implemented in hardware form that may be used in some embodiments to implement at least some aspects of the techniques described herein. The hardware elements may include integrated circuits or systems-on-chips, application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), complex Programmable Logic Devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, a hardware element may serve as a processing device that performs program tasks defined by instructions, modules, and/or logic embodied by the hardware element, as well as a hardware device for storing instructions for execution, such as the computer-readable storage medium described previously.
Combinations of the foregoing may also be used to implement the various techniques and modules described herein. Thus, software, hardware, or program modules and other program modules may be implemented as one or more instructions and/or logic embodied on some form of computer-readable storage medium and/or by one or more hardware elements 901. Computing device 900 may be configured to implement particular instructions and/or functions corresponding to software and/or hardware modules. Thus, implementing modules as modules executable by computing device 900 as software may be implemented at least partially in hardware, for example, using computer-readable storage media of a processing system and/or hardware elements 901. The instructions and/or functions may be executed/operable by, for example, one or more computing devices 900 and/or processing systems 901 to implement the techniques, modules, and examples described herein.
The techniques described herein may be supported by these various configurations of computing device 900 and are not limited to specific examples of the techniques described herein.
It will be appreciated that embodiments of the disclosure have been described with reference to different functional units for clarity. However, it will be apparent that the functionality of each functional unit may be implemented in a single unit, in a plurality of units or as part of other functional units without departing from the disclosure. For example, functionality illustrated to be performed by a single unit may be performed by a plurality of different units. Thus, references to specific functional units are only to be seen as references to suitable units for providing the described functionality rather than indicative of a strict logical or physical structure or organization. Thus, the present disclosure may be implemented in a single unit or may be physically and functionally distributed between different units and circuits.
The present disclosure provides a computer-readable storage medium having computer-readable instructions stored thereon that, when executed, implement the above-described method for determining potential recommendation information.
The present disclosure provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions are read by a processor of the computing device from the computer-readable storage medium, and the computer instructions are executed by the processor to cause the computing device to perform the method for determining the potential recommendation information provided in the various alternative implementations described above.
Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed subject matter, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
Claims (15)
1. A method for determining potential recommendation information, comprising:
acquiring multiple groups of historical sorting data associated with target recommendation information, wherein the historical sorting data comprises data related to multiple candidate recommendation information and winning recommendation information, and the winning recommendation information is exposure recommendation information determined from the multiple candidate recommendation information;
determining the relationship between the exposure promotion degree and the exposure cost promotion degree of the target recommendation information based on the obtained multiple groups of historical sorting data;
determining a relationship between the exposure promotion degree and the cost return rate promotion degree of the target recommendation information based on the determined relationship between the exposure promotion degree and the exposure cost promotion degree;
predicting a corresponding exposure boost according to one or more predetermined cost return rate boost according to the determined relationship between the exposure boost and the cost return rate boost; and
determining the target recommendation information as potential recommendation information in response to the predicted exposure boost for at least one of the one or more predetermined cost-rate-of-return boosts being greater than or equal to a corresponding boost threshold.
2. The method of claim 1, wherein the threshold of boost depends on a degree of cost return boost corresponding to the predicted degree of exposure boost.
3. The method of claim 1, wherein each of the sets of historical ranking data comprises at least one of: an exposure cost of the target recommendation information, an exposure cost of the recommendation information that wins in the ranking, an identification indicating that the target recommendation information wins in the ranking, an
Wherein the determining the relationship between the exposure boost degree and the exposure cost boost degree of the target recommendation information based on the obtained multiple sets of historical sorting data comprises:
determining the ratio of the exposure cost of the winning recommendation information to the exposure cost of the target recommendation information based on each group of historical sorting data in the multiple groups of historical sorting data;
determining the exposure promotion degree of the target recommendation information when the exposure cost promotion degree of the target recommendation information is equal to the ratio minus one aiming at the ratio determined based on each group of historical sorting data; and
determining a relationship between the exposure boost degree and the exposure cost boost degree of the target recommendation information based on the plurality of sets of exposure cost boost degrees and the exposure boost degrees determined for the plurality of sets of historical sorting data.
4. The method of claim 3, wherein the determining a relationship between the exposure boost and the exposure cost boost for the target recommendation information based on the determined sets of exposure cost boost and exposure boost for the sets of historical ordering data comprises:
aiming at the multiple groups of exposure cost promotion degrees and exposure promotion degrees, polynomial fitting is carried out so as to determine parameters corresponding to all terms in a polynomial; and
and determining the relationship between the exposure promotion degree and the exposure cost promotion degree of the target recommendation information according to the parameters corresponding to the items in the determined polynomial.
5. The method of claim 1, wherein the determining a relationship between the exposure boost and the cost return rate boost for the target recommendation information based on the determined relationship between the exposure boost and the exposure cost boost comprises:
obtaining a relationship between the exposure cost improvement degree and the cost return rate improvement degree, wherein the relationship is determined based on a negative correlation relationship between the exposure cost and the cost return rate; and
and determining the relationship between the exposure promotion degree and the cost return rate promotion degree of the target recommendation information based on the relationship between the exposure promotion degree and the exposure cost promotion degree and the obtained relationship.
6. The method of claim 1, further comprising:
in response to the target recommendation information being determined to be potential recommendation information, performing classification processing on the target recommendation information so as to classify the target recommendation information into one of final potential recommendation information and non-final potential recommendation information.
7. The method of claim 6, wherein in response to the target recommendation information being determined to be potential recommendation information, performing classification processing on the target recommendation information comprises:
in response to the target recommendation information being determined to be potential recommendation information, performing a classification process on the target recommendation information using a trained machine learning model, comprising:
inputting the characteristics of the target recommendation information into the trained machine learning model, and determining the target recommendation information as final potential recommendation information or non-final potential recommendation information according to a classification result output by the trained machine learning model, wherein the characteristics of the target recommendation information comprise a price raising proportion of the target recommendation information, and the price raising proportion is determined based on a ratio of an initial cost return rate and a raised cost return rate.
8. The method of claim 7, wherein the characteristics of the target recommendation information further include one or more of: the target recommendation information comprises an identification of the target recommendation information, a release party of the target recommendation information, an exposure amount of the target recommendation information in a latest statistical period, a release optimization type of the target recommendation information and a release site of the target recommendation information.
9. The method of claim 7, wherein the trained machine learning model is trained by:
obtaining historical price raising data of a plurality of recommendation information, wherein each piece of historical price raising data comprises a historical price raising proportion and a corresponding exposure raising degree, and the price raising proportion is determined based on one of the following items: the ratio of the initial cost return rate of the recommendation information to the promoted cost return rate, and the ratio of the promoted cost of the recommendation information to the initial cost;
marking the recommendation information with the exposure lifting degree larger than or equal to the lifting degree threshold value as final potential recommendation information, otherwise, marking the recommendation information as non-final potential recommendation information; and
training the machine learning model using historical pricing data of the labeled plurality of recommendation information such that an error of a classification result output by the machine learning model and a label of each labeled recommendation information is minimized.
10. The method of claim 1, further comprising:
in response to the target recommendation information being determined to be potential recommendation information, generating a potential tag for the target recommendation information, wherein the potential tag is configured to, when sent to a publisher client, cause the publisher client to generate and present an identification indicating that the target recommendation information is potential.
11. The method of claim 6, further comprising:
in response to the target recommendation information being determined to be final potential recommendation information, generating a potential tag for the target recommendation information, wherein the potential tag is configured to, when sent to a publisher client, cause the publisher client to generate and present an identification indicating that the target recommendation information is potential.
12. The method of claim 10 or 11, further comprising:
monitoring a request for viewing the identifier from a client of the delivering party; and
in response to monitoring the request, sending, to the publisher client, a corresponding exposure boost predicted for at least one of the one or more predetermined cost rate of return boosts.
13. An apparatus for determining potential recommendation information, comprising:
an obtaining module configured to obtain a plurality of sets of historical ranking data associated with target recommendation information, wherein the historical ranking data includes data related to a plurality of candidate recommendation information and one winning recommendation information, and the winning recommendation information is recommendation information of exposure determined from the plurality of candidate recommendation information;
a first determining module configured to determine a relationship between an exposure promotion degree and an exposure cost promotion degree of the target recommendation information based on the obtained multiple sets of historical sorting data;
a second determining module configured to determine a relationship between the exposure boost degree and the cost return rate boost degree of the target recommendation information based on the determined relationship between the exposure boost degree and the exposure cost boost degree;
a prediction module configured to predict, for one or more predetermined cost-return-rate boost degrees, a corresponding exposure boost degree according to the determined relationship between the exposure boost degree and the cost-return-rate boost degree; and
a third determination module configured to determine the target recommendation information as potential recommendation information in response to a predicted exposure boost for at least one of the one or more predetermined cost rate of return boosts being greater than or equal to a corresponding boost threshold.
14. A computing device, comprising:
a memory configured to store computer-executable instructions;
a processor configured to perform the method of any one of claims 1 to 12 when the computer-executable instructions are executed by the processor.
15. A computer-readable storage medium storing computer-executable instructions that, when executed, perform the method of any one of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110863912.1A CN115700692A (en) | 2021-07-29 | 2021-07-29 | Method and device for determining potential recommendation information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110863912.1A CN115700692A (en) | 2021-07-29 | 2021-07-29 | Method and device for determining potential recommendation information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115700692A true CN115700692A (en) | 2023-02-07 |
Family
ID=85120732
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110863912.1A Pending CN115700692A (en) | 2021-07-29 | 2021-07-29 | Method and device for determining potential recommendation information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115700692A (en) |
-
2021
- 2021-07-29 CN CN202110863912.1A patent/CN115700692A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11694221B2 (en) | Dynamically modifying digital content distribution campaigns based on triggering conditions and actions | |
| US20220101202A1 (en) | Utilizing machine learning and composite utility scores from multiple event categories to improve digital content distribution | |
| US10943267B2 (en) | Machine learning assisted target segment audience generation | |
| US7016936B2 (en) | Real time electronic service interaction management system and method | |
| US20170277691A1 (en) | Quantifying Social Influence | |
| JP5662446B2 (en) | A learning system for using competitive evaluation models for real-time advertising bidding | |
| US8725559B1 (en) | Attribute based advertisement categorization | |
| US11556960B2 (en) | Methods and apparatus to deliver targeted advertising | |
| US20110320395A1 (en) | Optimization of Multi-channel Commerce | |
| US20160189201A1 (en) | Enhanced targeted advertising system | |
| US20120290373A1 (en) | Apparatus and method for marketing-based dynamic attribution | |
| US11127032B2 (en) | Optimizing and predicting campaign attributes | |
| US11875380B2 (en) | Method for matching influencers to brands and pairing influencer videos to products for intra-video transactions | |
| KR102191486B1 (en) | Automatic advertisement execution device, method for automatically generating campaign information for an advertisement medium to execute an advertisement and computer program for executing the method | |
| US20200342500A1 (en) | Systems and methods for self-serve marketing pages with multi-armed bandit | |
| US20230230183A1 (en) | Intelligent Prediction of An Expected Value of User Conversion | |
| US11647238B2 (en) | Generation and delivery of content via remote rendering and data streaming | |
| US11972454B1 (en) | Attribution of response to multiple channels | |
| US10909567B1 (en) | Systems and methods to manage feedback for a multi-arm bandit algorithm | |
| US20180150886A1 (en) | Controlling a content auction with a threshold value | |
| KR102397014B1 (en) | Method and system for providing advertising to execute an advertisement budget based on price competitiveness and the bottleneck for each advertisement target | |
| CN115700692A (en) | Method and device for determining potential recommendation information | |
| US20220358522A1 (en) | Artificial-intelligence-based orchestration | |
| CN115330426A (en) | Method and device for calibrating pre-estimated conversion rate, computing equipment and storage medium | |
| US20220284320A1 (en) | Using machine-learned models to throttle content |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |