CN113298573A - Content delivery strategy comparison method and device, readable medium and electronic equipment - Google Patents
Content delivery strategy comparison method and device, readable medium and electronic equipment Download PDFInfo
- Publication number
- CN113298573A CN113298573A CN202110679121.3A CN202110679121A CN113298573A CN 113298573 A CN113298573 A CN 113298573A CN 202110679121 A CN202110679121 A CN 202110679121A CN 113298573 A CN113298573 A CN 113298573A
- Authority
- CN
- China
- Prior art keywords
- content delivery
- strategy
- content
- policy
- evaluation index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012384 transportation and delivery Methods 0.000 title claims abstract description 356
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000011156 evaluation Methods 0.000 claims abstract description 81
- 230000003578 releasing effect Effects 0.000 claims description 45
- 238000012360 testing method Methods 0.000 claims description 36
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 4
- 230000035945 sensitivity Effects 0.000 description 4
- 230000006399 behavior Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012353 t test Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0242—Determining effectiveness of advertisements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0277—Online advertisement
Landscapes
- Business, Economics & Management (AREA)
- Engineering & Computer Science (AREA)
- Accounting & Taxation (AREA)
- Development Economics (AREA)
- Strategic Management (AREA)
- Finance (AREA)
- Game Theory and Decision Science (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Marketing (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The disclosure relates to a content delivery policy comparison method and device, a readable medium and an electronic device. The method comprises the following steps: sequentially carrying out multiple groups of content delivery on the target object, wherein a delivery result generated based on a first content delivery strategy and a delivery result generated based on a second content delivery strategy are delivered in each group of content delivery in a crossed manner; acquiring access data corresponding to a plurality of groups of released contents as sample data; according to the sample data, calculating an evaluation index, wherein the evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy; and determining a comparison result between the first content delivery strategy and the second content delivery strategy according to the evaluation index, wherein the comparison result is at least used for indicating whether the first content delivery strategy and the second content delivery strategy have a significant difference.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a content delivery policy comparison method and apparatus, a readable medium, and an electronic device.
Background
The AB test is to make two (A/B) or a plurality of (A/B/n) versions for a Web or App interface or process, respectively make visitor groups (target population) with the same (or similar) components randomly access the versions in the same time dimension, obtain user experience data and service data of each group, finally analyze and evaluate the best version, and formally adopt the version. In the related art, in the process of starting the AB test, a plurality of (dozens or even twenty) experiments are usually started on the same flow layer, each experiment occupies a part of flow, and each experiment cannot quickly obtain convincing conclusions, so that the flow is mutually blocked among the experiments, and the iteration efficiency is very low. Meanwhile, besides the efficiency problem, the AB test also has a statistically significant difference problem.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In a first aspect, the present disclosure provides a content delivery policy comparison method, where the method includes:
sequentially carrying out multiple groups of content delivery on the target object, wherein a delivery result generated based on a first content delivery strategy and a delivery result generated based on a second content delivery strategy are delivered in each group of content delivery in a crossed manner;
acquiring access data corresponding to a plurality of groups of released contents as sample data;
according to the sample data, calculating an evaluation index, wherein the evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy;
and determining a comparison result between the first content delivery strategy and the second content delivery strategy according to the evaluation index, wherein the comparison result is at least used for indicating whether the first content delivery strategy and the second content delivery strategy have a significant difference.
In a second aspect, the present disclosure provides a content delivery policy comparing apparatus, including:
the releasing module is used for sequentially releasing a plurality of groups of contents to the target object, wherein releasing results generated based on the first content releasing strategy and releasing results generated based on the second content releasing strategy are released in each group of contents in a cross mode;
the acquisition module is used for acquiring access data corresponding to a plurality of groups of released contents as sample data;
the index calculation module is used for calculating an evaluation index according to the sample data, wherein the evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy;
a first determining module, configured to determine, according to the evaluation index, a comparison result between the first content delivery policy and the second content delivery policy, where the comparison result is at least used to indicate whether there is a significant difference between the first content delivery policy and the second content delivery policy.
In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which, when executed by a processing apparatus, performs the steps of the method of the first aspect of the present disclosure.
In a fourth aspect, the present disclosure provides an electronic device comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to implement the steps of the method of the first aspect of the present disclosure.
According to the technical scheme, multiple groups of content delivery are sequentially carried out on a target object, delivery results generated based on a first content delivery strategy and delivery results generated based on a second content delivery strategy are delivered in each group of content delivery in a crossed mode, then, access data corresponding to multiple groups of delivered content are obtained and serve as sample data, evaluation indexes used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy are calculated, and comparison results between the first content delivery strategy and the second content delivery strategy are determined according to the calculated evaluation indexes, wherein the comparison results are at least used for indicating whether the first content delivery strategy and the second content delivery strategy have obvious differences. Therefore, the evaluation index aiming at the content delivery strategy is calculated, the selection of the user aiming at multiple deliveries is aggregated at the same time, so that different content delivery strategies are compared, and the comparison result of the different content delivery strategies can be quickly and accurately obtained based on the high sensitivity of the evaluation index, so that a more excellent strategy can be conveniently screened based on the comparison result in the follow-up process. Therefore, when the strategy used by the AB test is selected, the strategy used for the AB test can be quickly determined, and the efficiency of the AB test is favorably improved.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale. In the drawings:
fig. 1 is a flowchart of a content delivery policy comparison method provided according to an embodiment of the present disclosure;
FIG. 2 is an exemplary flowchart illustrating a step of calculating an evaluation index according to sample data in a content delivery policy comparison method according to the present disclosure;
fig. 3 is a block diagram of a content delivery policy comparison apparatus provided according to an embodiment of the present disclosure;
FIG. 4 shows a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
Fig. 1 is a flowchart of a content delivery policy comparison method provided according to an embodiment of the present disclosure. As shown in fig. 1, the method provided by the present disclosure may include the following steps 11 to 14.
In step 11, a plurality of sets of content delivery are performed sequentially for the target object.
In one possible embodiment, the target object may include at least one user. When content delivery is carried out, multiple groups of content delivery are respectively carried out for each user in the target object, and content delivery actions of different users are not affected mutually. Wherein different users can be distinguished by a user identification, e.g. a user ID.
In another possible embodiment, the target object may include at least one content presentation window. The content display window is an area for displaying the content to be released, and taking a mobile phone as an example, the content display window may occupy all or part of a display screen of the mobile phone. When content releasing is carried out, a plurality of groups of content releasing are respectively carried out on each content display window in the target object, and the content releasing actions of different content display windows are not influenced mutually. Wherein different content presentation windows may be distinguished by a window identification, e.g. a window ID.
And in each group of content delivery, cross-delivering the delivery result generated based on the first content delivery strategy and the delivery result generated based on the second content delivery strategy. The cross delivery means that two adjacent delivery actions should be performed based on different content delivery strategies. One of the first content delivery policy and the second content delivery policy may be a currently used content delivery policy, and the other may be a new content delivery policy, so that the new policy and the old policy may be compared.
For example, in a group of content delivery, content delivery may be included only twice, wherein one time is delivered based on the delivery result generated by the first content delivery policy, and the other time is delivered based on the delivery result generated by the second content delivery policy. For another example, a set of content delivery may include more than two content deliveries, and assuming that the set of content deliveries includes 4 content deliveries, if the delivery is performed based on the delivery result generated by the first content delivery policy for the first time, the delivery is performed based on the delivery result generated by the second content delivery policy for the second time, the delivery is performed based on the delivery result generated by the first content delivery policy for the third time, and the delivery is performed based on the delivery result generated by the second content delivery policy for the fourth time. Taking advertisement content delivery as an example, assuming that advertisement delivery is performed based on policy C1 and policy C2, in a group of content delivery, an advertisement generated based on policy C1 may be delivered first, and then an advertisement generated based on policy C2 may be delivered.
In step 12, access data corresponding to a plurality of sets of delivered content is acquired as sample data.
After a plurality of groups of content delivery, the user can see the content delivered each time, so that the user can give feedback on the delivered content, for example, click on the delivered content, close the delivered content, browse the delivered content for a certain time, and the like. Based on the method, the feedback behavior of the user can be obtained and used as the access data of the user for the released content.
In one possible implementation, access data corresponding to each set of one-shot content may be obtained as sample data.
In one possible implementation, only a portion of the total access data may be obtained as sample data. Illustratively, step 12 may include the steps of:
receiving a strategy comparison request carrying statistical range information;
and responding to the strategy comparison request, and acquiring access data corresponding to the statistical range indicated by the statistical range information from the access data corresponding to each content delivery as sample data.
Wherein the statistical range information is used for limiting the acquisition range of the access data. The statistical range information may be used to define a statistical range of time, objects, etc. For example, the statistical range information may be used to indicate a specified time period, so that, based on the statistical range information, only access data corresponding to each set of released content within the specified time period needs to be acquired as sample data. For another example, the statistical range data may be used to indicate a specified user, so that, based on the statistical range information, only the access data of the specified user corresponding to each set of released content needs to be acquired as sample data.
In step 13, an evaluation index is calculated from the sample data.
The evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy.
For example, the evaluation index may be calculated with reference to a metric of an Interleaving mechanism. In brief, the Interleaving mechanism combines the sorting results D and E from the two policies to present to the user, then obtains the user clicks, restores the clicks to D and E, and compares the clicks to determine the preference of the user for the sorting results, which is generally used for searching scenes. In Interleaving, the following metrics are set:
where wins (D) represents the number of D wins, wins (E) represents the number of E wins, and ties (D, E) represents the number of D, E levelings.
With reference to the above-mentioned metrics of Interleaving, the evaluation index used in the present disclosure is calculated. Let n be the total number of content deliveries in groups, nAFor the number of times that the first content delivery policy outperforms the second content delivery policy in the n sets of deliveries, nBFor the number of times that the second content delivery policy outperforms the first content delivery policy in the n sets of deliveries, ntIn order to find the number of times that the first content delivery policy is equal to the first content delivery policy in the n groups of deliveries, the following result can be obtained by performing conversion according to the formula of the metric:
thus, the evaluation index score can be calculated according to the following formula:
thus, by determining whether the evaluation index score is significantly non-zero, it can be determined whether there is a significant difference between the first content delivery policy and the second content delivery policy.
For example, if a set of impressions for which the impression results generated based on the first content impression policy were accessed more times than the impression results generated based on the second content impression policy, it may be determined that the first content impression policy is superior to the second content impression policy, and if a set of impressions for which the impression results generated based on the first content impression policy were accessed less times than the impression results generated based on the second content impression policy, it may be determined that the second content impression policy is superior to the first content impression policy.
For another example, if the duration of time that the delivery result generated based on the first content delivery policy is accessed is longer than the duration of time that the delivery result generated based on the second content delivery policy is accessed in a group of deliveries, it may be determined that the first content delivery policy is superior to the second content delivery policy in the group of deliveries, and if the duration of time that the delivery result generated based on the first content delivery policy is accessed is shorter than the duration of time that the delivery result generated based on the second content delivery policy is accessed in a group of deliveries, it may be determined that the second content delivery policy is superior to the first content delivery policy in the group of deliveries.
On the basis of the above evaluation index score, consider a discrete random variable S, obeying the following distribution:
wherein P (a > B) + P (a < B) + P (a ═ B) ═ 1, and:
therefore, the value of the evaluation index score may be considered as 1/2 of the mean value of S obtained after the discrete random variable S is sampled n times, and further, the problem of determining whether the evaluation index score is significantly non-zero may be converted into a configurable single-sample t test problem, that is, a problem of testing whether the mean value of the random variable S is significantly non-zero. See the following equation:
based on the above equation, and based on the estimation of the variance of the random variable S, a sample-based test statistic Z can be further derivedSThe process is as follows:
based on the above idea, step 13 may include the following steps, as shown in fig. 2.
In step 21, a first number of times that the first content delivery policy is superior to the second content delivery policy and a second number of times that the second content delivery policy is superior to the first content delivery policy are determined according to the sample data.
The policy win may be predefined based on the actual scenario, e.g. set based on the user's click behavior, browsing duration, etc.
For example, if a set of impressions for which the impression results generated based on the first content impression policy were accessed more times than the impression results generated based on the second content impression policy, it may be determined that the first content impression policy is superior to the second content impression policy, and if a set of impressions for which the impression results generated based on the first content impression policy were accessed less times than the impression results generated based on the second content impression policy, it may be determined that the second content impression policy is superior to the first content impression policy.
For another example, if the duration of time that the delivery result generated based on the first content delivery policy is accessed is longer than the duration of time that the delivery result generated based on the second content delivery policy is accessed in a group of deliveries, it may be determined that the first content delivery policy is superior to the second content delivery policy in the group of deliveries, and if the duration of time that the delivery result generated based on the first content delivery policy is accessed is shorter than the duration of time that the delivery result generated based on the second content delivery policy is accessed in a group of deliveries, it may be determined that the second content delivery policy is superior to the first content delivery policy in the group of deliveries.
And based on the setting of the winning mode, combining the obtained sample data to easily determine the first times and the second times.
In step 22, it is determined whether the evaluation index is significantly non-zero based on the first number and the second number.
In one possible embodiment, step 22 may include the steps of:
determining the test statistic of the evaluation index according to the first frequency and the second frequency;
and if the absolute value of the test statistic is larger than the preset quantile, determining that the evaluation index is obviously not zero.
Wherein the test statistic for determining the evaluation index can be calculated using the calculation Z given aboveSThe formula (c) is directly obtained, namely:
after the test statistic is determined, the obtained test statistic may be further compared with a preset quantile to determine whether the evaluation index is significantly non-zero. Illustratively, the predetermined quantile is 1.96 (0.025 quantile of a standard normal distribution).
Therefore, if the absolute value of the test statistic is larger than the preset quantile, the evaluation index can be determined to be obviously not zero.
In step 14, a comparison result between the first content delivery policy and the second content delivery policy is determined according to the evaluation index.
Wherein the comparison result is at least used to indicate whether there is a significant difference between the first content delivery policy and the second content delivery policy.
If the evaluation index is significantly non-zero, it may be determined that the comparison result indicates that there is a significant difference between the first content delivery policy and the second content delivery policy.
Based on the above, if the evaluation index is significantly not zero, the method provided by the present disclosure may further include the following steps:
if the first time number is larger than the second time number, determining that the superiority of the first content delivery strategy is higher than that of the second content delivery strategy;
and if the first times is less than the second times, determining that the superiority of the second content delivery strategy is higher than that of the first content delivery strategy.
If the absolute value of the test statistic is greater than the preset quantile, it can be determined that the first content delivery strategy and the second content delivery strategy have significant differences. Meanwhile, if the first number is greater than the second number, it is indicated that the number of wins of the first content delivery policy is greater than the number of wins of the second content delivery policy, that is, the first content delivery policy is better, and therefore it can be determined that the superiority of the first content delivery policy is higher than that of the second content delivery policy. If the first number is smaller than the second number, it is indicated that the number of wins of the first content delivery policy is not greater than the number of wins of the second content delivery policy, that is, the second content delivery policy is better, and therefore it can be determined that the superiority of the second content delivery policy is higher than that of the first content delivery policy. That is, if there is a significant difference between the first content delivery policy and the second content delivery policy, the comparison result may also be used to indicate which policy is better between the first content delivery policy and the second content delivery policy.
Based on the above, on the basis of determining the superior heights of the first content policy and the second content policy, the method provided by the present disclosure may further include the following steps:
and determining one of the first content delivery strategy and the second content delivery strategy with higher superiority as a target content delivery strategy to be used.
For example, on the basis that a plurality of content policies exist, pairwise comparison may be performed, a policy with a higher superiority is selected as a target content delivery policy, and an AB test is performed subsequently on the determined target content delivery policy.
Through the mode, in other words, the strategy is pre-screened in advance before the AB test, the strategy with higher excellence is screened out through two-two comparison, the subsequent AB test is carried out, the AB test is not required to be opened for each strategy, the time and the resources spent by the AB test are effectively saved, and the AB test efficiency is improved.
In addition, on the basis of the above, the test sensitivity MDE may also be constructed to calculate the sample lifting amount. The MDE is the test sensitivity and is the absolute value of the difference between the direct effect index of the new strategy and the index baseline, namely, the difference between the new strategy and the old strategy is large, and the larger the MDE is, the smaller the required sample size is. Illustratively, the MDE calculation formula used by the present disclosure is as follows:
wherein k is a constant coefficient. Illustratively, k is 2.8.
According to the technical scheme, multiple groups of content delivery are sequentially carried out on a target object, delivery results generated based on a first content delivery strategy and delivery results generated based on a second content delivery strategy are delivered in each group of content delivery in a crossed mode, then, access data corresponding to multiple groups of delivered content are obtained and serve as sample data, evaluation indexes used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy are calculated, and comparison results between the first content delivery strategy and the second content delivery strategy are determined according to the calculated evaluation indexes, wherein the comparison results are at least used for indicating whether the first content delivery strategy and the second content delivery strategy have obvious differences. Therefore, the evaluation index aiming at the content delivery strategy is calculated, the selection of the user aiming at multiple deliveries is aggregated at the same time, so that different content delivery strategies are compared, and the comparison result of the different content delivery strategies can be quickly and accurately obtained based on the high sensitivity of the evaluation index, so that a more excellent strategy can be conveniently screened based on the comparison result in the follow-up process. Therefore, when the strategy used by the AB test is selected, the strategy used for the AB test can be quickly determined, and the efficiency of the AB test is favorably improved.
Fig. 3 is a block diagram of a content delivery policy comparison apparatus provided according to an embodiment of the present disclosure. As shown in fig. 3, the apparatus 30 includes:
the delivery module 31 is configured to sequentially deliver a plurality of groups of content to the target object, where a delivery result generated based on the first content delivery policy and a delivery result generated based on the second content delivery policy are delivered in each group of content delivery in a cross manner;
an obtaining module 32, configured to obtain access data corresponding to multiple sets of released content as sample data;
an index calculation module 33, configured to calculate an evaluation index according to the sample data, where the evaluation index is used to reflect a difference degree between the first content delivery policy and the second content delivery policy;
a first determining module 34, configured to determine, according to the evaluation index, a comparison result between the first content delivery policy and the second content delivery policy, where the comparison result is at least used to indicate whether there is a significant difference between the first content delivery policy and the second content delivery policy.
Optionally, the obtaining module 32 includes:
the receiving submodule is used for receiving a strategy comparison request carrying statistical range information;
and the obtaining submodule is used for responding to the strategy comparison request, and obtaining access data corresponding to the statistical range indicated by the statistical range information from the access data corresponding to each content delivery as the sample data.
Optionally, the index calculating module 33 includes:
a first determining submodule, configured to determine, according to the sample data, a first number of times that the first content delivery policy is superior to the second content delivery policy and a second number of times that the second content delivery policy is superior to the first content delivery policy;
the second determining submodule is used for determining whether the evaluation index is obviously not zero according to the first times and the second times;
the first determining module 34 includes:
and a third determining sub-module, configured to determine that the comparison result indicates that there is a significant difference between the first content delivery policy and the second content delivery policy if the evaluation index is significantly nonzero.
Optionally, if in a group of releases, the number of times that the release result generated based on the first content release policy is accessed is greater than the number of times that the release result generated based on the second content release policy is accessed, determining that the first content release policy is better than the second content release policy in the group of releases; and if the number of times of the accessed releasing results generated based on the first content releasing strategy is less than the number of times of the accessed releasing results generated based on the second content releasing strategy in a group of releasing, determining that the second content releasing strategy is superior to the first content releasing strategy in the group of releasing.
Optionally, the second determining submodule is configured to: determining the test statistic of the evaluation index according to the first times and the second times; and if the absolute value of the test statistic is larger than a preset quantile, determining that the evaluation index is obviously not zero.
Optionally, the third determining sub-module is further configured to: if the evaluation index is not significantly zero, if the first number of times is greater than the second number of times, determining that the superiority of the first content delivery strategy is higher than that of the second content delivery strategy; and if the absolute value of the test statistic is larger than a preset quantile, if the first time is smaller than the second time, determining that the superiority of the second content delivery strategy is higher than that of the first content delivery strategy.
Optionally, the apparatus 30 further comprises:
and the second determining module is used for determining one of the first content delivery strategy and the second content delivery strategy with higher superiority as a target content delivery strategy to be used.
Optionally, the target object comprises at least one user; alternatively, the target object comprises at least one content presentation window.
Alternatively, the evaluation index score is calculated according to the following formula:
wherein n is the total number of content delivery in group unit, and n isAN number of times that the first content delivery policy outperforms the second content delivery policy in n sets of deliveriesBThe number of times that the second content delivery policy outperforms the first content delivery policy in n sets of deliveries.
Optionally, the test statistic of the evaluation index is calculated according to the following formula:
with regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Referring now to FIG. 4, a block diagram of an electronic device 600 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 4, electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 4 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or may be installed from the storage means 608, or may be installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: sequentially carrying out multiple groups of content delivery on the target object, wherein a delivery result generated based on a first content delivery strategy and a delivery result generated based on a second content delivery strategy are delivered in each group of content delivery in a crossed manner; acquiring access data corresponding to a plurality of groups of released contents as sample data; according to the sample data, calculating an evaluation index, wherein the evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy; and determining a comparison result between the first content delivery strategy and the second content delivery strategy according to the evaluation index, wherein the comparison result is at least used for indicating whether the first content delivery strategy and the second content delivery strategy have a significant difference.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented by software or hardware. The name of a module does not in some cases form a limitation on the module itself, and for example, a delivery module may also be described as a "module that sequentially delivers multiple sets of content to a target object".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
According to one or more embodiments of the present disclosure, there is provided a content delivery policy comparison method, including:
sequentially carrying out multiple groups of content delivery on the target object, wherein a delivery result generated based on a first content delivery strategy and a delivery result generated based on a second content delivery strategy are delivered in each group of content delivery in a crossed manner;
acquiring access data corresponding to a plurality of groups of released contents as sample data;
according to the sample data, calculating an evaluation index, wherein the evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy;
and determining a comparison result between the first content delivery strategy and the second content delivery strategy according to the evaluation index, wherein the comparison result is at least used for indicating whether the first content delivery strategy and the second content delivery strategy have a significant difference.
According to one or more embodiments of the present disclosure, there is provided a content delivery policy comparison method, where acquiring access data corresponding to a plurality of sets of delivered content as sample data includes:
receiving a strategy comparison request carrying statistical range information;
and responding to the strategy comparison request, and acquiring access data corresponding to the statistical range indicated by the statistical range information from the access data corresponding to each content delivery as the sample data.
According to one or more embodiments of the present disclosure, a content delivery policy comparison method is provided, where calculating an evaluation index according to the sample data includes:
determining a first number of times that the first content delivery strategy is superior to the second content delivery strategy and a second number of times that the second content delivery strategy is superior to the first content delivery strategy according to the sample data;
determining whether the evaluation index is obviously not zero according to the first times and the second times;
the determining, according to the evaluation index, a comparison result between the first content delivery policy and the second content delivery policy includes:
and if the evaluation index is obviously not zero, determining that the comparison result indicates that the first content delivery strategy and the second content delivery strategy have obvious difference.
According to one or more embodiments of the present disclosure, a content delivery policy comparison method is provided, where if in a group of deliveries, a number of times that a delivery result generated based on a first content delivery policy is accessed is greater than a number of times that a delivery result generated based on a second content delivery policy is accessed, it is determined that the first content delivery policy is superior to the second content delivery policy in the group of deliveries;
and if the number of times of the accessed releasing results generated based on the first content releasing strategy is less than the number of times of the accessed releasing results generated based on the second content releasing strategy in a group of releasing, determining that the second content releasing strategy is superior to the first content releasing strategy in the group of releasing.
According to one or more embodiments of the present disclosure, a content delivery policy comparison method is provided, where determining whether the evaluation index is significantly non-zero according to the first number of times and the second number of times includes:
determining the test statistic of the evaluation index according to the first times and the second times;
and if the absolute value of the test statistic is larger than a preset quantile, determining that the evaluation index is obviously not zero.
According to one or more embodiments of the present disclosure, a content delivery policy comparison method is provided, where if the evaluation index is significantly not zero, the method further includes:
if the first number of times is larger than the second number of times, determining that the superiority of the first content delivery strategy is higher than that of the second content delivery strategy;
and if the first time number is less than the second time number, determining that the superiority of the second content delivery strategy is higher than that of the first content delivery strategy.
According to one or more embodiments of the present disclosure, there is provided a content delivery policy comparison method, the method further including:
and determining one of the first content delivery strategy and the second content delivery strategy with higher superiority as a target content delivery strategy to be used.
According to one or more embodiments of the present disclosure, a content delivery policy comparison method is provided, where the target object includes at least one user; alternatively, the target object comprises at least one content presentation window.
According to one or more embodiments of the present disclosure, a content delivery policy comparison method is provided, in which the evaluation index score is calculated according to the following formula:
wherein n is the total number of content delivery in group unit, and n isAN number of times that the first content delivery policy outperforms the second content delivery policy in n sets of deliveriesBThe number of times that the second content delivery policy outperforms the first content delivery policy in n sets of deliveries.
According to one or more embodiments of the present disclosure, a content delivery policy comparison method is provided, where test statistics of the evaluation index are calculated according to the following formula:
according to one or more embodiments of the present disclosure, there is provided a content delivery policy comparison apparatus, including:
the releasing module is used for sequentially releasing a plurality of groups of contents to the target object, wherein releasing results generated based on the first content releasing strategy and releasing results generated based on the second content releasing strategy are released in each group of contents in a cross mode;
the acquisition module is used for acquiring access data corresponding to a plurality of groups of released contents as sample data;
the index calculation module is used for calculating an evaluation index according to the sample data, wherein the evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy;
a first determining module, configured to determine, according to the evaluation index, a comparison result between the first content delivery policy and the second content delivery policy, where the comparison result is at least used to indicate whether there is a significant difference between the first content delivery policy and the second content delivery policy.
According to one or more embodiments of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, which when executed by a processing device, implements the steps of the content delivery policy comparison method provided by any of the embodiments of the present disclosure.
According to one or more embodiments of the present disclosure, there is provided an electronic device including:
a storage device having a computer program stored thereon;
and the processing device is used for executing the computer program in the storage device to realize the steps of the content delivery strategy comparison method provided by any embodiment of the disclosure.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Claims (13)
1. A content delivery policy comparison method, the method comprising:
sequentially carrying out multiple groups of content delivery on the target object, wherein a delivery result generated based on a first content delivery strategy and a delivery result generated based on a second content delivery strategy are delivered in each group of content delivery in a crossed manner;
acquiring access data corresponding to a plurality of groups of released contents as sample data;
according to the sample data, calculating an evaluation index, wherein the evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy;
and determining a comparison result between the first content delivery strategy and the second content delivery strategy according to the evaluation index, wherein the comparison result is at least used for indicating whether the first content delivery strategy and the second content delivery strategy have a significant difference.
2. The method according to claim 1, wherein the obtaining access data corresponding to the plurality of sets of delivered content as sample data comprises:
receiving a strategy comparison request carrying statistical range information;
and responding to the strategy comparison request, and acquiring access data corresponding to the statistical range indicated by the statistical range information from the access data corresponding to each content delivery as the sample data.
3. The method of claim 1, wherein said calculating an evaluation index from said sample data comprises:
determining a first number of times that the first content delivery strategy is superior to the second content delivery strategy and a second number of times that the second content delivery strategy is superior to the first content delivery strategy according to the sample data;
determining whether the evaluation index is obviously not zero according to the first times and the second times;
the determining, according to the evaluation index, a comparison result between the first content delivery policy and the second content delivery policy includes:
and if the evaluation index is obviously not zero, determining that the comparison result indicates that the first content delivery strategy and the second content delivery strategy have obvious difference.
4. The method of claim 3,
if the number of times of accessing the delivery result generated based on the first content delivery strategy is more than the number of times of accessing the delivery result generated based on the second content delivery strategy in a group of deliveries, determining that the first content delivery strategy is better than the second content delivery strategy in the group of deliveries;
and if the number of times of the accessed releasing results generated based on the first content releasing strategy is less than the number of times of the accessed releasing results generated based on the second content releasing strategy in a group of releasing, determining that the second content releasing strategy is superior to the first content releasing strategy in the group of releasing.
5. The method of claim 3, wherein said determining whether the evaluation metric is significantly non-zero based on the first number of times and the second number of times comprises:
determining the test statistic of the evaluation index according to the first times and the second times;
and if the absolute value of the test statistic is larger than a preset quantile, determining that the evaluation index is obviously not zero.
6. The method of claim 3, wherein if the evaluation metric is significantly non-zero, the method further comprises:
if the first number of times is larger than the second number of times, determining that the superiority of the first content delivery strategy is higher than that of the second content delivery strategy;
and if the first time number is less than the second time number, determining that the superiority of the second content delivery strategy is higher than that of the first content delivery strategy.
7. The method of claim 6, further comprising:
and determining one of the first content delivery strategy and the second content delivery strategy with higher superiority as a target content delivery strategy to be used.
8. The method of any one of claims 1-7, wherein the target object comprises at least one user; alternatively, the target object comprises at least one content presentation window.
9. The method according to any one of claims 1 to 7, wherein the evaluation index score is calculated according to the following formula:
wherein n is the total number of content delivery in group unit, and n isAN number of times that the first content delivery policy outperforms the second content delivery policy in n sets of deliveriesBThe number of times that the second content delivery policy outperforms the first content delivery policy in n sets of deliveries.
10. The method of claim 9, wherein the test statistic of the evaluation index is calculated according to the following formula:
11. a content delivery policy comparison apparatus, the apparatus comprising:
the releasing module is used for sequentially releasing a plurality of groups of contents to the target object, wherein releasing results generated based on the first content releasing strategy and releasing results generated based on the second content releasing strategy are released in each group of contents in a cross mode;
the acquisition module is used for acquiring access data corresponding to a plurality of groups of released contents as sample data;
the index calculation module is used for calculating an evaluation index according to the sample data, wherein the evaluation index is used for reflecting the difference degree between the first content delivery strategy and the second content delivery strategy;
a first determining module, configured to determine, according to the evaluation index, a comparison result between the first content delivery policy and the second content delivery policy, where the comparison result is at least used to indicate whether there is a significant difference between the first content delivery policy and the second content delivery policy.
12. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the steps of the method of any one of claims 1-10.
13. An electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to carry out the steps of the method according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110679121.3A CN113298573A (en) | 2021-06-18 | 2021-06-18 | Content delivery strategy comparison method and device, readable medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110679121.3A CN113298573A (en) | 2021-06-18 | 2021-06-18 | Content delivery strategy comparison method and device, readable medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113298573A true CN113298573A (en) | 2021-08-24 |
Family
ID=77328775
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110679121.3A Pending CN113298573A (en) | 2021-06-18 | 2021-06-18 | Content delivery strategy comparison method and device, readable medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113298573A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115034833A (en) * | 2022-08-10 | 2022-09-09 | 北京达佳互联信息技术有限公司 | Method and device for testing release strategy, electronic equipment and storage medium |
| CN116485262A (en) * | 2022-04-26 | 2023-07-25 | 深圳依时货拉拉科技有限公司 | Evaluation method of pricing strategy, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108845936A (en) * | 2018-05-31 | 2018-11-20 | 阿里巴巴集团控股有限公司 | A kind of AB test method and system based on mass users |
| CN110619548A (en) * | 2019-09-20 | 2019-12-27 | 腾讯科技（深圳）有限公司 | Method, device, equipment and storage medium for determining media content delivery strategy |
| WO2020000689A1 (en) * | 2018-06-29 | 2020-01-02 | 平安科技（深圳）有限公司 | Transfer-learning-based robo-advisor strategy generation method and apparatus, and electronic device and storage medium |
| CN111177561A (en) * | 2019-12-31 | 2020-05-19 | 北京奇艺世纪科技有限公司 | Test method and device |
| CN111669651A (en) * | 2020-05-27 | 2020-09-15 | 北京百度网讯科技有限公司 | Method and device for determining high-quality content, electronic equipment and storage medium |
-
2021
- 2021-06-18 CN CN202110679121.3A patent/CN113298573A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108845936A (en) * | 2018-05-31 | 2018-11-20 | 阿里巴巴集团控股有限公司 | A kind of AB test method and system based on mass users |
| WO2020000689A1 (en) * | 2018-06-29 | 2020-01-02 | 平安科技（深圳）有限公司 | Transfer-learning-based robo-advisor strategy generation method and apparatus, and electronic device and storage medium |
| CN110619548A (en) * | 2019-09-20 | 2019-12-27 | 腾讯科技（深圳）有限公司 | Method, device, equipment and storage medium for determining media content delivery strategy |
| CN111177561A (en) * | 2019-12-31 | 2020-05-19 | 北京奇艺世纪科技有限公司 | Test method and device |
| CN111669651A (en) * | 2020-05-27 | 2020-09-15 | 北京百度网讯科技有限公司 | Method and device for determining high-quality content, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| OLIVIER CHAPELLE: "Large-Scale Validation and Analysis of Interleaved Search Evaluation", ACM TRANSACTIONS ON INFORMATION SYSTEMS, vol. 30, no. 1, pages 2 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116485262A (en) * | 2022-04-26 | 2023-07-25 | 深圳依时货拉拉科技有限公司 | Evaluation method of pricing strategy, electronic equipment and storage medium |
| CN116485262B (en) * | 2022-04-26 | 2024-04-12 | 深圳依时货拉拉科技有限公司 | Evaluation method of pricing strategy, electronic equipment and storage medium |
| CN115034833A (en) * | 2022-08-10 | 2022-09-09 | 北京达佳互联信息技术有限公司 | Method and device for testing release strategy, electronic equipment and storage medium |
| CN115034833B (en) * | 2022-08-10 | 2022-12-27 | 北京达佳互联信息技术有限公司 | Method and device for testing delivery strategy, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111800513B (en) | Method and device for pushing information and computer readable medium of electronic equipment | |
| CN111784712B (en) | Image processing method, device, equipment and computer readable medium | |
| CN112311656B (en) | Message aggregation and display method and device, electronic equipment and computer readable medium | |
| CN111459364B (en) | Icon updating method and device and electronic equipment | |
| CN111400625B (en) | Page processing method and device, electronic equipment and computer readable storage medium | |
| CN111163324B (en) | Information processing method and device and electronic equipment | |
| CN113298573A (en) | Content delivery strategy comparison method and device, readable medium and electronic equipment | |
| CN111596991A (en) | Interactive operation execution method and device and electronic equipment | |
| CN111163336B (en) | Video resource pushing method and device, electronic equipment and computer readable medium | |
| CN112099982A (en) | Collapse information positioning method, device, medium and electronic equipment | |
| CN111262907A (en) | Service instance access method and device and electronic equipment | |
| CN110633411A (en) | Method and device for screening house resources, electronic equipment and storage medium | |
| CN112100211B (en) | Data storage method, apparatus, electronic device, and computer readable medium | |
| CN111628913B (en) | Online time length determining method and device, readable medium and electronic equipment | |
| CN114428925A (en) | Page rendering method and device, electronic equipment and computer readable medium | |
| CN113297277A (en) | Test statistic determination method, device, readable medium and electronic equipment | |
| CN112417276A (en) | Paging data acquisition method and device, electronic equipment and computer readable storage medium | |
| CN112905090A (en) | Spreadsheet processing method, device, terminal and storage medium | |
| CN112464039A (en) | Data display method and device of tree structure, electronic equipment and medium | |
| CN111782549A (en) | Test method and device and electronic equipment | |
| CN111898027A (en) | Method, device, electronic equipment and computer readable medium for determining feature dimension | |
| CN111797354A (en) | Interaction method and device and electronic equipment | |
| CN111625745A (en) | Recommendation method and device, electronic equipment and computer readable medium | |
| CN112685455B (en) | Real-time data classification display method and device, electronic equipment and readable medium | |
| CN112200643B (en) | Article information pushing method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |