CN110377521A

CN110377521A - A kind of target object verification method and device

Info

Publication number: CN110377521A
Application number: CN201910664919.3A
Authority: CN
Inventors: 王喆; 李涛
Original assignee: Zhengzhou Apas Technology Co Ltd
Current assignee: Zhengzhou Apas Technology Co Ltd
Priority date: 2019-07-23
Filing date: 2019-07-23
Publication date: 2019-10-25
Anticipated expiration: 2039-07-23
Also published as: CN110377521B

Abstract

The embodiment of the present application provides a kind of target object verification method and device, utilize empirical Bayes method and the prior probability distribution of each operational indicator of combination, obtain corresponding Posterior probability distribution, it recycles Monte Carlo method and is based on the Posterior probability distribution, obtain the verification result for each target object, simultaneously according to verification result can automatic identification whether meet default verifying termination condition, and then choose for providing the target object used of user on line；The case where for default verifying termination condition is unsatisfactory for, the ratio of the sample of users of each target object is distributed to further according to verification result dynamic adjustment, until verification result meets default verifying termination condition, in this way without being estimated in advance to sample size, it solves the problems, such as to cause verification result inaccurate because sample size has estimated deviation, the verification efficiency of target object is also improved, and the identification of verifying termination condition can be carried out automatically, improves the precision on verifying termination opportunity.

Description

A kind of target object verification method and device

Technical field

This application involves field of computer technology more particularly to a kind of target object verification methods and device.

Background technique

Currently, with the development of computer communication technology, Internet service and being widely used is commented using A/B test to compare Estimate distinct interaction design, the effect of policy logic or algorithm model.

However, selecting optimal side in the prioritization scheme from two or more products using A/B test in existing scheme During case, the method based on the assumption that examining, drawback are generallyd use are as follows: for fixed magnitude hypothesis testing, need Before obtaining sample size (flow), determine that required sample size (flow) is held very much if terminating in advance verifying in verification process Easily obtain false positive conclusion.In addition, carrying out sample in the prioritization scheme to two or more products to determine optimal scheme During this verifying, using based on the assumption that the method for inspection cannot provide most directly most specific conclusion, be very easy to obtain The conclusion of mistake；Moreover, needing to control False discovery rate using additional method, increase is tested in multiple check (grouping is more than 2) The complexity of card process.

It follows that in existing scheme in the prioritization scheme from two or more products optimum scheme comparison process In, need to estimate sample size needed for verification process (flow) in advance, and this is difficult to accurately estimate, it, cannot if over-evaluated Flow can be wasted by stopping verifying (stopping experiment) in advance, if underestimated, verifying, which terminates (i.e. experiment terminates), cannot still be obtained Reliable conclusion, in addition, needing to control False discovery rate using additional method, increasing the complexity of verification process in multiple check Degree.

Summary of the invention

The purpose of the embodiment of the present application is to provide a kind of target object verification method and device, without in advance to sample size It is estimated, solves the problems, such as to cause verification result inaccurate because sample size has estimated deviation, also improve target pair The verification efficiency of elephant, and the identification of verifying termination condition can be carried out automatically, improve the precision on verifying termination opportunity.

In order to solve the above technical problems, the embodiment of the present application is achieved in that

The embodiment of the present application provides a kind of target object verification method, comprising:

For each target object to be verified, the usage behavior for distributing to multiple sample of users of the target object is obtained Data, wherein the ratio of the multiple sample of users is tested based on determined by the last round of verifying for the target object It demonstrate,proves result and carries out dynamic adjustment；

According to the usage behavior data of the multiple sample of users, the characteristic value of each pre-set business index is determined, In, the pre-set business index is determined according to the corresponding application scenarios of the target object；

Using empirical Bayes method according to the prior probability distribution and the characteristic value of each pre-set business index, really The Posterior probability distribution of fixed each pre-set business index；

Using Monte Carlo method according to the Posterior probability distribution, the verification result for being directed to the target object is determined； Wherein, the verification result includes: opportunity values for characterizing target object relative advantage and is used for for characterizing target object The value-at-risk for the potential loss that user uses on line is provided；

If the verification result meets default verifying termination condition, according to the verifying knot of each target object Fruit is chosen in multiple target objects to be verified for providing the target object used of user on line.The embodiment of the present application mentions Supply a kind of target object verifying device, comprising:

Behavioral data obtains module, and for being directed to each target object to be verified, the target object is distributed in acquisition The usage behavior data of multiple sample of users, wherein the ratio of the multiple sample of users is based on for the target object Last round of verifying determined by verification result carry out dynamic adjustment；

Characteristic value determining module determines each default for the usage behavior data according to the multiple sample of users The characteristic value of operational indicator, wherein the pre-set business index is determined according to the corresponding application scenarios of the target object；

Posterior probability distribution determining module, for the elder generation using empirical Bayes method according to each pre-set business index Probability distribution and the characteristic value are tested, determines the Posterior probability distribution of each pre-set business index；

Verification result determining module, for, according to the Posterior probability distribution, determining using Monte Carlo method and being directed to institute State the verification result of target object；Wherein, the verification result include: for characterize the opportunity values of target object relative advantage and For characterizing target object for providing the value-at-risk for the potential loss that user on line uses；

First processing module, if meeting default verifying termination condition for the verification result, according to each target The verification result of object is chosen in multiple target objects to be verified for providing the target pair used of user on line As.

The embodiment of the present application provides a kind of target object verifying equipment, comprising: processor；And

It is arranged to the memory of storage computer executable instructions, the computer executable instructions make when executed The processor realizes following below scheme:

If the verification result meets default verifying termination condition, according to the verifying knot of each target object Fruit is chosen in multiple target objects to be verified for providing the target object used of user on line.The embodiment of the present application mentions A kind of storage medium is supplied, for storing computer executable instructions, the computer executable instructions are realized when executed Following below scheme:

If the verification result meets default verifying termination condition, according to the verifying knot of each target object Fruit is chosen in multiple target objects to be verified for providing the target object used of user on line.In the embodiment of the present application Target object verification method and device, using empirical Bayes method and combine each operational indicator prior probability distribution, obtain To corresponding Posterior probability distribution, recycles Monte Carlo method and be based on the Posterior probability distribution, obtain for each target pair The verification result of elephant, at the same according to verification result can automatic identification whether meet default verifying termination condition, and then choose and use In the target object that user uses on offer line；The case where for default verifying termination condition is unsatisfactory for, further according to verification result The ratio of the sample of users of each target object is distributed in dynamic adjustment, and next based on the sample of users ratio progress redistributed Wheel verifying is solved until verification result meets default verifying termination condition in this way without estimating in advance to sample size Lead to the problem of verification result inaccuracy because sample size estimates deviation, also improves the verification efficiency of target object, and And the identification of verifying termination condition can be carried out automatically, improve the precision on verifying termination opportunity.

Detailed description of the invention

In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this The some embodiments recorded in application, for those of ordinary skill in the art, in the premise of not making the creative labor property Under, it is also possible to obtain other drawings based on these drawings.

Fig. 1 is the first flow diagram of target object verification method provided by the embodiments of the present application；

Fig. 2 is second of flow diagram of target object verification method provided by the embodiments of the present application；

Fig. 3 is the third flow diagram of target object verification method provided by the embodiments of the present application；

Fig. 4 is the 4th kind of flow diagram of target object verification method provided by the embodiments of the present application；

Fig. 5 is the 5th kind of flow diagram of target object verification method provided by the embodiments of the present application；

Fig. 6 is the 6th kind of flow diagram of target object verification method provided by the embodiments of the present application；

Fig. 7 is the module composition schematic diagram that target object provided by the embodiments of the present application verifies device；

Fig. 8 is the structural schematic diagram that target object provided by the embodiments of the present application verifies equipment.

Specific embodiment

In order to make those skilled in the art better understand the technical solutions in the application, below in conjunction with the application reality The attached drawing in example is applied, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described implementation Example is merely a part but not all of the embodiments of the present application.Based on the embodiment in the application, this field is common The application protection all should belong in technical staff's every other embodiment obtained without creative efforts Range.

The embodiment of the present application provides a kind of target object verification method and device, using empirical Bayes method and combines The prior probability distribution of each operational indicator obtains corresponding Posterior probability distribution, recycles Monte Carlo method and based on after this Test probability distribution, obtain the verification result for each target object, at the same according to verification result can automatic identification whether meet Default verifying termination condition, and then choose for providing the target object used of user on line；For being unsatisfactory for, default verifying is whole Only the case where condition, the ratio of the sample of users of each target object is distributed to further according to verification result dynamic adjustment, and based on weight Newly assigned sample of users ratio carries out next round verifying, presets verifying termination condition until verification result meets, is not necessarily in this way Sample size is estimated in advance, solves the problems, such as to cause verification result inaccurate because sample size has estimated deviation, The verification efficiency of target object is also improved, and the identification of verifying termination condition can be carried out automatically, verifying is improved and terminates The precision on opportunity.

Fig. 1 is the first flow diagram for the target object verification method that one embodiment of the application provides, the side in Fig. 1 Method can be executed by server, the specific implementation step of the automatic performance objective banknote validation of server.As shown in Figure 1, this method It at least includes the following steps:

S101 obtains making for the multiple sample of users for distributing to the target object for each target object to be verified With behavioral data, wherein verified for the first run, the ratio of multiple sample of users is determined according to sample of users proportional allocations , it is verified for the non-first run, the ratio of multiple sample of users is tested based on determined by the last round of verifying for target object It demonstrate,proves result and carries out dynamic adjustment, multiple target objects include: the destination application of multiple versions or for target application Multiple information recommendation algorithms of program；

Specifically, for example: by taking the application of mobile terminal individualized video as an example, usage scenario is that user opens application program Afterwards, list of videos is browsed, and clicks interested video-see.There are three versions for the application program, to test these three versions In which version can more effectively improve user using viscosity, need for the Video Applications of these three versions to be used as to be tested Target object, and obtain distribute to each target object multiple sample of users usage behavior data, server be used for by pair The usage behavior data of multiple sample of users of each target object got are analyzed；

Alternatively, for example, usage scenario is that after user opens application, browsing is pushed away by taking the application of mobile terminal individualized video as an example List of videos is recommended, and clicks interested video-see.The current personalized recommendation algorithm of list of videos is A, it is now desired to be surveyed Whether examination two kinds of new proposed algorithm B and C, which can be improved, is recommended quality and user's viscosity, is needed these three proposed algorithms respectively Corresponding Video Applications obtain making for the multiple sample of users for distributing to each target object as target object to be tested With behavioral data, server is used to carry out by the usage behavior data of multiple sample of users to each target object got Analysis；

S102 determines the characteristic value of each pre-set business index according to the usage behavior data of multiple sample of users, wherein Pre-set business index is determined according to the corresponding application scenarios of target object；

Specifically, determining the target object according to the application scenarios of the target object for target object to be verified Specified evaluation index；According to the specified evaluation index of the target object, each pre-set business index of the target object is determined；Its In, for the first run verification process of multiple target objects to be verified, distribute to the ratio of multiple sample of users of each target object Example is by the sample of users proportional allocations of specified quantity, i.e., by sample of users according to each multiple targets pair of pro rate of 1:1 As；Specifically, the sum for mixing the sample with family is determined as distributing to the sample of users of each target object divided by the quantity of target object Quantity；For the non-first run verification process of multiple target objects to be verified, the multiple samples for distributing to each target object are used The ratio at family is that the verification result based on determined by the last round of verifying for target object carries out dynamic adjustment；For default The determination process of the characteristic value of operational indicator distributes to this according to the epicycle verifying got for each target object The usage behavior data of multiple sample of users of target object determine the characteristic value of each pre-set business index, wherein epicycle verifying Refer to first run verifying or the verifying of the non-first run, features described above value is that each pre-set business index institute in the preset time period got is right The usage behavior data count for the sample of users answered；

By taking target object to be verified is the Video Applications of tri- versions of A/B/C as an example, to be answered from the video of these three versions Selected in user be more prone to using version；It is video application for target object to be verified, according to the video Video playing duration per capita is determined as the specified evaluation index of the video application, passed through by the application scenarios of application program Judge corresponding to tri- versions of A/B/C the length of video playing duration per capita, can tentatively judge user be more likely to using The video application of which version in tri- versions of A/B/C, wherein the longer Video Applications version of video playing duration per capita This be user be more prone to using version；The behavior path occurred when using Video Applications by user is browsing video, It clicks video, play video, therefore, according to the specified evaluation index of the Video Applications i.e. video playing duration per capita, will browse Amount, click volume, playing duration are determined as the pre-set business index of the video playing duration per capita；

For example, the sample of users sum for the specified quantity chosen is 996,000, since target object is tri- versions of A/B/C This video application mixes the sample with family sum according to the ratio of 1:1:1 and is equally divided into three groups, i.e. A/ in first run verifying The video application of tri- versions of B/C 332,000 sample of users of each correspondence；According to the epicycle being collected into preset time period The usage behavior data of the sample of users of each version are distributed in verifying, are determined respectively default for what is determined needed for each version The characteristic value of operational indicator, i.e., total pageview, total click volume, total playing duration；For example, the total any active ues amount N of A group_AIt is 332, 000, total pageview K_AIt is 9,860,000, total click volume C_AIt is 1,680,000, total playing duration T_AIt is 26,550,000 seconds；B group Total any active ues amount N_BIt is 332,000, total pageview K_BIt is 9,750,000, total click volume C_BIt is 1,670,000, total playing duration T_BIt is 26,600,000 seconds；The total any active ues amount N of C group_CIt is 332,000, total pageview K_CIt is 9,880,000, total click volume C_CFor 1,685,000, total playing duration T_CIt is 26,520,000 seconds；In the verifying of the non-first run, based on upper one for each target object Wheel verifies identified verification result and determines the ratio for distributing to multiple sample of users of each target object, according to preset time period The usage behavior data of the sample of users of each version are distributed in the epicycle verifying being inside collected into, and determine be directed to each version respectively The characteristic value of the pre-set business index of required determination, i.e., total pageview, total click volume, total playing duration.

S103 is determined using empirical Bayes method according to the prior probability distribution and characteristic value of each pre-set business index The Posterior probability distribution of each pre-set business index；

Specifically, user is answered using video so that target object to be verified is the Video Applications of tri- versions of A/B/C as an example Behavior path is browsing video, clicks video, plays video, obeys Poisson distribution since user browses video behavior, i.e., k_i←Poisson(λ_k), it clicks video behavior and obeys Bernoulli Jacob's distribution, i.e. c_ij← Bernoulli (p) plays video behavior clothes From exponential distribution, i.e. t_ij←Exponential(λ_t), wherein wherein k_iFor the number of videos of user i browsing, c_ijIt is for user i No to click video j, 1 indicates to be clicked, and 0 indicates not clicking on, t_ijIndicate that user i plays the duration of video j, if non-point It hits, is then 0.So per capita video playing when it is a length ofThat is video playing per capita The video tour amount x video click rate x of duration=per capita is averaged playing duration, wherein λ_kFor the parameter of video tour amount per capita, P Parameter, λ for video click rate_tFor the parameter of average playing duration；

According to Bayesian formulaWherein, θ is each pre-set business index of specified evaluation index Parameter (such as λ_k、P、λ_t), S is the usage behavior data (such as k, c, t) of the sample of users got.P (θ) is prior probability, is represented Server is to stochastic variable θ, the i.e. true model of each pre-set business index (pageview, clicking rate, playing duration per capita) before verifying The reasonable estimation enclosed or hypothesis.P (S | θ) it is likelihood probability, i.e., under to the current estimation of θ or hypothesis, observe sample of users Usage behavior data S probability.P (S) is marginal probability, is the constant unrelated with θ.P (θ | S) it is posterior probability, represent base Under to the reasonable estimation of θ or hypothesis, after the usage behavior data S for combining sample of users, the obtained Synthesize estimation to θ.

With λ_kFor,Wherein P (S | λ_k)=P (k), because k obeys Poisson distribution, in order to Convenient for calculating, take the conjugate prior Gamma distribution of Poisson distribution as prior distribution, i.e. λ_k~Gamma (a_k,b_k), it is similar to obtain To p~Beta (a_p,b_p), λ_t~Gamma (a_t,b_t)；

The parameter a of prior probability distribution_k、b_k、a_p、b_p、a_t、b_tIt can be determined according to historical data: recent any active ues amount N₀(comprising repeating), total pageview K₀, total click volume C₀, obtain λ_k~Gamma (K₀,N₀), p~Beta (1+C₀,1+K₀-C₀), λ_t ~Gamma (C₀,T₀)；

Determine that the historical data in certain preset time period obtains any active ues amount N₀It is 1,000,000, total pageview K₀For 30,000,000, total click volume C₀It is 5,000,000, total playing duration T₀It is 80,000,000 second, can determines each default industry The prior probability distribution for index of being engaged in, total pageview K₀Prior probability distribution λ_k~Gamma (30000000,1000000), total point The amount of hitting C₀Prior probability distribution p~Beta (5000001,25000001), total playing duration T₀Prior probability distribution be λ_t~ Gamma(5000000,80000000)。

The Posterior probability distribution for each pre-set business index that X is grouped corresponding to target object is obtained using conjugate relation: Obtain total pageview K₀, total click volume C₀And total playing duration T₀Posterior probability distribution:p^x~Beta (a_p+C_x,b_p+K_x-C_x),

According to the characteristic value of pre-set business index corresponding to each version determined: total pageview, total click volume, total Playing duration；For example, for the process of first run verifying, if it is determined that go out the total any active ues amount N of A group_AIt is 332,000, it is total to browse Measure K_AIt is 9,860,000, total click volume C_AIt is 1,680,000, total playing duration T_AIt is 26,550,000 seconds；The total any active ues of B group Measure N_BIt is 332,000, total pageview K_BIt is 9,750,000, total click volume C_BIt is 1,670,000, total playing duration T_BIt is 26, 600,000 seconds；The total any active ues amount N of C group_CIt is 332,000, total pageview K_CIt is 9,880,000, total click volume C_CIt is 1,685, 000, total playing duration T_CIt is 26,520,000 seconds；And above-mentioned total pageview K₀, total click volume C₀And total playing duration T₀'s Prior probability distribution obtains each pre-set business index corresponding to A version: i.e. total pageview K₀, total click volume C₀And total broadcasting Duration T₀Posterior probability distribution: be respectivelyp^A~Beta (6680001, 33180001),Each pre-set business index corresponding to B version: i.e. total clear The amount of looking at K₀, total click volume C₀And total playing duration T₀Posterior probability distribution: be respectively p^B~Beta (6670001,33080001),Each pre-set business index corresponding to C version: i.e. total pageview K₀, total point The amount of hitting C₀And total playing duration T₀Posterior probability distribution: be respectively p^C~ Beta (6685001,33195001),Similarly, for the verifying of the non-first run Process determines each pre- according to the characteristic value and prior probability distribution of pre-set business index corresponding to each version determined If the Posterior probability distribution of operational indicator；

S104 determines the verification result for being directed to target object using Monte Carlo method according to Posterior probability distribution；Its In, the verification result includes: opportunity values for characterizing target object relative advantage and for characterizing target object for mentioning For the value-at-risk for the potential loss that user on line uses；Specifically, each pre-set business index is directed to, to the pre-set business index Posterior probability distribution carry out random sampling processing, obtain the randomly sampled data of pre-set business index；According to the pre-set business The randomly sampled data of index determines the posterior probability sampled result of the specified evaluation index of target object, utilizes Monte Carlo Method determines the verification result for being directed to target object according to posterior probability sampled result.

S105, if verification result meets default verifying termination condition, according to the verification result of each target object, to be tested It chooses in multiple target objects of card for providing the target object used of user on line, wherein above-mentioned verification result meets pre- If verifying termination condition refers to that the verification result of at least one target object meets default verifying termination condition.

Wherein, above-mentioned target object verification method is the online randomised controlled trials point for being directed to multiple target experimental subjects Analysis method obtains corresponding posterior probability point using empirical Bayes method and the prior probability distribution of each operational indicator of combination Cloth recycles Monte Carlo method and is based on the Posterior probability distribution, obtains the analysis for each target experimental subjects as a result, same When can judge automatically whether experiment meets preset termination condition based on the analysis results, and then choose and make for providing user on line Target experimental subjects；The case where for default verifying termination condition is unsatisfactory for, distributes further according to analysis result dynamic adjustment Next round verifying is carried out to the ratio of the sample of users of each target object, and based on the sample of users ratio redistributed, until Analysis result meets preset termination condition, in this way can be sufficiently based on the usage behavior data of the sample of users acquired in real time to more The superiority and inferiority of a target experimental subjects carries out experimental analysis, is analyzed accordingly as a result, reducing to sample size needed for testing Requirement is estimated, judges automatically whether experiment meets preset termination condition based on the analysis results, further improves conventional efficient, is solved The low problem of conventional efficient of having determined；

In the embodiment of the present application, using empirical Bayes method and the prior probability distribution of each operational indicator of combination, obtain Corresponding Posterior probability distribution recycles Monte Carlo method and is based on the Posterior probability distribution, obtains for each target object Verification result, while according to verification result can automatic identification whether meet default verifying termination condition, and then choose and be used for The target object that user uses on line is provided；It is the case where for default verifying termination condition is unsatisfactory for, dynamic further according to verification result The ratio of the sample of users of each target object is distributed in state adjustment, and carries out next round based on the sample of users ratio redistributed Verifying is not necessarily in advance estimate sample size in this way until verification result meets default verifying termination condition, solve because Sample size, which estimates deviation i.e., can not accurately estimate the problem tested required sample size and lead to verification result inaccuracy, also The verification efficiency of target object is improved, and the identification of verifying termination condition can be carried out automatically, when improving verifying termination The precision of machine.

Wherein, during to the optimal verifying of multiple target objects progress using effect, it may be necessary to be used in conjunction with sample The usage behavior data at family carry out more wheel verifyings, as shown in Fig. 2, the above method further include:

S106, if verification result is unsatisfactory for default verifying termination condition and is determined according to the verification result of each target object Carry out the ratio for the sample of users that each target object is distributed to when next round verifying for multiple target objects to be verified, and after It is continuous the step of executing the usage behavior data for obtaining the multiple sample of users for distributing to the target object, pre- until determining to meet If verifying termination condition, wherein above-mentioned verification result is unsatisfactory for the verifying knot that default verifying termination condition refers to each target object Default verifying termination condition is not satisfied in fruit.

Specifically, during to the optimal verifying of multiple target objects progress using effect, it may be necessary in conjunction with sample The usage behavior data of user carry out more wheel verifyings, until verification result meets default verifying termination condition, just stop being directed to More wheel verification process of multiple target objects of verifying；Wherein, above-mentioned steps S101 is performed both by step for each round verifying S104 obtains epicycle and verifies corresponding verification result, if verification result meets default verifying termination condition, executes S105, root According to the verification result of each target object, choose in multiple target objects to be verified for providing the target used of user on line Object；If verification result is unsatisfactory for default verifying termination condition, S106 is executed, according to the verification result of each target object, really Surely the ratio for the sample of users that each target object is distributed to when next round verifying is carried out for multiple target objects to be verified, and The step of continuing to execute the usage behavior data for obtaining the multiple sample of users for distributing to the target object, until determining to meet Default verifying termination condition.

For the first run verification process of multiple target objects to be verified, the ratio of the sample of users of each target object is distributed to Example is respectively to be determined the sample of users of specified quantity, i.e., by sample of users according to each multiple mesh of pro rate of 1:1 Object is marked, specifically, the sum for mixing the sample with family is determined as distributing to the sample of each target object divided by the quantity of target object The quantity of user；Subsequent next round verifying (the i.e. non-first run is verified) process for multiple target objects to be verified, according to upper The ratio of the sample of users of each target object is distributed in the verification result of one wheel, dynamic adjustment.

Wherein, it as shown in figure 3, in above-mentioned S104, using Monte Carlo method according to Posterior probability distribution, determines and is directed to mesh After the verification result for marking object, further includes:

S107 judges whether to meet default verifying termination condition according to the verification result of each target object；

If the determination result is YES, then S105 is executed, according to the verification result of each target object, in multiple targets to be verified It chooses in object for providing the target object used of user on line；Wherein, the judging result be the presence of opportunity values and wind Danger value is all satisfied at least one target object of preset threshold condition；

If judging result be it is no, execute S106, according to the verification result of each target object, determine for be verified more A target object carries out the ratio that the sample of users of each target object is distributed to when next round verifying, and continues to execute acquisition distribution To multiple sample of users of the target object usage behavior data the step of, terminate item until determining to meet default verifying Part；Wherein, it is that there is no at least one targets pair that opportunity values and value-at-risk meet preset threshold condition which, which is no, As.

Wherein, in order to improve verification efficiency, sample of users is made to concentrate on the biggish target object of opportunity values, so that opportunity values Biggish target object faster distinguishes superiority and inferiority, and above-mentioned S106 is determined according to the verification result of each target object for be verified Multiple target objects carry out next round verifying when distribute to each target object sample of users ratio, comprising:

According to the size relation of the opportunity values of each target object, determine next for multiple target objects progress to be verified The ratio of the sample of users of each target object is distributed to when wheel verifying；

Wherein, the opportunity values of target object are directly proportional to the ratio for the sample of users for distributing to the target object.

Specifically, by taking target object to be verified is the Video Applications of tri- versions of A/B/C as an example, however, it is determined that the A version gone out Video Applications verification result in opportunity values be 40%, B version Video Applications verification result in opportunity values be 30%, C Opportunity values are 30% in the verification result of the Video Applications of version, and the video of tri- versions of A/B/C is distributed in first run verifying The ratio of the sample of users amount of application is 1:1:1, and the sample of users amount for distributing to the Video Applications of tri- versions of A/B/C is 332,000；It is corresponding, in the non-first run verifying of the Video Applications for tri- versions of A/B/C, then answered according to each version video Opportunity values size will distribute to the ratio of the sample of users amount of the Video Applications of tri- versions of A/B/C, be adjusted to corresponding Opportunity values ratio, i.e. 4:3:3, obtain next round verifying when distribute to tri- versions of A/B/C Video Applications sample use Family, which is measured, is respectively as follows: 398,000,298, and 000,298,000；In this way, the low Video Applications version of opportunity values will assign to less sample This user makes sample of users concentrate on the biggish Video Applications version of opportunity values, can make the biggish Video Applications of opportunity values Version faster distinguishes superiority and inferiority, improves verification process sample of users utilization efficiency.

Wherein, as shown in figure 4, above-mentioned S104 is determined using Monte Carlo method according to Posterior probability distribution and is directed to target The verification result of object, comprising:

S1041 carries out random sampling to the Posterior probability distribution of the pre-set business index for each pre-set business index Processing, obtains the randomly sampled data of pre-set business index；

Specifically, for example, setting the size of Monte Carlo sampling as M, with target object to be verified for tri- versions of A/B/C Video Applications for, for every pre-set business index of the Video Applications of A version: i.e. total pageview K₀, total click volume C₀With And total playing duration T₀Posterior probability distribution: be respectivelyp^A~Beta (6680001,33180001), Therefrom extracting M random number respectively can obtain To vector Wherein, R is real number, and M is the number of the random number extracted, R^MFor length For the vector of M；

Similarly, general from the posteriority of every pre-set business index for every pre-set business index of the Video Applications of B version Rate distributionp^B~Beta (6670001,33080001),Therefrom extracting M random number respectively can be obtained vectorWherein, R is real number, and M is the number of the random number extracted, R^MIt is M's for length Vector；

Similarly, general from the posteriority of every pre-set business index for every pre-set business index of the Video Applications of C version Rate distributionp^C~Beta (6685001,33195001),Therefrom extracting M random number respectively can be obtained vectorWherein, R is real number, and M is the number of the random number extracted, R^MIt is M's for length Vector；

S1042 determines target object using Monte Carlo method according to the randomly sampled data of each pre-set business index The posterior probability sampled result of specified evaluation index；

Specifically, by taking target object to be verified is the Video Applications of tri- versions of A/B/C as an example, according to video playing per capita Shi ChangweiAnd vector It determinesAvailable video playing duration d per capita^xM Posterior probability distribution sample D^x；

Determine D^xThe available E (d) of mean value, determine D^xThe 2.5th percentile and the 97.5th percentile of middle M sampling Value, the available duration of video playing per capita d^x95% credibility interval；Such as:

Take random sampling number M=1,000,000, it can determine the duration E of the video playing per capita (d of A version^A) be 79.99 seconds, 95% credibility interval was 79.91~80.08 seconds.Similar, it can determine the video playing duration per capita of B version E(d^B) it is 80.03 seconds, 95% credibility interval is 79.94~80.12 seconds, the duration E of the video playing per capita (d of C version^C) be 79.97 seconds, 95% credibility interval was 79.88~80.06 seconds.

S1043 determines the verifying knot for being directed to target object using Monte Carlo method according to posterior probability sampled result Fruit；

Wherein, by taking target object to be verified is the Video Applications of tri- versions of A/B/C as an example, tri- versions of A/B/C to be compared This specified evaluation index i.e. video playing duration per capita, it is thus necessary to determine that video playing duration is excellent per capita corresponding to each version In the probability of other versions video playing duration per capita, that is, determine that tri- versions of A/B/C are the probability of optimal version respectively, referred to as The opportunity values of each version；

Specifically, determining finger corresponding to the target object using Monte Carlo method according to posterior probability sampled result Determine probability value of the evaluation index better than the specified evaluation index of other target objects, and the probability value is determined as the target object Opportunity values；

Wherein, within a preset period of time, the behavioral data of the sample of users obtained with server is increasing, really Optimal objective object may be inconsistent with current judgement, therefore at any one time, if it is optimal for selecting some target object Target object terminates to verify, and can all have the risk of misjudgment, in order to ensure server can be determined according to verification result The risk of optimal objective object is controllable, it is necessary to quantify to determine the potential loss that optimal objective object terminates verifying, referred to as target pair The value-at-risk of elephant；

Specifically, determining that the target object is optimal objective using Monte Carlo method according to posterior probability sampled result The process of the value-at-risk of object are as follows: it is directed to each target object, it is general according to the posteriority of the target object using Monte Carlo method Rate sampled result determines the mathematics of the difference of the value of the specified evaluation index of the target object and potential optimal target object It is expected that；Mathematic expectaion is determined as the value-at-risk for target object.

Specifically, as shown in figure 5, above-mentioned S1043 is determined using Monte Carlo method according to posterior probability sampled result For the verification result of target object, comprising:

S10431 is sampled using Monte Carlo method according to the posterior probability of the target object for each target object As a result, determining that the specified evaluation index of the target object is better than the probability value of other target objects；Probability value is determined as being directed to The opportunity values of target object；

Specifically, being sampled according to Posterior probability distribution, by Monte Carlo method, determineObtain the probability that target object is better than other target objects, gained probability For the opportunity values of target object to be measured, whereinFor indicator function, 1 is taken when x is true, when x is that fictitious time takes 0, d^a It is target object a to be measured better than object to be measured object 1,2 ..., the probability of n, M is the preset quantity of the random number extracted,For The Posterior probability distribution of the target of serial number i object a to be measured samples, and n is the mark of target object；

S10432 is sampled using Monte Carlo method according to the posterior probability of the target object for each target object As a result, determining the mathematic expectaion of the difference of the value of the specified evaluation index of the target object and potential optimal target object； Mathematic expectaion is determined as to the value-at-risk for target object, wherein potential optimal target object refers to be verified multiple The maximum target object of value of the specified evaluation index of target object；

Specifically, being sampled according to Posterior probability distribution, by Monte Carlo method, determineObtain the value-at-risk of target object a to be measured, wherein d^bFor target The value-at-risk of object a to be measured, M are the preset quantity of the random number extracted,For the posteriority of the target object a to be measured of serial number i Probability distribution sampling, 1,2, n be target object mark；

Such as: as i=1,It is 20,It is 15,It is 18, determines respectivelyValue be 5, withValue be 3, take value of the difference maximum difference 5 as this max function, i.e. max1=5, wherein B version is potential optimal target object in secondary determination process；

As i=2,It is 15,It is 10,It is 11, determines respectivelyValue be -5, with's Value is -4, it follows that the value of A group is greater than B group, C group in this determination process, can determine and select A group as optimal at this time The no risk of group, therefore, max2=0 at this time；

As i=M, it is assumed that M 100,It is 30,It is 40,It is 50, determines respectivelyValue be 10, withValue be 20, take value of the maximum difference 20 of difference as this max function, at this time max100= 20, wherein C version is potential optimal target object in this determination process；

The value for+the max100 that determines max1+ ... is MAX, then obtains the value-at-risk of A group divided by 100 with MAX；

Using above-mentioned example (5+0+ ... .+20)/100, obtained result is exactly the value-at-risk of A group.

Wherein, as shown in fig. 6, above-mentioned S107 judges whether to meet default verifying according to the verification result of each target object Termination condition, comprising:

S1071, judges whether there is opportunity values and value-at-risk is all satisfied at least one target object of preset threshold condition, Wherein, the preset threshold condition is that the opportunity values are greater than the first preset threshold and the value-at-risk less than the second default threshold Value；

At least one target object for meeting preset threshold condition if it exists, then execute S1072, determines that verification result meets Default verifying termination condition；

Specifically, by taking target object to be verified is the Video Applications of tri- versions of A/B/C as an example, if preset threshold condition is The opportunity values of target object are greater than 40%, and value-at-risk is less than 0.1；The opportunity values of the Video Applications of A version are 45%, value-at-risk Opportunity values for the Video Applications of 0.0509, B version are 41%, value-at-risk 0.0134, the opportunity values of the Video Applications of C version It is 14%, value-at-risk 0.0735；Since there are two targets pair that opportunity values and value-at-risk are all satisfied preset threshold condition As i.e. Video Applications of the Video Applications of A version and B version, accordingly, it is determined that verification result meets default verifying termination condition；

Corresponding, above-mentioned S105 chooses in multiple target objects to be verified according to the verification result of each target object For providing the target object used of user on line, comprising:

S1051, it is at least one target object for meeting preset threshold condition, the maximum target object of opportunity values is true It is set to for providing the target object used of user on line.

Specifically, since the opportunity values of the Video Applications of above-mentioned A version are 45%, greater than the machine of the Video Applications of B version Can value 41% therefore the Video Applications of A version are determined as to be used to provide the target object that user uses on line；

At least one target object for meeting preset threshold condition if it does not exist, then execute S1073, determines verification result not Meet default verifying termination condition, and, S106 is executed, according to the verification result of each target object, is determined for be verified Multiple target objects carry out the ratio that the sample of users of each target object is distributed to when next round verifying, and continue to execute acquisition point The step of usage behavior data of multiple sample of users of the dispensing target object, until determining that meeting default verifying terminates item Part；

Wherein, it in order to guarantee that server determines for providing the accuracy for the target object that user on line uses, keeps away Exempting to occur the sample of users of server acquisition, there are morning and evening behavioral difference, the caused sample of users rows got in one day The usage behavior data for distributing to multiple sample of users of the target object are obtained in above-mentioned S101 for the inaccuracy of data Before, further includes:

Step 1, for each target object to be verified, judgement carries out the use row of sample of users for epicycle verifying Whether it is greater than preset time threshold for the acquisition time of data；

Specifically, server obtain distribute to the target object multiple sample of users usage behavior data before, It is necessary to ensure that the usage behavior data for receiving certain sample of users for each target object, needs to guarantee for epicycle The acquisition time that verifying carries out the usage behavior data of sample of users is greater than preset time threshold, such as: preset time threshold can Think 24 hours；The purpose for the arrangement is that avoiding since user's usage behavior is there are time segment difference is different, and cause the verifying obtained As a result inaccurate problem；Such as: since user's morning and evening behavior has differences, such as: some users tend to browsing in morning newly It hears, tends to browse cuisines video at night；Or there are active user (such as the elderly) in morning and users active at night (such as young people)；It, can if the time of experiment is too short by taking the proposed algorithm for tri- version Video Applications of A/B/C as an example Can there can be the feelings that the sample of users behavioral data got for different types of video and different sample of users differs greatly Condition, therefore, if the time of experiment is too short, the superiority and inferiority knot of the proposed algorithm of finally obtained tri- version Video Applications of A/B/C Fruit may have very big error；

Therefore, after the acquisition time for determining usage behavior data is greater than preset time threshold, explanation is collected at this time Sample of users usage behavior data it is more comprehensive, the verification result of each target object is just determined for epicycle verifying, otherwise The acquisition for carrying out the usage behavior data of sample of users is verified for epicycle, until acquisition time is greater than preset time threshold；

Step 2 if the determination result is YES then executes the use for obtaining the multiple sample of users for distributing to the target object The step of behavioral data；

Step 3, if judging result be it is no, the usage behavior data for carrying out sample of users are verified continuing with epicycle Acquisition, until acquisition time is greater than preset time threshold, execution obtains making for the multiple sample of users for distributing to the target object The step of with behavioral data.

Specifically, above-mentioned multiple target objects include: the destination application of multiple versions or answer for target With multiple information recommendation algorithms of program；

Pre-set business index includes: the click parameter of multimedia messages, browsing parameter, at least one of play parameter, In, which includes: the webpage shown in application program, video, picture, article etc..

Target object verification method in the embodiment of the present application using empirical Bayes method and combines each operational indicator Prior probability distribution obtains corresponding Posterior probability distribution, recycles Monte Carlo method and is based on the Posterior probability distribution, obtains To the verification result for being directed to each target object, at the same according to verification result can automatic identification whether meet default verifying and terminate item Part, and then choose for providing the target object used of user on line；The case where for default verifying termination condition is unsatisfactory for, then The ratio of the sample of users of each target object is distributed to according to verification result dynamic adjustment, and based on the sample of users redistributed Ratio carries out next round verifying, until verification result meets default verifying termination condition, be not necessarily to so in advance to sample size into Row is estimated, and is solved the problems, such as to cause verification result inaccurate because sample size has estimated deviation, is also improved target object Verification efficiency, and can carry out automatically verifying termination condition identification, improve verifying termination opportunity precision.

The target object verification method that corresponding above-mentioned Fig. 1 to Fig. 6 is described, based on the same technical idea, the application is implemented Example additionally provides a kind of target object verifying device, and Fig. 7 is the module that target object provided by the embodiments of the present application verifies device Composition schematic diagram, the device is for executing the target object verification method that Fig. 1 to Fig. 6 is described, as shown in fig. 7, the device includes:

Behavioral data obtains module 701, and for being directed to each target object to be verified, the target object is distributed in acquisition Multiple sample of users usage behavior data, wherein the ratio of the multiple sample of users is based on for the target pair Verification result determined by the last round of verifying of elephant carries out dynamic adjustment；

Characteristic value determining module 702 determines each pre- for the usage behavior data according to the multiple sample of users If the characteristic value of operational indicator, wherein the pre-set business index is determined according to the corresponding application scenarios of the target object 's；

Posterior probability distribution determining module 703, for utilizing empirical Bayes method according to each pre-set business index Prior probability distribution and the characteristic value, determine the Posterior probability distribution of each pre-set business index；

Verification result determining module 704, for, according to the Posterior probability distribution, determination to be directed to using Monte Carlo method The verification result of the target object；Wherein, the verification result includes: the opportunity values for characterizing target object relative advantage It is used to provide the value-at-risk for the potential loss that user on line uses with for characterizing target object；

First processing module 705, if meeting default verifying termination condition for the verification result, according to each mesh The verification result for marking object is chosen in multiple target objects to be verified for providing the target pair used of user on line As.

Target object provided by the embodiments of the present application verifies device, using empirical Bayes method and combines each operational indicator Prior probability distribution, obtain corresponding Posterior probability distribution, Monte Carlo method recycled simultaneously to be based on the Posterior probability distribution, Obtain the verification result for each target object, at the same according to verification result can automatic identification whether meet default verifying and terminate Condition, and then choose for providing the target object used of user on line；The case where for default verifying termination condition is unsatisfactory for, The ratio of the sample of users of each target object is distributed to further according to verification result dynamic adjustment, and is used based on the sample redistributed Family ratio carries out next round verifying, until verification result meets default verifying termination condition, in this way without in advance to sample size It is estimated, solves the problems, such as to cause verification result inaccurate because sample size has estimated deviation, also improve target pair The verification efficiency of elephant, and the identification of verifying termination condition can be carried out automatically, improve the precision on verifying termination opportunity.

Optionally, above-mentioned apparatus further includes Second processing module, is used for:

If verification result is unsatisfactory for default verifying termination condition, according to the verification result of each target object, It determines and carries out the sample of users for distributing to each target object when next round verifying for multiple target objects to be verified Ratio, and the step of continuing to execute the usage behavior data for obtaining the multiple sample of users for distributing to the target object, until true Make the default verifying termination condition of satisfaction.

Optionally, the Second processing module, is specifically used for:

According to the size relation of the opportunity values of each target object, determines and be directed to multiple target objects to be verified Carry out the ratio that the sample of users of each target object is distributed to when next round verifying；

Optionally, the verification result determining module 704, is specifically used for:

For each pre-set business index, the Posterior probability distribution of the pre-set business index is taken out at random Sample processing, obtains the randomly sampled data of the pre-set business index；

Using Monte Carlo method according to the randomly sampled data of each pre-set business index, the target is determined The posterior probability sampled result of the specified evaluation index of object；

Using Monte Carlo method according to the posterior probability sampled result, the verifying knot for being directed to the target object is determined Fruit.

The verification result determining module 704, is further specifically used for:

For each target object, sampled using Monte Carlo method according to the posterior probability of the target object As a result, determining that the specified evaluation index of the target object is better than the probability value of other target objects；The probability value is true It is set to the opportunity values for the target object；

For each target object, sampled using Monte Carlo method according to the posterior probability of the target object As a result, determining the mathematics phase of the difference of the value of the specified evaluation index of the target object and potential optimal target object It hopes；The mathematic expectaion is determined as to the value-at-risk for the target object, wherein the potential optimal target object is Refer to the maximum target object of value of the specified evaluation index of multiple target objects to be verified.

Optionally, described device further includes termination condition judgment module, is used for:

It judges whether there is the opportunity values and the value-at-risk is all satisfied at least one target pair of preset threshold condition As, wherein the preset threshold condition is that the opportunity values are greater than the first preset threshold and the value-at-risk is default less than second Threshold value；

Meet at least one target object of preset threshold condition if it exists, it is determined that the verification result meets default test Demonstrate,prove termination condition；

Corresponding, the first processing module 705 is specifically used for:

In at least one target object for meeting preset threshold condition, the maximum target object of opportunity values is determined as using In the target object that user uses on offer line.

Optionally, above-mentioned apparatus further includes acquisition parameter judgment module, is used for:

For each target object to be verified, the usage behavior data for carrying out sample of users are verified in judgement for epicycle Whether acquisition time is greater than preset time threshold；

If the determination result is YES, then the usage behavior data for obtaining the multiple sample of users for distributing to the target object are executed The step of；

If judging result be it is no, continuing with epicycle verify carry out sample of users usage behavior data acquisition, directly It is greater than preset time threshold to acquisition time, executes the usage behavior number for obtaining the multiple sample of users for distributing to the target object According to the step of.

Optionally, multiple target objects include: the destination application of multiple versions or for target application journey Multiple information recommendation algorithms of sequence；

The pre-set business index includes: the click parameter of multimedia messages, browsing parameter, at least one in play parameter ?.

Target object in the embodiment of the present application verifies device, using empirical Bayes method and combines each operational indicator Prior probability distribution obtains corresponding Posterior probability distribution, recycles Monte Carlo method and is based on the Posterior probability distribution, obtains To the verification result for being directed to each target object, at the same according to verification result can automatic identification whether meet default verifying and terminate item Part, and then choose for providing the target object used of user on line；The case where for default verifying termination condition is unsatisfactory for, then The ratio of the sample of users of each target object is distributed to according to verification result dynamic adjustment, and based on the sample of users redistributed Ratio carries out next round verifying, until verification result meets default verifying termination condition, be not necessarily to so in advance to sample size into Row is estimated, and is solved the problems, such as to cause verification result inaccurate because sample size has estimated deviation, is also improved target object Verification efficiency, and can carry out automatically verifying termination condition identification, improve verifying termination opportunity precision.

It should be noted that target object verifying device provided by the embodiments of the present application and mesh provided by the embodiments of the present application Mark banknote validation method based on the same inventive concept, therefore the specific implementation of the embodiment may refer to preceding aim banknote validation The implementation of method, overlaps will not be repeated.

Further, corresponding above-mentioned Fig. 1 is to method shown in fig. 6, and based on the same technical idea, the embodiment of the present application is also A kind of target object verifying equipment is provided, which is that the application is real for executing above-mentioned target object verification method, Fig. 8 The structural schematic diagram of the target object verifying equipment of example offer is provided.

As shown in figure 8, target object verifying equipment can generate bigger difference because configuration or performance are different, can wrap One or more processor 801 and memory 802 are included, one or more has been can store in memory 802 and has deposited Store up application program or data.Wherein, memory 802 can be of short duration storage or persistent storage.It is stored in the application of memory 802 Program may include one or more modules (diagram is not shown), and each module may include verifying equipment to target object In series of computation machine executable instruction.Further, processor 801 can be set to communicate with memory 802, in mesh Mark the series of computation machine executable instruction executed in memory 802 in banknote validation equipment.Target object verifying equipment may be used also To include one or more power supplys 803, one or more wired or wireless network interfaces 804, one or one with Upper input/output interface 805, one or more keyboards 806 etc..

In a specific embodiment, target object verifying equipment include memory and one or more Program, perhaps more than one program is stored in memory and one or more than one program may include for one of them One or more modules, and each module may include executable to the series of computation machine in target object verifying equipment Instruction, and be configured to execute this or more than one program by one or more than one processor to include for carrying out Following computer executable instructions:

If the verification result meets default verifying termination condition, according to the verifying knot of each target object Fruit is chosen in multiple target objects to be verified for providing the target object used of user on line.

Optionally, computer executable instructions also include for carrying out following computer executable instructions when executed:

If the verification result is unsatisfactory for default verifying termination condition, according to the verifying knot of each target object Fruit determines the sample of users for carrying out distributing to each target object when next round verifying for multiple target objects to be verified Ratio, and continue to execute obtain distribute to the target object multiple sample of users usage behavior data the step of, until It determines to meet default verifying termination condition.

Optionally, computer executable instructions when executed, the verifying knot according to each target object Fruit determines the sample of users for carrying out distributing to each target object when next round verifying for multiple target objects to be verified Ratio, comprising:

Optionally, computer executable instructions are when executed, described general according to the posteriority using Monte Carlo method Rate distribution, determines the verification result for being directed to the target object, comprising:

Optionally, computer executable instructions are when executed, described general according to the posteriority using Monte Carlo method Rate sampled result determines the verification result for being directed to the target object, comprising:

Optionally, computer executable instructions also include for carrying out following computer executable instructions when executed: It, according to the Posterior probability distribution, after determining the verification result for the target object, is gone back using Monte Carlo method Include:

It is corresponding, the verification result according to each target object, in multiple target objects to be verified It chooses for providing the target object used of user on line, comprising:

Optionally, computer executable instructions also include for carrying out following computer executable instructions when executed: Before the usage behavior data for obtaining the multiple sample of users for distributing to the target object, further includes:

Optionally, when executed, multiple target objects include: the target of multiple versions to computer executable instructions Application program or multiple information recommendation algorithms for destination application；

Target object in the embodiment of the present application verifies equipment, using empirical Bayes method and combines each operational indicator Prior probability distribution obtains corresponding Posterior probability distribution, recycles Monte Carlo method and is based on the Posterior probability distribution, obtains To the verification result for being directed to each target object, at the same according to verification result can automatic identification whether meet default verifying and terminate item Part, and then choose for providing the target object used of user on line；The case where for default verifying termination condition is unsatisfactory for, then The ratio of the sample of users of each target object is distributed to according to verification result dynamic adjustment, and based on the sample of users redistributed Ratio carries out next round verifying, until verification result meets default verifying termination condition, be not necessarily to so in advance to sample size into Row is estimated, and is solved the problems, such as to cause verification result inaccurate because sample size has estimated deviation, is also improved target object Verification efficiency, and can carry out automatically verifying termination condition identification, improve verifying termination opportunity precision.

Preferably, the embodiment of the present application also provides a kind of target object verifying equipment, including processor 801, memory 802, it is stored in the computer program that can be run on memory 802 and on processor 801, the computer program is by processor 801 realize each process of above-mentioned target object verification method embodiment when executing, and can reach identical technical effect, to keep away Exempt to repeat, which is not described herein again.

Further, corresponding above-mentioned Fig. 1 is to method shown in fig. 6, and based on the same technical idea, the embodiment of the present application is also A kind of computer readable storage medium is provided, is stored with computer program on computer readable storage medium, the computer journey Each process of above-mentioned target object verification method embodiment is realized when sequence is executed by processor, and can reach identical technology effect Fruit, to avoid repeating, which is not described herein again.Wherein, the computer readable storage medium, such as read-only memory (Read- Only Memory, abbreviation ROM), random access memory (Random Access Memory, abbreviation RAM), magnetic disk or light Disk etc..

All the embodiments in this specification are described in a progressive manner, same and similar portion between each embodiment Dividing may refer to each other, and each embodiment focuses on the differences from other embodiments.Especially for system reality For applying example, since it is substantially similar to the method embodiment, so being described relatively simple, related place is referring to embodiment of the method Part explanation.

The above description is only an example of the present application, is not intended to limit this application.For those skilled in the art For, various changes and changes are possible in this application.All any modifications made within the spirit and principles of the present application are equal Replacement, improvement etc., should be included within the scope of the claims of this application.

Claims

1. a kind of target object verification method characterized by comprising

For each target object to be verified, the usage behavior number for distributing to multiple sample of users of the target object is obtained According to, wherein the ratio of the multiple sample of users is the verifying based on determined by the last round of verifying for the target object As a result dynamic adjustment is carried out；

According to the usage behavior data of the multiple sample of users, the characteristic value of each pre-set business index is determined, wherein institute Stating pre-set business index is determined according to the corresponding application scenarios of the target object；

Using empirical Bayes method according to the prior probability distribution and the characteristic value of each pre-set business index, determine each The Posterior probability distribution of the pre-set business index；

Using Monte Carlo method according to the Posterior probability distribution, the verification result for being directed to the target object is determined；Wherein, The verification result includes: opportunity values for characterizing target object relative advantage and for characterizing target object for providing line The value-at-risk for the potential loss that upper user uses；

If the verification result meets default verifying termination condition, according to the verification result of each target object, It chooses in multiple target objects to be verified for providing the target object used of user on line.

2. the method according to claim 1, wherein further include:

If the verification result is unsatisfactory for default verifying termination condition, according to the verification result of each target object, It determines and carries out the sample of users for distributing to each target object when next round verifying for multiple target objects to be verified Ratio, and the step of continuing to execute the usage behavior data for obtaining the multiple sample of users for distributing to the target object, until true Make the default verifying termination condition of satisfaction.

3. according to the method described in claim 2, it is characterized in that, the verifying knot according to each target object Fruit determines the sample of users for carrying out distributing to each target object when next round verifying for multiple target objects to be verified Ratio, comprising:

According to the size relation of the opportunity values of each target object, determines and carried out for multiple target objects to be verified Next round distributes to the ratio of the sample of users of each target object when verifying；

4. the method according to claim 1, wherein described utilize Monte Carlo method according to the posterior probability Distribution determines the verification result for being directed to the target object, comprising:

For each pre-set business index, the Posterior probability distribution of the pre-set business index is carried out at random sampling Reason, obtains the randomly sampled data of the pre-set business index；

Using Monte Carlo method according to the randomly sampled data of each pre-set business index, the target object is determined Specified evaluation index posterior probability sampled result；

Using Monte Carlo method according to the posterior probability sampled result, the verification result for being directed to the target object is determined.

5. according to the method described in claim 4, it is characterized in that, described utilize Monte Carlo method according to the posterior probability Sampled result determines the verification result for being directed to the target object, comprising:

For each target object, is sampled and tied according to the posterior probability of the target object using Monte Carlo method Fruit determines that the specified evaluation index of the target object is better than the probability value of other target objects；The probability value is determined For the opportunity values for the target object；

For each target object, is sampled and tied according to the posterior probability of the target object using Monte Carlo method Fruit determines the mathematics phase of the difference of the value of the specified evaluation index of the target object and potential optimal target object It hopes；The mathematic expectaion is determined as to the value-at-risk for the target object, wherein the potential optimal target object is Refer to the maximum target object of value of the specified evaluation index of multiple target objects to be verified.

6. the method according to claim 1, wherein being divided using Monte Carlo method according to the posterior probability Cloth is determined and is directed to after the verification result of the target object, further includes:

It judges whether there is the opportunity values and the value-at-risk is all satisfied at least one target object of preset threshold condition, In, the preset threshold condition is that the opportunity values are greater than the first preset threshold and the value-at-risk less than the second preset threshold；

Meet at least one target object of preset threshold condition if it exists, it is determined that the verification result meets default verifying eventually Only condition；

Corresponding, the verification result according to each target object is chosen in multiple target objects to be verified For providing the target object used of user on line, comprising:

In at least one target object for meeting preset threshold condition, the maximum target object of opportunity values is determined as being used to mention The target object used for user on line.

7. the method according to claim 1, wherein obtaining the multiple sample of users for distributing to the target object Usage behavior data before, further includes:

For each target object to be verified, judgement carries out the acquisition of the usage behavior data of sample of users for epicycle verifying Whether the time is greater than preset time threshold；

If the determination result is YES, then the step for obtaining the usage behavior data for the multiple sample of users for distributing to the target object is executed Suddenly；

If judging result be it is no, continuing with epicycle verify carry out sample of users usage behavior data acquisition, until adopting Collect the time greater than preset time threshold, executes the usage behavior data for obtaining the multiple sample of users for distributing to the target object Step.

8. method according to any one of claims 1 to 7, which is characterized in that multiple target objects include: multiple versions This destination application or multiple information recommendation algorithms for destination application；

The pre-set business index includes: at least one of the click parameter of multimedia messages, browsing parameter, play parameter.

9. a kind of target object verifies device characterized by comprising

Behavioral data obtains module, and for being directed to each target object to be verified, the multiple of the target object are distributed in acquisition The usage behavior data of sample of users, wherein the ratio of the multiple sample of users is based on for the upper of the target object One wheel verifies identified verification result and carries out dynamic adjustment；

Characteristic value determining module determines each pre-set business for the usage behavior data according to the multiple sample of users The characteristic value of index, wherein the pre-set business index is determined according to the corresponding application scenarios of the target object；

Posterior probability distribution determining module, for general according to the priori of each pre-set business index using empirical Bayes method Rate distribution and the characteristic value, determine the Posterior probability distribution of each pre-set business index；

Verification result determining module, for, according to the Posterior probability distribution, determining using Monte Carlo method and being directed to the mesh Mark the verification result of object；Wherein, the verification result includes: opportunity values for characterizing target object relative advantage and is used for Characterization target object is for providing the value-at-risk for the potential loss that user on line uses；

First processing module, if meeting default verifying termination condition for the verification result, according to each target object The verification result, in multiple target objects to be verified choose for providing the target object used of user on line.

10. device according to claim 9, which is characterized in that described device further include:

Second processing module, if default verifying termination condition is unsatisfactory for for the verification result, according to each target pair The verification result of elephant determines and distributes to each target when carrying out next round verifying for multiple target objects to be verified The ratio of the sample of users of object, and continue to execute the usage behavior number for obtaining the multiple sample of users for distributing to the target object According to the step of, until determining to meet default verifying termination condition.

11. device according to claim 10, which is characterized in that the Second processing module is specifically used for:

12. device according to claim 9, which is characterized in that the verification result determining module is specifically used for:

13. device according to claim 12, which is characterized in that the verification result determining module is further specific to use In:

14. device according to claim 9, which is characterized in that further include acquisition parameter judgment module, be used for: