CN103324566A - Multi-version test method and device for web page product - Google Patents
Multi-version test method and device for web page product Download PDFInfo
- Publication number
- CN103324566A CN103324566A CN2012100740651A CN201210074065A CN103324566A CN 103324566 A CN103324566 A CN 103324566A CN 2012100740651 A CN2012100740651 A CN 2012100740651A CN 201210074065 A CN201210074065 A CN 201210074065A CN 103324566 A CN103324566 A CN 103324566A
- Authority
- CN
- China
- Prior art keywords
- version
- test
- user
- beta
- benchmark
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a multi-version test method and device for a web page product, which are used for accurately estimating time required for multi-version tests, improving the test efficiency of the multi-version tests and the accuracy of multi-version test results. The multi-version test method comprises the following steps: aiming at any reference index, obtaining an expected change value between the index value of the reference index and a benchmark version in each version to-be-tested; confirming test time required by each version to-be-tested respectively according to the expected change value; receiving the access request sent by each accessing user in the test time; allocating each accessing user to a corresponding version test bin according to a preset user shunting rule; recording the operation information of each accessing user according to any test version; carrying out statistical analysis; obtaining a test result.
Description
Technical field
The application relates to page technical field of measurement and test, relates in particular to a kind of webpage product many versions method of testing and device.
Background technology
In the process of webpage development, to the development link that is absolutely necessary of testing.Whether this product up-gradation can satisfy final user's demand, often has bigger test for the product owner, can only estimate user's expectation by means of the experience of oneself.Adopt traditional approach to come the iterative development product to exist certain risk simultaneously may not one to reach expection surely.
Along with the development of Internet technology, many versions measuring technology has appearred, and the test of hypothesis in the mathematical statistics provides theoretical foundation for many version tests.Many version tests are webpage products of user's Random assignment being given different editions (comprising benchmark version and beta version), because this randomness is not subjected to the interference of user behavior and other external factors, thereby make test result more objective, can carry out actual test at region or user group, and by collecting and the analytical test result, assessing which version can meet specific requirement, final, carries out the full dose issue with selecting by the webpage product that meets the demands that test is determined.Adopt this iterative issue, can farthest cater to user's request, reduce the injury to user's experience, thereby also provide the foundation that continues upgrading for the later stage product.
At present, the test of many versions is being used in a lot of industries, and for example medical science, agricultural etc. are because the internet product renewal frequency is fast.Short and the low characteristic of cost of cycle makes this mode obtain wideling popularize.Well-known Internet firm mostly can provide the implementation of many version tests.At present, comparatively typical implementation is as follows:
Step 1, provide the webpage product of a plurality of versions of test;
Wherein, each beta version can be the URL of full page, also can be a certain zone of the page.
Step 2, the test address that step 1 is provided or zone name are created a test case as parameter;
After step 3, test case are created and finished, embed shunting and the statistics script that generates in benchmark version corresponding page, embed the statistics script in beta version corresponding page and landing page (landing page) corresponding page;
Step 4, when the user visits the benchmark version, can the user be branched to each beta version by equal proportion, and add up the index of correlation of each beta version corresponding page.
In existing many versions testing scheme, rule of thumb estimated the test duration that needs by professional responsible official, make test result depend on professional responsible official's experience, influenced testing efficiency, simultaneously, just begun not have empirical value can with reference to the time, be accurately to estimate test how long could cover all test variable, thereby can't accurately estimate the time that test needs, make each test variable can not get abundant test, influence precision of test result.
Summary of the invention
The embodiment of the present application provides a kind of webpage product many versions method of testing and device, in order to accurately to estimate the time that many version tests need, improves testing efficiency and the precision of test result of many version tests.
The embodiment of the present application provides a kind of many versions of webpage product method of testing, comprising:
At arbitrary reference index, obtain in each beta version of expection, the desired value of this reference index compares the expectation value of variation with the benchmark version;
According to described expectation value, determine the test duration that each beta version needs respectively;
In the described test duration, receive the request of access that each calling party is submitted to; And
According to pre-configured selector switch parameter, determine that the user of described selector switch parameter correspondence shunts rule;
Shunt rule according to described user, each calling party is dispensed in the corresponding version test bucket; And
Record the operation of each calling party respectively, obtain test result.
The embodiment of the present application provides a kind of many versions of webpage product proving installation, comprising:
Acquiring unit is used at arbitrary reference index, obtains in each beta version of expection, the desired value of this reference index compares the expectation value of variation with the benchmark version;
First determining unit is used for according to described expectation value, determines the test duration that each beta version needs respectively;
Receiving element was used for receiving the request of access that each calling party is submitted in the test duration of determining in described first determining unit;
Second determining unit is used for according to pre-configured selector switch parameter, determines that the user of described selector switch parameter correspondence shunts rule;
Allocation units, the user who is used for determining according to described second determining unit shunts rule, and each calling party is dispensed in the corresponding version test bucket;
Record cell is used for recording respectively the operation of each calling party, obtains test result.
Webpage product many versions method of testing and device that the embodiment of the present application provides, at first obtain the expectation value that a certain reference index of expection reaches, and then the test duration of calculating needs according to the expectation value of this reference index, like this, when implementing test, can be in the test duration that calculates, according to default shunting rule, the request of access that the user is submitted to is dispensed in the different version test buckets tests, because the calculating of test duration need not to rely on people's experience, thereby can effectively avoid human factor and lack experience and the test that causes is insufficient, improve testing efficiency and precision of test result.
The application's further feature and advantage will be set forth in the following description, and, partly from instructions, become apparent, perhaps understand by implementing the application.The application's purpose and other advantages can realize and obtain by specifically noted structure in the instructions of writing, claims and accompanying drawing.
Description of drawings
Fig. 1 is in the embodiment of the present application, the implementing procedure synoptic diagram of many versions of webpage product method of testing;
Fig. 2 a is in the embodiment of the present application, the advertisement pattern that the benchmark version is thrown in;
Fig. 2 b is in the embodiment of the present application, the advertisement pattern that beta version is thrown in;
Fig. 3 is in the embodiment of the present application, distributes the logical organization synoptic diagram of version test bucket for the user;
Fig. 4 a is in the embodiment of the present application, the test frame synoptic diagram of many version tests of a test variable;
Fig. 4 b is in the embodiment of the present application, the test frame synoptic diagram of many version tests of a plurality of test variable;
Fig. 5 is in the embodiment of the present application, the effect synoptic diagram of the configuration page;
Fig. 6 is in the embodiment of the present application, the schematic flow sheet that many versions of the webpage product method of testing that adopts the embodiment of the present application to provide is tested;
Fig. 7 is in the embodiment of the present application, the structural representation of many versions of webpage product proving installation.
Embodiment
In order accurately to estimate the test duration of many version tests, make each test variable can access abundant test, to reach the purpose that improves testing efficiency and test result accuracy, the embodiment of the present application provides a kind of webpage product many versions method of testing and device.
Describe below in conjunction with the preferred embodiment of Figure of description to the application, be to be understood that, preferred embodiment described herein only is used for description and interpretation the application, and be not used in restriction the application, and under the situation of not conflicting, embodiment and the feature among the embodiment among the application can make up mutually.
Implement one
Embodiment one is in the embodiment of the present application, and the implementing procedure of many versions of webpage product method of testing as shown in Figure 1, may further comprise the steps:
S101, at arbitrary reference index, obtain in each beta version of expection, the desired value of this reference index compares the expectation value of variation with the benchmark version;
During concrete enforcement, different webpage products have different measurement indexs, therefore, the reference index that different webpage products is paid close attention to is also different, for example, for e-commerce website, the reference index of concern mainly contains conversion ratio, order conversion ratio and spring mistake rate etc., wherein conversion ratio refers in a measurement period, and the number of times of finishing the conversion behavior accounts for the ratio of the total number of clicks of promotion message; The order conversion ratio refers in a measurement period, and in the middle of the all-access person, there have the number of how many ratios to carry out to be a certain to the favourable action behavior in website, for example is success and places an order the behavior of payment for e-commerce website; Spring mistake rate refers to that in a measurement period, the user browses first page accounts for the total access times of this entrance with regard to the access times of leaving ratio.When testing, need at first obtain this test of expectation, the expectation value that the desired value of a certain reference index changes with respect to the benchmark version, namely expect the range value that beta version and benchmark version promote or reduce, be that conversion ratio is example with the reference index, suppose that the conversion ratio in the benchmark version is 2%, the expectation beta version rises to 40% than the conversion ratio of benchmark version, expects that namely the expectation value that beta version can promote than benchmark version is 2%*40%=0.008.
S102, according to this expectation value, determine the test duration that each beta version needs respectively;
Concrete, in the embodiment of the present application, adopt sample value estimation algorithm in the probability statistics, in average user visit capacity, expectation value and this test according to the benchmark version, the number of beta version and benchmark difference in version variable calculates the needed test duration, concrete formula is as follows:
Wherein: n represents the test duration of these beta version needs; δ is standard deviation, in the sample distribution to variational a kind of measurement factor,
Cov represents that in the benchmark version, the desired value of a certain reference index correspondence need to prove, the reference index that different webpage products is paid close attention to can be different; R is the variable factor, the number of the otherness variable between expression benchmark version and the beta version; Δ represents to expect that this beta version compares the expectation value of variation with the benchmark version, expects that namely beta version compares the range value that promotes or reduce with the benchmark version; S is customer volume, the average user visit capacity of expression benchmark version; Tra is user's partition coefficient, and expression desires to distribute to user's ratio of beta version.
In the above-mentioned formula, the test duration has two parts to calculate, namely the needed customer volume of test variable and the visit beta version customer volume, wherein:
The needed customer volume cou of test variable can be expressed as:
The standard error that expression causes; The customer volume d_cou of visit beta version can be expressed as: d_cou=s*tra.Therefore, when estimating the test duration of these test needs, need pre-determine following data: the average user visit capacity s (can be set to the daily visit of benchmark version) of benchmark version, desire to distribute to user's ratio tra of beta version, the number r of the otherness variable (being the difference between benchmark version and the beta version) that benchmark version and beta version contrast exist, in the benchmark version, the desired value cov of a certain reference index, the expected test version is compared with the benchmark version, the range value Δ that the desired value of this reference index promotes or reduces.
In order better to understand the embodiment of the present application, below by in the e-commerce website, the mode of throwing in dissimilar advertisements according to Different Strategies at same advertisement position illustrates definite method of the test duration of many versions test needs.
Suppose to have in certain ecommerce the advertisement position of a 250*250 on the net, existing two types advertisement pattern, respectively shown in Fig. 2 a and Fig. 2 b, suppose, Fig. 2 a is the benchmark version, is the advertisement of throwing in, and Fig. 2 b is beta version, beta version is adjusted page layout with respect to the benchmark version.Comprise two test variable, be respectively: the adjustment of first test variable---page layout, the adjustment of second test variable---weight factor, suppose that the reference index that this test is paid close attention to is landing page (landing page) conversion ratio, and the related data of benchmark version is as follows: the average user visit capacity of benchmark version is 30000, the landing page conversion ratio of benchmark version is 2%, at first test variable, the expected test version is compared with basic version, landing page conversion ratio improves 40%, at first test variable, the expected test version is compared with basic version, landing page conversion ratio improves 20%, and determines in this test, distributes 20% user to each beta version each, the user who namely distributes 30000*20%=6000 at the beta version of the first test variable correspondence, distribute the user of 30000*20%=6000 at the beta version of the second test variable correspondence, below at each test variable, determine the time that its corresponding beta version needs.
One, the adjustment of page layout
By benchmark version and beta version, mainly contain 4 otherness variablees as can be seen.Calculating the test duration by above-mentioned formula is:
In order to ensure can this test variable accessing abundant test, can test according to 4 days test duration.
Two, the adjustment of weight factor
Adjust owing to have two weight factors in the matching strategy, therefore, beta version is compared with the benchmark version, has two otherness variablees, and like this, the test duration that is needed by above-mentioned formula calculating is:
Follow-up test duration that needs basis to determine tests beta version and gets final product, and therefore, many versions of webpage product method of testing can also may further comprise the steps:
S103, in the test duration of determining, receive the request of access that each calling party is submitted to;
S104, the default user of basis shunt rule, and each calling party is dispensed in the corresponding version test bucket;
S105, at arbitrary beta version, record the operation information of each calling party respectively, and carry out statistical study and obtain test result.
When the many versions of specific implementation are tested, different implementations can be arranged, consider that the singularity of Web environment and user experience etc., can not all adopt new shunting rule that all users are redistributed at every turn, therefore, in the embodiment of the present application, adopt the theory of random sampling Jianyang in the mathematical statistics to adopt a branch bucket mode to realize many versions tests.Arbitrary calling party is when visiting first, according to default shunting rule this calling party is dispensed in a certain version test bucket, and be returned as the bucket sign of the version test bucket of its distribution to the user, when the user is follow-up when visiting again, need judge at first whether this user carries the bucket sign, if do not have, then distribute again, otherwise directly this calling party location is dispensed in the corresponding version test bucket.Owing to when calling party conducts interviews first, according to the shunting rule of presetting the user is dispensed in a certain version test bucket, when user base number was enough big, distribution can be satisfied the prospective users formation according to the 0-1 in the theory of probability.As shown in Figure 3, the logical organization synoptic diagram that distributes version test bucket for the user.
For the different web pages product, can select different shunting rules that calling party is shunted.The shunting rule is actually a kind of allocation algorithm, distributes as Random assignment, poll, distributes or distributes etc. according to IP_hash according to Cookie.
Based on above-mentioned principle, the embodiment of the present application has proposed a kind of general many versions test frame that divides barrel mode, and its structure is respectively shown in Fig. 4 a and Fig. 4 b, wherein:
The test frame that Fig. 4 a tests for the many versions that only relate to a test variable, Fig. 4 b is the test frame that relates to many version tests of a plurality of test variable, for example, in certain test, need carry out adjusting the corresponding beta version in back to page layout, and sort algorithm has been carried out adjusting corresponding beta version tested simultaneously, at this moment, just relate to two test variable: page layout and sort algorithm.Many version tests for only relating to a test variable include only a test group, can comprise test bucket 1, test bucket 2...... in this test group; For the many version tests that relate to a plurality of test variable, comprise a plurality of test groups, each test group can comprise test bucket 1, test bucket 2......, when the user asks to visit, to at first this user be assigned to a test group, and then according to shunting rule the user is dispensed in a certain test barrel, in each test bucket the variable that needs dynamic replacement is arranged, as when relating to the adjustment of page layout, the variable of this dynamic replacement can be front end page block, and when relating to sort algorithm and adjusting, the address that the variable of this dynamic replacement can call for the rear end sort algorithm.
Because for e-commerce website, it has its unique information issue, index, ordering and treatment scheme, the website visiting user is according to the own information of interest of keyword lookup, and the information search of inside, website, screening, to sort and represent the user be fully transparent.In order to allow the user find the information that satisfies its demand in the short as far as possible time, the website needs the webpage product to be upgraded at aspects such as index, ordering scheduling algorithm and page structures regularly.In the process of webpage product up-gradation, may relate to the adjustment of many aspects, according to existing many version test implementations, because test variable is too much, difference is excessive, can't know that also each test variable is to the influence of test result even adopt many version tests, and current most of webpage product adopts model-view-controller (MVC, Model-View-Controller) layered model designs, in the prior art, based on the many versions test of this hierarchy, the shunting result on upper strata has influence on the user base number of lower floor, makes the user base number value of lower floor constantly diminish and the test result that causes is inaccurate.In the embodiment of the present application, based on many versions test frame of minute bucket formula, each test group can correspond to the test variable that each layer relates to, for example, and the adjustment of page elements, structure, the variation of the adjustment of sort algorithm and intermediary service logical layer etc.Like this, the webpage product of layer-stepping design has been carried out the flattening processing, in webpage product up-gradation process, the test variable that each layer relates to is isolated at horizontal structure, it can not influenced each other, comparatively the adjustment of the test variable of each layer of objective assessment is to the influence of test result, thereby efficiently solves the above-mentioned problems in the prior art.
Embodiment two
Embodiment two is on the basis of embodiment one, increase selector switch parameter configuration step, by the selector switch parameter is configured, can determine different shunting rules according to actual needs, to satisfy actual testing requirement better, make test result more accurate.
Concrete, the selector switch parameter can include but not limited to IP (Internet protocol) address, the Cookie into calling party, also the white list that can arrange for network side and custom parameter etc.
Wherein, when the selector switch parameter is the IP of calling party, the shunting rule of its correspondence can for: according to the IP address of calling party, determine the cryptographic hash of described IP address correspondence; And determine to specify the corresponding same version of cryptographic hash in the interval range to test bucket.After receiving user's request of access, according to the IP address of carrying in the request of access, utilize hash function to determine the cryptographic hash of this IP address correspondence, when specifically implementing, can specify the corresponding identical version test bucket of cryptographic hash in a certain interval range, and finish the shunting to the user accordingly; When the selector switch parameter is the Cookie of calling party, the shunting rule of its correspondence can for: determine to specify the corresponding same version test of the Cookie bucket in the interval range, and finish the shunting to calling party accordingly.
During concrete enforcement, also can adopt self-defining mode to determine the selector switch parameter, for example, can extract some user's characteristic information of comprising among Cookie, Session (session) or the Request (request) as the selector switch parameter, correspondingly, determine the shunting rule according to the user's characteristic information of extracting.Be example with the key word information of carrying among the Request, the shunting rule of its correspondence can for: the user that will inquire about same key word information is dispensed in the same version test bucket.
Preferably, in order to reach the visual configuration to the selector switch parameter, selector switch reuse and the selector switch parameter modification after come into force in real time, need not to revise the selector switch parameter issues at every turn again, in the embodiment of the present application, finish the configuration of selector switch parameter by the management end instrument, concrete treatment scheme is as follows:
Step 1, when creating a new test, the different beta versions that comprise in configuration testing entry-into-force time, selector switch parameter, test group and each the test group as shown in Figure 5, are the effect synoptic diagram that disposes the page dynamically;
Step 2, externally provide the service of webpage product test configuration change, the service of obtaining configuration parameter corresponding information according to the webpage name of product is provided simultaneously;
Step 3, when new test is created or is revised, the modification information of configuration parameter can be stored in this locality, before beginning test, at first whether the query configuration parameter has renewal, if having, then can ask the data message of configuration parameter; If do not have, the data message of access configuration parameter not then like this, can effectively avoid all obtaining the wasting of resources that the data message of configuration parameter causes at every turn.
Embodiment three
In order to understand the embodiment of the present application better, describe below in conjunction with the application scenarios of the embodiment of the present application embodiment to the embodiment of the present application.
For the upgrading of webpage product, no matter be the variation of page layout or page elements, the adjustment of backstage algorithm logic, the still increase and decrease of flow process must have the application layer that externally represents, thereby the formulation of testing scheme is based on also that this layer design.As shown in Figure 6, many versions of the webpage product method of testing that adopts the embodiment of the present application the to provide flow process of testing is as follows:
S601, reception user's request of access;
Wherein, can carry relevant user information in the request of access that the user submits to, for example, user's IP address, Cookie etc.
The user profile of carrying in S602, the extraction request of access, and be stored in the local thread of this visit;
S603, obtain the selector switch parameter of the title correspondence of webpage product;
Concrete, the title of Sheng Ji webpage product is obtained the selector switch parameter of configuration from the management tool end as required.
The selector switch parameter that S604, basis are obtained is determined corresponding shunting rule;
S605, according to the shunting rule determined, the user of request visit is dispensed in the corresponding version test bucket;
Be that the IP address is example with the selector switch parameter, at first need to calculate the cryptographic hash of this IP address correspondence, and the interval range under definite this cryptographic hash, this user is dispensed in the version test bucket of the cryptographic hash correspondence in this interval range, and from version test bucket, get access to the dynamic variable that needs, utilize these dynamic variables that content corresponding in the benchmark version is replaced, finish this test.
S606, record the operation information of each calling party, carry out statistical study and obtain test result.
Embodiment four
The final purpose of many version tests is to select a version that more meets user's experience to carry out the full dose issue.But, estimate the good of this version for the variation from reference index that the webpage product of arbitrary version is simple, consequently unilateral, and be inaccurate, for the difference of compare test version and benchmark version, adopt the card side of test of hypothesis in the mathematical statistics to distribute the multifactorial evaluation beta version whether than benchmark version tangible improvement to be arranged in the embodiment of the present application.Certainly, should consider that noise data is to the influence of statistics, adopt the normal distribution of confidence alpha=0.05 in the embodiment of the present application, namely when the probability of outcome value of chi-square value greater than 95% the time, illustrate that test result is believable, beta version is that obvious improvement is arranged than the benchmark version, and namely beta version is better than the benchmark version.During concrete enforcement, do not do restriction for the setting of degree of confidence, can determine degree of confidence according to actual needs.
Based on above-mentioned analysis, step S105 can comprise:
Step 1, add up actual user's visit capacity of this beta version and the actual desired value of reference index correspondence respectively;
Concrete, after receiving user's request of access, can this user be dispensed in a certain version test bucket according to certain shunting rule, the corresponding version information of each version test bucket, when returning response message to the user, this version information is set in the head response, and is back to the page of user's visit, in order to operation and the record of following the tracks of user's later stage, so that follow-up operation information according to the user carries out statistical study etc.
Step 2, according to actual user's visit capacity of beta version and the actual desired value of reference index correspondence, and in the average user visit capacity and benchmark version of benchmark version, the desired value of reference index correspondence, adopt card side's formula to determine this opinion test benchmark version chi-square value corresponding with beta version;
Concrete, card side's formula of employing is as follows:
Wherein: χ
2Represent the chi-square value that this beta version and benchmark version calculate; f
0Obtain data value in the actual test of beta version; f
eThe actual value that expression calculates according to the card mode; N represents actual user's visit capacity sum of benchmark version and beta version; R represents the number of beta version; C represents the data dimension that beta version and benchmark version need compare; f
XyThe real data that expression counts at each beta version; f
xAnd f
yRepresent with in each beta version the data sum of same dimension respectively.
The computing formula of degree of freedom df is: df=(R-1) * (C-1), and wherein: R represents the number of beta version; C represents the data dimension that beta version and benchmark version need compare.
Step 3, when the chi-square value of determining surpasses predetermined threshold value, determine that beta version is better than the benchmark version.
In order to understand the process of test result assessment better, be that example describes with two types advertisement pattern shown in Fig. 2 a and Fig. 2 b still below, beta version is with respect to the benchmark version, page layout is adjusted, simultaneously, back-end algorithm has had certain adjustment for the matching strategy of advertisement, has revised two weight factors, also need test in this test.Seen from the above description, in the embodiment of the present application, comprise two test variable, be respectively: the adjustment of first test variable---page layout, the adjustment of second test variable---weight factor, suppose that the reference index that this test is paid close attention to is landing page (landing page) conversion ratio, and the related data of benchmark version is as follows: the average user visit capacity of benchmark version is 30000, the landing page conversion ratio of benchmark version is 2%, below at each test variable, respectively the beta version of this test variable correspondence is assessed.
One, the adjustment of page layout
Suppose to implement after the test, the data that actual test is collected are: actual user's visit capacity is: 7349, really the user to landing page is 224, then can obtain actual conversion is 224/7349*100%=3.04%, than the conversion ratio (2%) of benchmark version to a certain degree lifting is arranged, but can only illustrate that beta version is better than benchmark version within the specific limits, but between two versions whether notable difference be arranged, need further to determine according to the chi-square value of beta version.
Seen from the above description, the data dimension that this assessment relates to is two dimension, be C=2, the data that comprise beta version and benchmark version, in each version by comprising in the calling party user that success transforms and the number of users that transforms of success, wherein, in the benchmark version, the number of users that success transforms is: 30000*2%=600, and the number of users of success conversion is not: 30000-600=29400; In the beta version, the number of users that success transforms is: 7349*3.04%=224, and the number of users that transforms of success is not: 7349-224=7125, specifically can be with following matrix representation:
For convenience of description, with f
XyEach element in the representing matrix, wherein x represents that row number, y represent row number, owing to only comprise benchmark version and a beta version, therefore, R=2 then according to card side formula calculating chi-square value is:
When degree of freedom df=(R-1) * (C-1)=(2-1) * (2-1)=1, according to Ka Fangbiao as can be known,
Hence one can see that, and degree of confidence is higher than 99%, illustrates that beta version and benchmark version have notable difference.
Two, weight factor adjustment
Suppose to implement after the test, the data that actual test is collected are: actual user's visit capacity: 2487, really the user to landing page is 57, then can obtain actual conversion is 57/2487*100%=2.29%, than the conversion ratio (2%) of benchmark version to a certain degree lifting is arranged also, following chi-square value according to beta version determines whether beta version has notable difference than benchmark version.
Seen from the above description, the data dimension that this assessment relates to is two dimension, be C=2, the data that comprise beta version and benchmark version, in each version by comprising in the calling party user that success transforms and the number of users that transforms of success, wherein, in the benchmark version, the number of users that success transforms is: 30000*2%=600, and the number of users of success conversion is not: 30000-600=29400; In the beta version, the number of users that success transforms is: 2487*2.29%=57, and the number of users that transforms of success is not: 2487-57=2430, specifically can be with following matrix representation:
Equally, for convenience of description, with f
XyEach element in the representing matrix, wherein x represents that row number, y represent row number, owing to only comprise benchmark version and a beta version, therefore, R=2 then according to card side formula calculating chi-square value is:
When degree of freedom df=(R-1) * (C-1)=(2-1) * (2-1)=1, according to Ka Fangbiao as can be known,
Hence one can see that, and degree of confidence illustrates beta version and benchmark version no significant difference less than 80%.
Further, assessment result according to beta version, can adjust the selector switch parameter, for example adjust user's ratio of distributing to beta version, proceed test, and continue test result is assessed, thereby, being implemented in the test duration, test is implemented in circulation constantly, can improve the webpage product gradually, and produce a desired effect.
Many versions of webpage product method of testing that the embodiment of the present application provides, can adopt Java language to finish, and can be applied in the different business scenarios, whether test duration and beta version have the appraisal procedure of notable difference than the benchmark version, can adopt any language to realize, because the calculated amount that relates to is little, and soft, hardware are not had specific (special) requirements.
Based on same inventive concept, a kind of many versions of webpage product proving installation also is provided in the embodiment of the present application, because the principle of this device solves problem is similar to many versions of webpage product method of testing, so the enforcement of this device can repeat part and repeat no more referring to the enforcement of method.
As shown in Figure 7, in the embodiment of the present application, the structural representation of many versions of webpage product proving installation comprises:
Acquiring unit 701 is used at arbitrary reference index, obtains in each beta version of expection, the desired value of this reference index compares the expectation value of variation with the benchmark version;
First determining unit 702 for the expectation value of obtaining according to acquiring unit, is determined the test duration that each beta version needs respectively;
Receiving element 703 was used for receiving the request of access that each calling party is submitted in the test duration that first determining unit 702 is determined;
Wherein, first determining unit 702 can be used for determining the test duration that arbitrary beta version needs according to following formula:
Wherein: n represents the test duration of these beta version needs;
Cov represents in the benchmark version, the desired value of arbitrary reference index correspondence; R represents the number of the otherness variable between benchmark version and the beta version; Δ represents to expect that this beta version compares the expectation value of variation with the benchmark version; S represents the average user visit capacity of benchmark version; Tra represents to desire to distribute to user's ratio of beta version.
During concrete enforcement, webpage product institute version method of testing can also comprise:
The 3rd determining unit, be used for if when needing test the beta version of a plurality of test variable correspondences simultaneously, shunting rule according to described user, before being dispensed to each calling party in the corresponding version test bucket, determine the test group of each calling party correspondence, the test group is corresponding one by one with test variable, and each test group comprises a plurality of version test buckets.
During concrete enforcement, webpage product institute version method of testing can also comprise:
Second determining unit is used for according to pre-configured selector switch parameter, determines that the user of this selector switch parameter correspondence shunts rule.
During concrete enforcement, the selector switch parameter can but be not limited to comprise the IP address of calling party and the Cookie of calling party, when the selector switch parameter is the IP address of calling party, second determining unit, concrete for the IP address according to calling party, determine the cryptographic hash of this IP address correspondence; And determine to specify the corresponding same version of cryptographic hash in the interval range to test bucket; When the selector switch parameter is the Cookie of calling party, second determining unit is concrete for determining to specify the corresponding same version of Cookie in the interval range to test bucket.
During concrete enforcement, statistical analysis unit 705 can comprise:
The record subelement is used at arbitrary beta version, records each user's operation information respectively;
The statistics subelement is used for the operation information according to record subelement record, adds up actual user's visit capacity of this beta version and the actual desired value of reference index correspondence respectively;
First determines subelement, be used for according to actual user's visit capacity of beta version and the actual desired value of reference index correspondence, and in the average user visit capacity and benchmark version of benchmark version, the desired value of reference index correspondence, determine that according to card side's formula this takes turns the test benchmark version chi-square value corresponding with beta version;
Second determines subelement, is used for determining that beta version is better than the benchmark version when described chi-square value surpasses predetermined threshold value.
Webpage product many versions method of testing and device that the embodiment of the present application provides, at first obtain the expectation value that a certain reference index of expection reaches, and then the test duration of calculating needs according to the expectation value of this reference index, like this, when implementing test, can be in the test duration that calculates, according to default shunting rule, the request of access that the user is submitted to is dispensed in the different version test buckets tests, because the calculating of test duration need not to rely on people's experience, thereby can effectively avoid human factor and lack experience and the test that causes is insufficient, improve testing efficiency and precision of test result.
Those skilled in the art should understand that the application's embodiment can be provided as method, system or computer program.Therefore, the application can adopt complete hardware embodiment, complete software embodiment or in conjunction with the form of the embodiment of software and hardware aspect.And the application can adopt the form of the computer program of implementing in one or more computer-usable storage medium (including but not limited to magnetic disk memory, CD-ROM, optical memory etc.) that wherein include computer usable program code.
The application is that reference is described according to process flow diagram and/or the block scheme of method, equipment (system) and the computer program of the embodiment of the present application.Should understand can be by the flow process in each flow process in computer program instructions realization flow figure and/or the block scheme and/or square frame and process flow diagram and/or the block scheme and/or the combination of square frame.Can provide these computer program instructions to the processor of multi-purpose computer, special purpose computer, Embedded Processor or other programmable data processing device to produce a machine, make the instruction of carrying out by the processor of computing machine or other programmable data processing device produce to be used for the device of the function that is implemented in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame appointments.
These computer program instructions also can be stored in energy vectoring computer or the computer-readable memory of other programmable data processing device with ad hoc fashion work, make the instruction that is stored in this computer-readable memory produce the manufacture that comprises command device, this command device is implemented in the function of appointment in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame.
These computer program instructions also can be loaded on computing machine or other programmable data processing device, make and carry out the sequence of operations step producing computer implemented processing at computing machine or other programmable devices, thereby be provided for being implemented in the step of the function of appointment in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame in the instruction that computing machine or other programmable devices are carried out.
Although described the application's preferred embodiment, in a single day those skilled in the art get the basic creative concept of cicada, then can make other change and modification to these embodiment.So claims are intended to all changes and the modification that are interpreted as comprising preferred embodiment and fall into the application's scope.
Obviously, those skilled in the art can carry out various changes and modification and the spirit and scope that do not break away from the application to the application.Like this, if these of the application are revised and modification belongs within the scope of the application's claim and equivalent technologies thereof, then the application also is intended to comprise these changes and modification interior.
Claims (14)
1. many versions of webpage product method of testing is characterized in that, comprising:
At arbitrary reference index, obtain in each beta version of expection, the desired value of this reference index compares the expectation value of variation with the benchmark version;
According to described expectation value, determine the test duration that each beta version needs respectively;
In the described test duration, receive the request of access that each calling party is submitted to; And
Shunt rule according to default user, each calling party is dispensed in the corresponding version test bucket; And
At arbitrary beta version, record the operation information of each calling party respectively, and carry out statistical study and obtain test result.
2. the method for claim 1 is characterized in that, according to described expectation value, determines the test duration that each beta version needs respectively, specifically comprises:
At arbitrary beta version, determine the test duration that this beta version needs according to following formula:
N represents the test duration of these beta version needs;
R represents the number of the otherness variable between benchmark version and the beta version;
Δ represents to expect that this beta version compares the expectation value of variation with the benchmark version;
S represents the average user visit capacity of benchmark version;
Tra represents to desire to distribute to user's ratio of beta version.
3. the method for claim 1 is characterized in that, if when needing to test the beta version of a plurality of test variable correspondences simultaneously, is shunting rule according to described user, be dispensed to each calling party in the corresponding version test bucket before, also comprise:
Determine the test group of each calling party correspondence, described test group is corresponding one by one with test variable, and each test group comprises a plurality of version test buckets.
4. the method for claim 1 is characterized in that, determines that in accordance with the following methods the user shunts rule:
According to pre-configured selector switch parameter, determine that the user of this selector switch parameter correspondence shunts rule.
5. method as claimed in claim 4 is characterized in that, described selector switch parameter comprises the Internet protocol IP address of calling party; And
The user who determines described selector switch parameter correspondence shunts rule, specifically comprises:
According to the IP address of calling party, determine the cryptographic hash of described IP address correspondence; And
Determine to specify the corresponding same version test of the cryptographic hash bucket in the interval range.
6. method as claimed in claim 4 is characterized in that, described selector switch parameter comprises the Cookie of calling party; And
The user who determines described selector switch parameter correspondence shunts rule, specifically comprises:
Determine to specify the corresponding same version test of the Cookie bucket in the interval range.
7. the method for claim 1 is characterized in that, carries out statistical study and obtains test result, specifically comprises:
Add up actual user's visit capacity of this beta version and the actual desired value of reference index correspondence respectively;
According to actual user's visit capacity of beta version and the actual desired value of reference index correspondence, and in the average user visit capacity and benchmark version of benchmark version, the desired value of reference index correspondence, adopt card side's formula to determine this opinion test benchmark version chi-square value corresponding with beta version;
When described chi-square value surpasses predetermined threshold value, determine that described beta version is better than described benchmark version.
8. many versions of webpage product proving installation is characterized in that, comprising:
Acquiring unit is used at arbitrary reference index, obtains in each beta version of expection, the desired value of this reference index compares the expectation value of variation with the benchmark version;
First determining unit is used for according to described expectation value, determines the test duration that each beta version needs respectively;
Receiving element was used for receiving the request of access that each calling party is submitted in the test duration of determining in described first determining unit;
Allocation units are used for shunting rule according to default user, and each calling party is dispensed in the corresponding version test bucket;
Statistical analysis unit is used for recording the operation of each calling party respectively at arbitrary beta version, and carries out statistical study and obtain test result.
9. device as claimed in claim 8 is characterized in that,
Described first determining unit, concrete being used for determined the test duration that arbitrary beta version needs according to following formula:
Wherein:
N represents the test duration of these beta version needs;
R represents the number of the otherness variable between benchmark version and the beta version;
Δ represents to expect that this beta version compares the expectation value of variation with the benchmark version;
S represents the average user visit capacity of benchmark version;
Tra represents to desire to distribute to user's ratio of beta version.
10. device as claimed in claim 8 is characterized in that, also comprises:
The 3rd determining unit, be used for if when needing test the beta version of a plurality of test variable correspondences simultaneously, shunting rule according to described user, before being dispensed to each calling party in the corresponding version test bucket, determine the test group of each calling party correspondence, described test group is corresponding one by one with test variable, and each test group comprises a plurality of version test buckets.
11. device as claimed in claim 8 is characterized in that, also comprises:
Second determining unit is used for according to pre-configured selector switch parameter, determines that the user of this selector switch parameter correspondence shunts rule.
12. device as claimed in claim 11 is characterized in that, described selector switch parameter comprises the Internet protocol IP address of calling party; And
Described second determining unit, concrete for the IP address according to calling party, determine the cryptographic hash of described IP address correspondence; And determine to specify the corresponding same version of cryptographic hash in the interval range to test bucket.
13. device as claimed in claim 11 is characterized in that, described selector switch parameter comprises the Cookie of calling party; And
Described second determining unit is concrete for determining to specify the corresponding same version of Cookie in the interval range to test bucket.
14. device as claimed in claim 8 is characterized in that, described statistical analysis unit comprises:
The record subelement is used at arbitrary beta version, records each user's operation information respectively;
The statistics subelement is used for the operation information according to record subelement record, adds up actual user's visit capacity of this beta version and the actual desired value of reference index correspondence respectively;
First determines subelement, be used for according to actual user's visit capacity of beta version and the actual desired value of reference index correspondence, and in the average user visit capacity and benchmark version of benchmark version, the desired value of reference index correspondence, determine that according to card side's formula this takes turns the test benchmark version chi-square value corresponding with beta version;
Second determines subelement, is used for determining that described beta version is better than described benchmark version when described chi-square value surpasses predetermined threshold value.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210074065.1A CN103324566B (en) | 2012-03-20 | 2012-03-20 | A kind of multi-version test method and device |
| HK13113231.7A HK1185968A1 (en) | 2012-03-20 | 2013-11-27 | Method and device for multi-version verification of web page product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210074065.1A CN103324566B (en) | 2012-03-20 | 2012-03-20 | A kind of multi-version test method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103324566A true CN103324566A (en) | 2013-09-25 |
| CN103324566B CN103324566B (en) | 2016-04-06 |
Family
ID=49193325
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210074065.1A Active CN103324566B (en) | 2012-03-20 | 2012-03-20 | A kind of multi-version test method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN103324566B (en) |
| HK (1) | HK1185968A1 (en) |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104052809A (en) * | 2014-06-17 | 2014-09-17 | 北京京东尚科信息技术有限公司 | Distribution control method and device for web testing |
| CN104539481A (en) * | 2014-12-26 | 2015-04-22 | 刘永健 | Website testing method and device |
| CN104993946A (en) * | 2015-06-01 | 2015-10-21 | 华为技术有限公司 | Evaluation method, device and system for gated launch |
| CN105262636A (en) * | 2015-09-08 | 2016-01-20 | 摩贝（上海）生物科技有限公司 | Online testing system and method |
| CN105450461A (en) * | 2014-06-05 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Shunting method and network equipment |
| CN105610654A (en) * | 2016-03-02 | 2016-05-25 | 合一网络技术(北京)有限公司 | Server, and policy online test method and system |
| CN105701004A (en) * | 2014-11-28 | 2016-06-22 | 深圳市腾讯计算机系统有限公司 | Application test method and apparatus |
| CN106294124A (en) * | 2016-07-21 | 2017-01-04 | 北京金山安全软件有限公司 | Application product test configuration method and device |
| CN106708787A (en) * | 2015-11-13 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Method and apparatus for determining correctness of business function module |
| CN106959925A (en) * | 2017-04-25 | 2017-07-18 | 北京云测信息技术有限公司 | A kind of version method of testing and device |
| CN107122303A (en) * | 2017-05-03 | 2017-09-01 | 北京云测信息技术有限公司 | A kind of method of testing and device |
| CN107918562A (en) * | 2016-10-11 | 2018-04-17 | 北京京东尚科信息技术有限公司 | A kind of unified interface management method and system |
| CN107957938A (en) * | 2016-10-14 | 2018-04-24 | 北京酷我科技有限公司 | A kind of method and system for obtaining website test data |
| CN108280024A (en) * | 2017-12-29 | 2018-07-13 | 五八有限公司 | Traffic distribution strategy test method, device and electronic equipment |
| CN108334444A (en) * | 2017-12-29 | 2018-07-27 | 广州品唯软件有限公司 | Various dimensions dynamic combined shunts method of servicing, device, terminal and storage medium |
| WO2018137324A1 (en) * | 2017-01-25 | 2018-08-02 | 上海蔚来汽车有限公司 | System upgrade assessment method based on system parameter correlation coefficients |
| CN108829602A (en) * | 2018-06-21 | 2018-11-16 | 北京金山安全软件有限公司 | Test method, test device, electronic equipment and storage medium |
| CN109120720A (en) * | 2018-09-28 | 2019-01-01 | 北京云测信息技术有限公司 | A method of automatic adjustment version tests flow |
| CN109359027A (en) * | 2018-08-15 | 2019-02-19 | 中国平安人寿保险股份有限公司 | Monkey test method, device, electronic equipment and computer readable storage medium |
| CN109684168A (en) * | 2018-12-12 | 2019-04-26 | 厦门理工学院 | A kind of display screen crash detection method |
| CN110020233A (en) * | 2017-07-28 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Investigational data processing method, apparatus and system |
| CN111258912A (en) * | 2020-02-14 | 2020-06-09 | 苏州浪潮智能科技有限公司 | Page testing method, device, terminal and storage medium |
| CN111309614A (en) * | 2020-02-17 | 2020-06-19 | 支付宝(杭州)信息技术有限公司 | A/B test method and device and electronic equipment |
| CN111338607A (en) * | 2020-02-28 | 2020-06-26 | 北京奇艺世纪科技有限公司 | Method and device for selecting service data for testing and electronic equipment |
| CN111352678A (en) * | 2018-12-20 | 2020-06-30 | 阿里巴巴集团控股有限公司 | Information processing method and device |
| CN113032255A (en) * | 2021-03-19 | 2021-06-25 | 广州虎牙科技有限公司 | Response noise recognition method, model, electronic device, and computer storage medium |
| CN114390105A (en) * | 2022-03-01 | 2022-04-22 | 阿里巴巴（中国）有限公司 | Enterprise user distribution method and device based on test |
| CN116633812A (en) * | 2023-05-15 | 2023-08-22 | 之江实验室 | Multi-version synchronous test method and system based on nginx intelligent fault-tolerant routing |
| US11909655B1 (en) | 2022-08-09 | 2024-02-20 | Coupang Corp. | Systems and methods for selective deployment of experimental configurations to mutually exclusive groups |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101118515A (en) * | 2007-09-11 | 2008-02-06 | 腾讯科技（深圳）有限公司 | Automatically testing method and apparatus for list |
| CN102012857A (en) * | 2009-09-08 | 2011-04-13 | 中国工商银行股份有限公司 | Device and method for automatically testing web page |
| US20120054553A1 (en) * | 2010-09-01 | 2012-03-01 | International Business Machines Corporation | Fault localization using condition modeling and return value modeling |
-
2012
- 2012-03-20 CN CN201210074065.1A patent/CN103324566B/en active Active
-
2013
- 2013-11-27 HK HK13113231.7A patent/HK1185968A1/en unknown
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101118515A (en) * | 2007-09-11 | 2008-02-06 | 腾讯科技（深圳）有限公司 | Automatically testing method and apparatus for list |
| CN102012857A (en) * | 2009-09-08 | 2011-04-13 | 中国工商银行股份有限公司 | Device and method for automatically testing web page |
| US20120054553A1 (en) * | 2010-09-01 | 2012-03-01 | International Business Machines Corporation | Fault localization using condition modeling and return value modeling |
Non-Patent Citations (1)
| Title |
|---|
| 杨彬，陈丽容: "基于多版本较准的软件可靠性评估", 《计算机工程与设计》, vol. 28, no. 19, 31 October 2007 (2007-10-31), pages 4597 - 4599 * |
Cited By (42)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105450461A (en) * | 2014-06-05 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Shunting method and network equipment |
| CN105450461B (en) * | 2014-06-05 | 2018-12-14 | 阿里巴巴集团控股有限公司 | A kind of shunt method and the network equipment |
| CN104052809A (en) * | 2014-06-17 | 2014-09-17 | 北京京东尚科信息技术有限公司 | Distribution control method and device for web testing |
| CN104052809B (en) * | 2014-06-17 | 2017-09-26 | 北京京东尚科信息技术有限公司 | A kind of flow-dividing control method and apparatus of website test |
| CN105701004B (en) * | 2014-11-28 | 2020-12-08 | 深圳市腾讯计算机系统有限公司 | Application testing method and device |
| CN105701004A (en) * | 2014-11-28 | 2016-06-22 | 深圳市腾讯计算机系统有限公司 | Application test method and apparatus |
| CN104539481A (en) * | 2014-12-26 | 2015-04-22 | 刘永健 | Website testing method and device |
| CN104993946A (en) * | 2015-06-01 | 2015-10-21 | 华为技术有限公司 | Evaluation method, device and system for gated launch |
| CN104993946B (en) * | 2015-06-01 | 2019-04-05 | 华为技术有限公司 | Appraisal procedure, the device and system of gray scale publication |
| WO2016192488A1 (en) * | 2015-06-01 | 2016-12-08 | 华为技术有限公司 | Rolling release assessment method, device, and system |
| CN105262636A (en) * | 2015-09-08 | 2016-01-20 | 摩贝（上海）生物科技有限公司 | Online testing system and method |
| CN106708787B (en) * | 2015-11-13 | 2019-09-13 | 阿里巴巴集团控股有限公司 | The determination method and device of business function module correctness |
| CN106708787A (en) * | 2015-11-13 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Method and apparatus for determining correctness of business function module |
| CN105610654A (en) * | 2016-03-02 | 2016-05-25 | 合一网络技术(北京)有限公司 | Server, and policy online test method and system |
| CN106294124A (en) * | 2016-07-21 | 2017-01-04 | 北京金山安全软件有限公司 | Application product test configuration method and device |
| CN106294124B (en) * | 2016-07-21 | 2018-12-28 | 北京金山安全软件有限公司 | Application product test configuration method and device |
| CN107918562A (en) * | 2016-10-11 | 2018-04-17 | 北京京东尚科信息技术有限公司 | A kind of unified interface management method and system |
| CN107918562B (en) * | 2016-10-11 | 2021-11-26 | 北京京东尚科信息技术有限公司 | Unified interface management method and system |
| CN107957938A (en) * | 2016-10-14 | 2018-04-24 | 北京酷我科技有限公司 | A kind of method and system for obtaining website test data |
| US10983779B2 (en) | 2017-01-25 | 2021-04-20 | Nio Co., Ltd. | System upgrade assessment method based on system parameter correlation coefficients |
| WO2018137324A1 (en) * | 2017-01-25 | 2018-08-02 | 上海蔚来汽车有限公司 | System upgrade assessment method based on system parameter correlation coefficients |
| CN106959925A (en) * | 2017-04-25 | 2017-07-18 | 北京云测信息技术有限公司 | A kind of version method of testing and device |
| CN107122303A (en) * | 2017-05-03 | 2017-09-01 | 北京云测信息技术有限公司 | A kind of method of testing and device |
| CN107122303B (en) * | 2017-05-03 | 2020-05-15 | 北京云测信息技术有限公司 | Test method and device |
| CN110020233A (en) * | 2017-07-28 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Investigational data processing method, apparatus and system |
| CN108334444A (en) * | 2017-12-29 | 2018-07-27 | 广州品唯软件有限公司 | Various dimensions dynamic combined shunts method of servicing, device, terminal and storage medium |
| CN108280024A (en) * | 2017-12-29 | 2018-07-13 | 五八有限公司 | Traffic distribution strategy test method, device and electronic equipment |
| CN108829602A (en) * | 2018-06-21 | 2018-11-16 | 北京金山安全软件有限公司 | Test method, test device, electronic equipment and storage medium |
| CN109359027A (en) * | 2018-08-15 | 2019-02-19 | 中国平安人寿保险股份有限公司 | Monkey test method, device, electronic equipment and computer readable storage medium |
| CN109120720A (en) * | 2018-09-28 | 2019-01-01 | 北京云测信息技术有限公司 | A method of automatic adjustment version tests flow |
| CN109684168A (en) * | 2018-12-12 | 2019-04-26 | 厦门理工学院 | A kind of display screen crash detection method |
| CN111352678A (en) * | 2018-12-20 | 2020-06-30 | 阿里巴巴集团控股有限公司 | Information processing method and device |
| CN111258912B (en) * | 2020-02-14 | 2023-01-10 | 苏州浪潮智能科技有限公司 | Page testing method, device, terminal and storage medium |
| CN111258912A (en) * | 2020-02-14 | 2020-06-09 | 苏州浪潮智能科技有限公司 | Page testing method, device, terminal and storage medium |
| CN111309614A (en) * | 2020-02-17 | 2020-06-19 | 支付宝(杭州)信息技术有限公司 | A/B test method and device and electronic equipment |
| CN111338607A (en) * | 2020-02-28 | 2020-06-26 | 北京奇艺世纪科技有限公司 | Method and device for selecting service data for testing and electronic equipment |
| CN113032255A (en) * | 2021-03-19 | 2021-06-25 | 广州虎牙科技有限公司 | Response noise recognition method, model, electronic device, and computer storage medium |
| CN113032255B (en) * | 2021-03-19 | 2024-03-01 | 广州虎牙科技有限公司 | Response noise identification method, model, electronic device and computer storage medium |
| CN114390105A (en) * | 2022-03-01 | 2022-04-22 | 阿里巴巴（中国）有限公司 | Enterprise user distribution method and device based on test |
| US11909655B1 (en) | 2022-08-09 | 2024-02-20 | Coupang Corp. | Systems and methods for selective deployment of experimental configurations to mutually exclusive groups |
| CN116633812A (en) * | 2023-05-15 | 2023-08-22 | 之江实验室 | Multi-version synchronous test method and system based on nginx intelligent fault-tolerant routing |
| CN116633812B (en) * | 2023-05-15 | 2023-12-22 | 之江实验室 | Multi-version synchronous test method and system based on nginx intelligent fault-tolerant routing |
Also Published As
| Publication number | Publication date |
|---|---|
| HK1185968A1 (en) | 2014-02-28 |
| CN103324566B (en) | 2016-04-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103324566A (en) | Multi-version test method and device for web page product | |
| CN104348679B (en) | A kind of methods, devices and systems of point of bucket test | |
| Javadi et al. | Characterizing spot price dynamics in public cloud environments | |
| CN103295147B (en) | method, device and system for advertising | |
| US8374912B2 (en) | System and method for managing and optimizing advertising campaigns managed on the internet | |
| US20150248412A1 (en) | Information pushing method and apparatus | |
| Rosli | Determinants of small and medium enterprises performance in the Malaysian auto-parts industry | |
| CN105631600A (en) | Method and system for publishing auto insurance survey tasks | |
| CN104424598A (en) | Cash demand quantity predicating device and method | |
| CN103593350A (en) | Method and device for recommending promotion keyword price parameters | |
| CN101025815A (en) | Advertising system using personal website and method of generating profit | |
| DE102017201173A1 (en) | LATENCY REDUCTION IN THE FEEDBACK-BASED DETERMINATION OF A SYSTEM PERFORMANCE | |
| CN104850627A (en) | Method and apparatus for performing paging display | |
| CN107957938A (en) | A kind of method and system for obtaining website test data | |
| CN102081624B (en) | Data inquiring method and data inquiring device | |
| CN105631732A (en) | Method and device for determining user authority | |
| Cieślik et al. | Agglomeration externalities, competition and productivity: empirical evidence from firms located in Ukraine | |
| CN106611339B (en) | Seed user screening method, and product user influence evaluation method and device | |
| CN103207835A (en) | Mass data storage method through self-adaptive Range partitions | |
| CN104077713A (en) | Method and system of demand assessment and advertisement push for user visiting website | |
| CN109660623A (en) | A kind of distribution method, device and the computer readable storage medium of cloud service resource | |
| CN111222757A (en) | Statistical method and device for use condition of charging pile | |
| Ma et al. | Does better Internet access lead to more adoption? A new empirical study using household relocation | |
| Wang et al. | A nonlinear collective credit allocation in scientific publications | |
| EP2633450A1 (en) | Systems and methods for scheduling changes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1185968 Country of ref document: HK |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: GR Ref document number: 1185968 Country of ref document: HK |