CN111538653B - Method and device for testing scheme, electronic equipment and storage medium - Google Patents
Method and device for testing scheme, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111538653B CN111538653B CN202010266793.7A CN202010266793A CN111538653B CN 111538653 B CN111538653 B CN 111538653B CN 202010266793 A CN202010266793 A CN 202010266793A CN 111538653 B CN111538653 B CN 111538653B
- Authority
- CN
- China
- Prior art keywords
- scheme
- statistic
- schemes
- distribution
- users
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3692—Test management for test results analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Complex Calculations (AREA)
- Debugging And Monitoring (AREA)
Abstract
The disclosure relates to a method, an apparatus, an electronic device and a storage medium for testing a scheme, wherein the method comprises the following steps: determining probability distribution of n operation modes contained in the j-th operation information; sampling the n operation modes according to the probability distribution to determine the frequency distribution of the times of extracting each operation mode; calculating a statistical value according to the frequency distribution; repeating the steps to obtain K statistical values to form a j-th statistical value sequence; calculating K difference values of corresponding statistic values in the j-th statistic value sequence and the j-th statistic value sequence; and determining the probability that the test effect of the j scheme is better than that of the j' scheme according to the numbers D and K of the differences larger than 0 in the K differences. According to the embodiment of the disclosure, the number of times distribution of the operation modes is obtained by sampling the n operation modes based on the probability distribution of the n operation modes, so that the number of times distribution is more consistent with the probability distribution of each operation mode, and the method is suitable for scenes with a large number of samples.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a method and apparatus for testing a scheme, an electronic device, and a storage medium.
Background
a/B testing is currently one way to compare the merits between two or more schemes, for example, two schemes a and B may be tested, scheme a being assigned to user group α and scheme B being assigned to user group β. For example, if the schemes a and B specifically display pages in different documents on the user's terminal, it is possible to detect that the user clicking on the page displayed in the scheme a in the user group α occupies the proportion x of all users in the user group α, and detect that the user clicking on the page displayed in the scheme B in the user group B occupies the proportion y of all users in the user group β.
It can then be determined which solution is better based on the ratio x and the ratio y, e.g. the ratio x is greater than the ratio y, then the text generally describing solution a is more attractive to the user, i.e. solution a is better than solution B. However, there is a certain uncertainty in this determination, and it is possible that the ratio x is larger than the ratio y only once and accidentally.
To quantify this uncertainty, it is not straightforward to determine which scheme is better, but rather a probability needs to be calculated, for example, for the case of scheme a above being better than scheme B, in the related art, the probability of scheme a being better than scheme B may be calculated by both a formula method based on statistical theory and a self-service (Bootstrap) sampling method.
The advantage of the formula method is that the calculation speed is high, but the formula method is only suitable for scenes with large user quantity and dense user behaviors (such as a large proportion of users in the user group alpha click on a page displayed by the scheme A); however, the self-service method is often used in isolation in the related art for sampling, which makes it difficult to adapt to scenes with large user quantity and high calculation complexity.
Disclosure of Invention
The present disclosure provides a testing method, apparatus, electronic device, and storage medium for solutions to at least the technical problems in the related art. The technical scheme of the present disclosure is as follows:
according to a first aspect of an embodiment of the present disclosure, a test method of a solution is provided, including:
acquiring operation information of m groups of users for operating m schemes, wherein j groups of users in the m groups of users operate j schemes in the m schemes, j is more than or equal to 1 and less than or equal to m, and m is more than 1;
determining n operation modes contained in j operation information of the j group of users for operating the j scheme and probability distribution of the n operation modes;
step A, sampling the n operation modes according to the probability distribution to determine the frequency distribution of the times of extracting each operation mode;
step B, calculating a statistical value according to the frequency distribution;
repeating the step A and the step B K times to obtain a j-th statistic value sequence corresponding to the j-th scheme, wherein the j-th statistic value sequence is formed by K statistic values;
calculating the difference value of the corresponding statistic value in the j 'statistic value sequence corresponding to the j' scheme in the m schemes to obtain K difference values, wherein j 'is not less than 1 and not more than m, and j' is not less than j;
and determining the probability that the test effect of the j scheme is better than that of the j' scheme according to the ratio of the number D to the number K of the difference values larger than 0 in the K difference values.
Optionally, the method further comprises:
the K differences are arranged from small to large to obtain a difference sequence;
determining a first quantile and a second quantile according to a preset confidence;
and determining the upper limit and the lower limit of a confidence interval in the difference sequence according to the first quantile and the second quantile.
Optionally, the method further comprises:
generating a histogram according to the distribution of each statistic in the j-th statistic sequence;
and displaying a histogram corresponding to each of the m statistic sequences.
Optionally, the obtaining operation information of the m groups of users to operate the m schemes includes:
and acquiring operation information of m groups of users for operating m schemes from the column storage database.
Optionally, the statistics include, but are not limited to, one of:
average, sum, cost performance.
According to a first aspect of an embodiment of the present disclosure, a test apparatus of an aspect is provided, including:
an operation information acquisition module configured to perform acquisition of operation information of m groups of users operating m schemes, wherein j-th group of users in the m groups of users operate j-th scheme in the m schemes, j is not less than 1 and not more than m, and m is more than 1;
a probability distribution determining module configured to perform determining n kinds of operation modes included in j-th operation information of the j-th group user operating the j-th scheme, and probability distribution of the n kinds of operation modes;
a number of times distribution determining module configured to perform sampling of the n operation modes according to the probability distribution to determine a number of times distribution of the number of times each of the operation modes is extracted;
a statistics calculation module configured to perform a calculation of statistics from the frequency distribution;
the number distribution determining module and the statistic value calculating module are further configured to execute respective actions repeatedly for K times to obtain a j-th statistic value sequence formed by K statistic values and corresponding to the j-th scheme;
a difference value calculating module configured to perform calculation of difference values of corresponding statistic values in a j 'th statistic value sequence corresponding to a j' th statistic value sequence in the m schemes to obtain K difference values, wherein j 'is equal to or less than 1 and is equal to or less than m, and j' is equal to or less than j;
and the probability determining module is configured to determine the probability that the test effect of the j scheme is better than the test effect of the j' scheme according to the ratio of the number D of the difference value greater than 0 to K in the K difference values.
Optionally, the apparatus further comprises:
the sequence determining module is configured to perform the steps of arranging the K differences from small to large to obtain a difference sequence;
a quantile determination module configured to perform determining a first quantile and a second quantile according to a preset confidence;
a confidence interval determination module configured to perform determining an upper bound and a lower bound of a confidence interval in the difference sequence from the first quantile and the second quantile.
Optionally, the apparatus further comprises:
a histogram generation module configured to perform generation of a histogram from a distribution of each statistic in the j-th statistic sequence;
and the histogram display module is configured to display a histogram corresponding to each of the m statistic value sequences.
Optionally, the operation information obtaining module is configured to obtain operation information of m groups of users operating m schemes from a columnar storage database.
Optionally, the statistics include, but are not limited to, one of:
average, sum, cost performance.
According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, including:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the test method of the solution as described in any of the embodiments above.
According to a fourth aspect of the embodiments of the present disclosure, a storage medium is provided, which when executed by a processor of an electronic device, enables the electronic device to perform the test method of the solution described in any of the embodiments above.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product configured to perform the test method of the solution described in any of the embodiments above.
According to the embodiment of the disclosure, the frequency distribution of the operation modes is obtained by sampling the n operation modes based on the probability distribution of the n operation modes, so that the frequency distribution is more consistent with the probability distribution of each operation mode, and is suitable for a scene of a large number of samples.
Furthermore, according to the statistics value calculated by the frequency distribution, because the specific content of the statistics value can be set according to the required indexes, indexes of different schemes can be compared by comparing the statistics values, and compared with the statistics value calculated by a formula method in the related technology, the statistics value calculated by the embodiment can be according to pertinence, thereby being beneficial to reducing the calculated amount.
And a plurality of statistical values can be obtained based on the frequency distribution calculation obtained by sampling, and then when comparing the advantages and disadvantages of the two scheme test effects, the difference value of the corresponding statistical values in the plurality of statistical values can be calculated, and then the probability of the good and poor relation of the two scheme test effects is represented by the ratio of the number of the difference values which are larger than 0 in the plurality of difference values to the total number of the difference values, rather than directly obtaining the conclusion of the good and poor relation of the two scheme test effects, thereby being beneficial to reflecting the good and poor relation of the two scheme test effects more accurately.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.
Fig. 1 is a schematic flow chart of a test method of one scenario shown in accordance with an embodiment of the present disclosure.
Fig. 2 is a schematic flow chart of a test method of another scenario illustrated in accordance with an embodiment of the present disclosure.
Fig. 3 is a schematic flow chart of a test method of yet another aspect shown in accordance with an embodiment of the present disclosure.
Fig. 4 is a schematic diagram of a histogram corresponding to a sequence of statistics shown in accordance with an embodiment of the present disclosure.
Fig. 5 is a schematic flow chart diagram of a test method of yet another aspect shown in accordance with an embodiment of the present disclosure.
Fig. 6 is a schematic block diagram of a test apparatus according to one aspect shown in an embodiment of the present disclosure.
Fig. 7 is a schematic block diagram of a testing device according to another aspect shown in an embodiment of the present disclosure.
Fig. 8 is a schematic block diagram of a test apparatus according to yet another aspect shown in an embodiment of the present disclosure.
Fig. 9 is a hardware configuration diagram of a device in which a test apparatus of the scheme shown in the embodiment of the present disclosure is located.
Detailed Description
In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
Fig. 1 is a schematic flow chart of a test method of one scenario shown in accordance with an embodiment of the present disclosure. The method shown in the embodiment can be applied to a server, and the server can be a server of an application program on a terminal such as a mobile phone, a computer and the like.
In one embodiment, a tester may formulate m schemes and send the m schemes to m groups of users (e.g., to terminals of the users) through a server, where each group of users may include one or more users and each group of users includes different users, and the server may send a j-th scheme of the m schemes to a j-th group of users of the m groups of users, so that each group of users is assigned to a different scheme, and thus the users may operate the schemes, the server may collect operation information of the users on the scheme operation, and store the collected operation information in a database.
The manner in which a user operates a scheme may vary based on the scheme, and there may be multiple manners of operation for one scheme. For example, the scheme specifically displays the page in different documents on the terminal of the user, and the operation mode of the scheme by the user includes, but is not limited to, clicking the page, scrolling the page, refreshing the page, and the like.
As shown in fig. 1, the test method of the scheme may include the steps of:
in step S101, operation information of m groups of users for operating m schemes is obtained, wherein j groups of users in the m groups of users operate j schemes in the m schemes, j is more than or equal to 1 and less than or equal to m, and m is more than 1;
in step S102, determining n operation modes included in the j operation information of the j user for operating the j scheme and probability distribution of the n operation modes;
in step a (hereinafter referred to as step S103), the n operation modes are sampled in accordance with the probability distribution to determine a distribution of the number of times each of the operation modes is extracted;
in step B (hereinafter referred to as step S104), a statistical value is calculated from the frequency distribution;
in step S105, repeating the step S103 and the step S104K times to obtain a j-th statistic sequence corresponding to the j-th scheme, where the j-th statistic sequence is formed by K statistic values;
in step S106, calculating the difference between the jth statistic sequence and the corresponding statistic in the jth statistic sequence corresponding to the jth statistic in the mth scheme to obtain K differences, where 1 is less than or equal to j 'is less than or equal to m, and j' is not equal to j;
in step S107, a probability that the test effect of the jth scheme is better than the test effect of the jth' scheme is determined according to the ratio of the number D of differences greater than 0 to K of the K differences.
In one embodiment, operation information of m groups of users operating m schemes may be obtained from a database, and then n operation modes contained in j operation information of j groups of users operating the j schemes and probability distribution of the n operation modes are determined.
Since the jth scheme is allocated to the jth group of users, the jth group of users can operate the jth scheme, and the operation modes for operating the jth scheme can be n, that is, the jth operation information can include n operation modes, each user can perform one or more operations on the scheme, and taking one operation mode of each user on the scheme as an example, for each operation mode, the probability distribution F thereof can be determined as follows:
wherein a is 1 To a n Representing n modes of operation, p 1 To p n Representing the probability of occurrence of each of n operating modes, p i Represents an ith operation mode a of n operation modes i Probability of occurrence of 1.ltoreq.i.ltoreq.n, p 1 To p n A non-negative number less than 1, and p 1 To p n The sum of (2) is 1.
Taking 3 operation modes including a click page, a scroll page and a refresh page as examples, in which 20 ten thousand users perform the operation of refreshing the page, 30 ten thousand users perform the operation of scrolling the page, and 50 ten thousand users perform the operation of clicking the page, it is determined that the probability of occurrence of the operation mode of clicking the page is 20%, the probability of occurrence of the operation mode of scrolling the page is 30% and the probability of occurrence of the operation mode of clicking the page is 50%.
Then n operation modes can be taken as samples, the n operation modes are sampled according to the probability distribution, for example, the operation modes are sampled by a self-service method, so as to determine the frequency distribution of the times each operation mode is extracted, namely, the operation modes are sampled by a self-service method according to the probability p 1 For operation mode a 1 Sampling according to probability p 2 For operation mode a 2 Sampling … according to probability p n For operation mode a n Sampling is performed, so that the number of times each operation mode is extracted can be determined, and the number distribution is formed according to the number of timesThe following are provided:
wherein k is 1 To k n Represents the number of times each of n operating modes is extracted, k i Represents an ith operation mode a of n operation modes i Number of times of being extracted, k 1 To k n Being non-negative, e.g.The total number of samples is N, then k 1 To k n The sum of (2) is equal to N.
The statistics may then be calculated from the distribution of times, the statistics being different according to the desired index (e.g. calculation caliber), e.g. the desired index is the mean value, thenStatistics of +.>Is the mean value (I)>For example, the desired indicator is the sum, then +.>Statistics of +.>Is the sum of->
It should be noted that the calculation statistics need to be calculated from the frequency distribution, but may be calculated not only from the frequency distribution but also from the frequency distribution and other information.
Due to frequency distributionObtained by sampling only once, the accuracy of which is not sufficiently high, then distributed according to the number of timesThe statistics obtained->The accuracy is not very high, and the embodiment can repeat K (the value can be set according to the needs, such as 5000, 10000, 20000, etc.)Sub-steps S103 and S104, whereby K statistics can be obtained +.>To->These K statistics can then be formed into a j-th statistics sequence corresponding to a j-th scheme +.>Since the jth scheme is any one of the m schemes, not only the jth statistical value sequence corresponding to the jth scheme but also the statistical value sequence corresponding to each of the m schemes can be calculated according to the above steps.
And comparing the test effect of any two schemes in m schemes according to the calculated statistical value sequence, for example, one scheme is the j scheme, and the statistical value sequence is T j Another scheme is the j' th scheme, wherein the statistical value sequence is T j' The differences of the corresponding (i-identical) statistics in the j-th and j-th' statistics sequences can be calculated to obtain K differences, i.e. calculated
Since the statistics can characterize the index of a certain aspect of the scheme (such as the mean, the sum, etc.), the difference between the statistics of the two schemes can determine the merits of the index of the two schemes, for example, a difference greater than 0 indicates that the test effect of the index of the j scheme in a certain aspect is better than that of the j scheme, and the present embodiment comprehensively considers K differences due to the larger uncertainty of the individual differences, namely for d 1 To d K The K differences can be determined to be the ratio of D to K, i.e. K is 10000, 8000 differences greater than 0 are present, the resulting ratio of D to K is 80%, thus determining that the j-th scheme has a 80% probability of being better than the j-th schemeTest effect of j' protocols.
The j scheme and the j' scheme are any two schemes in the m schemes, so that the test effects of any two schemes in the m schemes can be compared according to the mode, and the scheme with the optimal test effect in the m schemes can be determined.
According to the embodiment of the disclosure, the number of times distribution of the operation modes is obtained by sampling the n operation modes by self-help method based on the probability distribution of the n operation modes instead of sampling the operation modes independently, for example, sampling by self-help method, so that the number of times distribution is more consistent with the probability distribution of each operation mode, and is suitable for a scene of a large number of samples.
Furthermore, according to the statistics value calculated by the frequency distribution, because the specific content of the statistics value can be set according to the required indexes, indexes of different schemes can be compared by comparing the statistics values, and compared with the statistics value calculated by a formula method in the related technology, the statistics value calculated by the embodiment can be according to pertinence, thereby being beneficial to reducing the calculated amount.
And a plurality of statistical values can be obtained based on the frequency distribution calculation obtained by sampling by a self-service method, and then when comparing the advantages and disadvantages of the two scheme test effects, the difference value of the corresponding statistical values in the plurality of statistical values can be calculated, and then the probability of the good and poor relation of the two scheme test effects is represented by the ratio of the number of the difference values which are larger than 0 in the plurality of difference values to the total number of the difference values, rather than directly obtaining the conclusion of the good and poor relation of the two scheme test effects, thereby being beneficial to reflecting the good and poor relation of the two scheme test effects more accurately.
Fig. 2 is a schematic flow chart of a test method of another scenario illustrated in accordance with an embodiment of the present disclosure. As shown in fig. 2, the method further includes:
in step S108, the K differences are arranged from small to large to obtain a difference sequence;
in step S109, determining a first quantile and a second quantile according to a preset confidence;
in step S110, an upper limit and a lower limit of a confidence interval are determined in the difference sequence according to the first quantile and the second quantile.
In one embodiment, the confidence interval may be further determined based on the probability that the test effect of the jth scenario is better than the test effect of the jth' scenario.
The K differences may be arranged from small to large to obtain a difference sequence, and then the first quantile and the second quantile may be determined according to a preset confidence, where the preset confidence may be set according to needs, and the difference between the second quantile and the first quantile is equal to the preset confidence, and generally 0.5 may be selected as a midpoint of the confidence interval, and 0.95 may be used as a confidence, where the first quantile and the second quantile may be 0.025 and 0.975.
And then, the upper limit and the lower limit of the confidence interval can be determined in the difference sequence according to the first quantile and the second quantile, namely, the difference value of the corresponding quantile 0.025 is selected as the lower limit of the confidence interval in the difference sequence, and the difference value of the corresponding quantile 0.975 is selected as the upper limit of the confidence interval. For example, the sequence of differences includes 10000 differences, then a lower limit may be selected in which the 250 th difference is the confidence interval and an upper limit may be selected in which the 9750 th difference is the confidence interval.
Accordingly, not only the probability that the test effect of the jth scheme is better than the test effect of the jth scheme can be determined, but also the confidence interval can be determined, and the test effect of the jth scheme is better than the test effect of the jth scheme can be quantized, and the average value of the K difference values can be calculated to represent the test effect.
For the calculated probabilities, and upper and lower confidence interval limits, a display may be made for the tester to view.
Fig. 3 is a schematic flow chart of a test method of yet another aspect shown in accordance with an embodiment of the present disclosure. As shown in fig. 3, the method further includes:
in step S111, generating a histogram according to the distribution of each statistic in the jth statistic sequence;
in step S112, a histogram corresponding to each of the m statistic sequences is displayed.
The step S111 and the step S112 may be performed after the step S107 as shown in fig. 3, or the order of execution may be adjusted as needed, and it is only necessary to ensure that the steps are performed after the step S105.
In one embodiment, since the statistics of different times in the sequence of statistics may be the same, and the number of the statistics formed by several pages for K statistics may be smaller than K, then a histogram may be generated according to the distribution of each statistics (i.e., different statistics), so as to display the histograms corresponding to each of the m sequences of statistics, so that the tester intuitively compares the differences of the test effects of each of the m schemes according to the histograms.
For example, for a video creation website, a video creator may be motivated to promote more active creation of the video by the creator by subsidizing exposure. However, the exposure of the patch is not as much as better, and the cost performance is required to be considered, and the cost performance can be calculated according to the frequency distribution and the exposure of the patch in the above embodiment, for example, the operation modes in the frequency distribution include n kinds, a 1 Representing that the creator authored 0 videos, a 2 Representing that the creator authored 1 video, …, a n The method indicates that an creator creates n-1 videos, the cost performance can be set to be positively related to the total amount of the created videos, the cost performance can be inversely related to the total amount of the exposure of the subsidy, and a specific calculation formula can be set according to requirements.
For example, a total of 4 schemes of 3 patch schemes and the original scheme are designed, the original scheme θ is to patch exposure not for the creator (as 1 st scheme), the scheme α is to patch 50 exposures for the creator (as 2 nd scheme), the scheme β is to patch 100 exposures for the creator (as 3 rd scheme), and the scheme γ is to patch 200 exposures for the creator (as 4 th scheme).
Then to compare the cost performance of these 4 schemes, 4 groups of users (creators) can be selected, each group comprising 200 tens of thousands of users, and then 4 schemes are assigned to the 4 groups of users. For example, according to steps S101 to S103 in the above embodiment, the distribution of the number of times corresponding to the scheme α can be obtained as follows:
then, according to the obtained frequency distribution, statistics values can be further calculated, and K times of step S103 and step S104 are repeated to obtain a jth statistics value sequence corresponding to a jth scheme formed by K statistics values, and then a histogram is generated according to the distribution of each statistics value in the jth statistics value sequence, wherein the abscissa of the histogram can be the statistics value, that is, the cost performance, and the ordinate can be the number of each cost performance, and then the histogram corresponding to each statistics value sequence in m statistics value sequences, for example, the histogram corresponding to each statistics value sequence in 4 statistics value sequences corresponding to 4 schemes θ, α, β and γ is displayed as shown in fig. 4.
Although the probability that the test effect of one scheme is better than the test effect of the other scheme can be calculated between every two schemes in the 4 schemes according to the steps S101 to S106, the difference of the test effects between the two schemes is represented by the probability, and no image is intuitive for a tester; besides the probability, the average value of the corresponding statistical value sequence of each party can be calculated for the tester to refer to, for example, the average value of the 1 st statistical value sequence corresponding to the scheme theta is 0.13328, the average value of the 2 nd statistical value sequence corresponding to the scheme alpha is 0.13326, the average value of the 3 rd statistical value sequence corresponding to the scheme beta is 0.13349, the average value of the 4 th statistical value sequence corresponding to the scheme gamma is 0.13352, so that the difference between the average values is very small, and according to a single numerical value, the tester is difficult to judge whether the tester can reflect the obvious effect of the scheme test or not, and the test is only randomly fluctuated.
By displaying the histogram corresponding to each of the m statistic sequences, for example, the histogram shown in fig. 4, the user can intuitively determine the difference of the test effect of each solution by observing the histogram, for example, the average value of the statistic sequence corresponding to the 3 rd solution is smaller than the average value of the statistic sequence corresponding to the 4 th solution, but the tester can know through observing fig. 4 that the test effect of the solution β is optimal because the number of occurrences of the individual statistic in the solution 4 is more, and the overall statistic sequence is still the cost performance of the 3 rd solution as a whole is higher than the cost performance of the 4 th solution as a whole, so that the tester can determine that the cost performance of the solution β as a whole is higher than the cost performance of the solution γ as a whole, and is also higher than the cost performance of the solution α and the solution θ as a whole based on observing fig. 4.
Fig. 5 is a schematic flow chart diagram of a test method of yet another aspect shown in accordance with an embodiment of the present disclosure. As shown in fig. 5, the obtaining operation information of m groups of users to operate m schemes includes:
in step S1011, operation information of m groups of users operating m schemes is acquired from the columnar storage database.
In one embodiment, after m schemes are allocated to m groups of users, operation information of the user on the scheme operation may be collected, and for the collected operation information, the collected operation information may be stored in a column-type storage database (for example, a clickhouse database), and further, the operation information of the m groups of users on the m schemes may be obtained from the column-type storage database, and due to the characteristics of the column-type storage database storing data, a larger data amount (for example, hundreds of millions of pieces) of data may be stored relative to other types of databases (for example, a line-type storage database), and the data may be obtained therefrom more quickly.
Optionally, the statistics include, but are not limited to, one of:
average, sum, cost performance.
The present disclosure also proposes embodiments of the test apparatus of the solution, corresponding to the embodiments of the test method of the solution described above.
Fig. 6 is a schematic block diagram of a test apparatus according to one aspect shown in an embodiment of the present disclosure. The device shown in this embodiment may be suitable for a server, where the server may be a server for an application program on a terminal such as a mobile phone, a computer, or the like.
As shown in fig. 6, the test device of the scheme may include:
an operation information acquisition module 1 configured to perform acquisition of operation information of m groups of users operating m schemes, wherein j-th group of users in the m groups of users operate j-th scheme in the m schemes, j is not less than 1 and not more than m, and m is more than 1;
a probability distribution determining module 2 configured to perform determination of n kinds of operation manners included in j-th operation information of the j-th group user for operating the j-th scheme, and probability distribution of the n kinds of operation manners;
a number distribution determining module 3 configured to perform sampling of the n operation modes in accordance with the probability distribution to determine a number distribution of times each of the operation modes is extracted;
a statistics calculation module 4 configured to perform a calculation of statistics from the frequency distribution;
the number distribution determining module and the statistic value calculating module are further configured to execute respective actions repeatedly for K times to obtain a j-th statistic value sequence formed by K statistic values and corresponding to the j-th scheme;
a difference calculating module 5 configured to perform calculation of differences between the j-th statistic sequence and corresponding statistics in the j '-th statistic sequence corresponding to the j' -th scheme in the m schemes to obtain K differences, wherein 1 is less than or equal to j 'is less than or equal to m, and j' is not equal to j;
a probability determination module 6 configured to determine a probability that the test effect of the j-th scheme is better than the test effect of the j-th scheme according to a ratio of a number D of differences greater than 0 to K of the K differences.
Fig. 7 is a schematic block diagram of a testing device according to another aspect shown in an embodiment of the present disclosure. As shown in fig. 7, the apparatus further includes:
a sequence determining module 7 configured to perform the order of the K differences from small to large to obtain a difference sequence;
a quantile determination module 8 configured to perform determining a first quantile and a second quantile according to a preset confidence;
a confidence interval determination module 9 configured to perform a determination of an upper and a lower limit of a confidence interval in the difference sequence from the first and the second quantile.
Fig. 8 is a schematic block diagram of a test apparatus according to yet another aspect shown in an embodiment of the present disclosure. As shown in fig. 8, the apparatus further includes:
a histogram generation module 10 configured to perform generation of a histogram from the distribution of each statistic in the j-th statistic sequence;
the histogram display module 11 is configured to perform displaying a histogram corresponding to each of the m statistic sequences.
Optionally, the operation information obtaining module is configured to obtain operation information of m groups of users operating m schemes from a columnar storage database.
Optionally, the statistics include, but are not limited to, one of:
average, sum, cost performance.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
The embodiment of the test device of the scheme shown in the embodiment of the disclosure can be applied to a server and other devices. The apparatus embodiments may be implemented by software, or may be implemented by hardware or a combination of hardware and software. Taking a software implementation as an example, the device in a logic sense is formed by reading corresponding computer program instructions in a nonvolatile memory into a memory by a processor of a device where the device is located for operation. In terms of hardware, as shown in fig. 9, a hardware structure diagram of a device where a testing apparatus according to an embodiment of the present disclosure is located is shown, where in addition to a processor, a network interface, a memory, and a nonvolatile memory shown in fig. 9, the device where the embodiment is located may generally include other hardware, such as a forwarding chip responsible for processing a packet, and so on; the device may also be a distributed device in terms of hardware architecture, possibly comprising a plurality of interface cards, for the extension of the message processing at the hardware level.
The embodiment of the disclosure also proposes an electronic device, including:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the test method of the solution described in any of the embodiments above.
Embodiments of the present disclosure also propose a storage medium, which when executed by a processor of an electronic device, enables the electronic device to perform the test method of the solution described in any of the above embodiments.
Embodiments of the present disclosure also propose a computer program product configured to perform the test method of the solution described in any of the above embodiments.
Alternatively, the storage medium may be a non-transitory computer readable storage medium, which may be, for example, ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.
Embodiments of the present disclosure also propose a computer program product configured to perform the test method of the solution described in any of the above embodiments.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing has outlined the detailed description of the method and apparatus provided by the embodiments of the present disclosure, and the detailed description of the principles and embodiments of the present disclosure has been provided herein with the application of the specific examples, the above examples being provided only to facilitate the understanding of the method of the present disclosure and its core ideas; meanwhile, as one of ordinary skill in the art will have variations in the detailed description and the application scope in light of the ideas of the present disclosure, the present disclosure should not be construed as being limited to the above description.
Claims (10)
1. A method for testing a solution, comprising:
acquiring operation information of m groups of users for operating m schemes, wherein j groups of users in the m groups of users operate j schemes in the m schemes, j is more than or equal to 1 and less than or equal to m, and m is more than 1;
determining n operation modes contained in j operation information of the j group of users for operating the j scheme and probability distribution of the n operation modes;
step A, sampling the n operation modes according to the probability distribution to determine the frequency distribution of the times of extracting each operation mode;
step B, calculating a statistical value according to the frequency distribution;
repeating the step A and the step B K times to obtain a j-th statistic value sequence corresponding to the j-th scheme, wherein the j-th statistic value sequence is formed by K statistic values;
calculating the difference value of the corresponding statistic value in the j 'statistic value sequence corresponding to the j' scheme in the m schemes to obtain K difference values, wherein j 'is not less than 1 and not more than m, and j' is not less than j;
and determining the probability that the test effect of the j scheme is better than that of the j' scheme according to the ratio of the number D to the number K of the difference values larger than 0 in the K difference values.
2. The method according to claim 1, wherein the method further comprises:
the K differences are arranged from small to large to obtain a difference sequence;
determining a first quantile and a second quantile according to a preset confidence;
and determining the upper limit and the lower limit of a confidence interval in the difference sequence according to the first quantile and the second quantile.
3. The method according to claim 1, wherein the method further comprises:
generating a histogram according to the distribution of each statistic in the j-th statistic sequence;
and displaying a histogram corresponding to each of the m statistic sequences.
4. A method according to any one of claims 1 to 3, wherein the obtaining operation information of m groups of users operating m schemes comprises:
and acquiring operation information of m groups of users for operating m schemes from the column storage database.
5. A test device according to one aspect, comprising:
an operation information acquisition module configured to perform acquisition of operation information of m groups of users operating m schemes, wherein j-th group of users in the m groups of users operate j-th scheme in the m schemes, j is not less than 1 and not more than m, and m is more than 1;
a probability distribution determining module configured to perform determining n kinds of operation modes included in j-th operation information of the j-th group user operating the j-th scheme, and probability distribution of the n kinds of operation modes;
a number of times distribution determining module configured to perform sampling of the n operation modes according to the probability distribution to determine a number of times distribution of the number of times each of the operation modes is extracted;
a statistics calculation module configured to perform a calculation of statistics from the frequency distribution;
the number distribution determining module and the statistic value calculating module are further configured to execute respective actions repeatedly for K times to obtain a j-th statistic value sequence formed by K statistic values and corresponding to the j-th scheme;
a difference value calculating module configured to perform calculation of difference values of corresponding statistic values in a j 'th statistic value sequence corresponding to a j' th statistic value sequence in the m schemes to obtain K difference values, wherein j 'is equal to or less than 1 and is equal to or less than m, and j' is equal to or less than j;
and the probability determining module is configured to determine the probability that the test effect of the j scheme is better than the test effect of the j' scheme according to the ratio of the number D of the difference value greater than 0 to K in the K difference values.
6. The apparatus of claim 5, wherein the apparatus further comprises:
the sequence determining module is configured to perform the steps of arranging the K differences from small to large to obtain a difference sequence;
a quantile determination module configured to perform determining a first quantile and a second quantile according to a preset confidence;
a confidence interval determination module configured to perform determining an upper bound and a lower bound of a confidence interval in the difference sequence from the first quantile and the second quantile.
7. The apparatus of claim 5, wherein the apparatus further comprises:
a histogram generation module configured to perform generation of a histogram from a distribution of each statistic in the j-th statistic sequence;
and the histogram display module is configured to display a histogram corresponding to each of the m statistic value sequences.
8. The apparatus according to any one of claims 5 to 7, wherein the operation information acquisition module is configured to perform acquisition of operation information of m groups of users operating m schemes from a columnar storage database.
9. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the test method of the solution according to any one of claims 1 to 4.
10. A storage medium, characterized in that instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the test method of the solution according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010266793.7A CN111538653B (en) | 2020-04-07 | 2020-04-07 | Method and device for testing scheme, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010266793.7A CN111538653B (en) | 2020-04-07 | 2020-04-07 | Method and device for testing scheme, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111538653A CN111538653A (en) | 2020-08-14 |
| CN111538653B true CN111538653B (en) | 2023-08-18 |
Family
ID=71978562
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010266793.7A Active CN111538653B (en) | 2020-04-07 | 2020-04-07 | Method and device for testing scheme, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111538653B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273290A (en) * | 2017-06-13 | 2017-10-20 | 北京奇艺世纪科技有限公司 | The A/B method of testings and device of a kind of Page Service |
| CN109308255A (en) * | 2017-07-28 | 2019-02-05 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus for A/B test experiments |
| CN109614318A (en) * | 2018-11-14 | 2019-04-12 | 金色熊猫有限公司 | Automated testing method, device, electronic equipment and computer-readable medium |
| CN109669875A (en) * | 2018-12-28 | 2019-04-23 | 北京金山安全软件有限公司 | Application crash test method and device, electronic equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI571737B (en) * | 2015-11-18 | 2017-02-21 | 財團法人資訊工業策進會 | Software test system, method and non-transitory computer readable medium of the same |
-
2020
- 2020-04-07 CN CN202010266793.7A patent/CN111538653B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273290A (en) * | 2017-06-13 | 2017-10-20 | 北京奇艺世纪科技有限公司 | The A/B method of testings and device of a kind of Page Service |
| CN109308255A (en) * | 2017-07-28 | 2019-02-05 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus for A/B test experiments |
| CN109614318A (en) * | 2018-11-14 | 2019-04-12 | 金色熊猫有限公司 | Automated testing method, device, electronic equipment and computer-readable medium |
| CN109669875A (en) * | 2018-12-28 | 2019-04-23 | 北京金山安全软件有限公司 | Application crash test method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111538653A (en) | 2020-08-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109241461B (en) | User portrait construction method and device | |
| CN109299271B (en) | Training sample generation method, text data method, public opinion event classification method and related equipment | |
| CN113099475A (en) | Network quality detection method and device, electronic equipment and readable storage medium | |
| CN114490375A (en) | Method, device and equipment for testing performance of application program and storage medium | |
| CN113485931A (en) | Test method, test device, electronic equipment and computer readable storage medium | |
| US20180061264A1 (en) | Self-learning surveys for open-ended analysis | |
| CN111538653B (en) | Method and device for testing scheme, electronic equipment and storage medium | |
| CN107332681A (en) | A kind of failure dimensional analysis method and the network equipment | |
| CN112162905A (en) | Log processing method and device, electronic equipment and storage medium | |
| CN115757973A (en) | Sorting method, device and equipment of recommendation system and readable storage medium | |
| CN112256552B (en) | Pressure testing method and device for map engine | |
| CN112291625B (en) | Information quality processing method, information quality processing device, electronic equipment and storage medium | |
| CN105653645B (en) | Network information attention degree evaluation method and device | |
| CN107885649A (en) | The performance test methods and device of IOS application programs | |
| CN116187299B (en) | Scientific and technological project text data verification and evaluation method, system and medium | |
| CN110337015B (en) | Method for correcting audience rating error of cable television user under large sample | |
| CN115509909B (en) | Test method, test device, electronic equipment and storage medium | |
| CN111752984B (en) | Information processing method, device and storage medium | |
| CN115146084B (en) | Method and device for acquiring equipment fault and maintenance data from unstructured data | |
| CN117667691A (en) | Interface testing method, device and system | |
| CN111309623B (en) | Coordinate class data classification test method and device | |
| CN114417003A (en) | Call bill common contact person analysis method, device and storage medium based on knowledge graph technology | |
| CN115293816A (en) | User portrait display method and device, electronic equipment and storage medium | |
| CN111414397A (en) | Alarm event fault positioning method and device, computer equipment and storage medium | |
| CN115795120A (en) | User portrait information verification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |