CN114328166A

CN114328166A - AB test algorithm performance information acquisition method and device and storage medium

Info

Publication number: CN114328166A
Application number: CN202011063834.9A
Authority: CN
Inventors: 赵一飞; 欧明栋; 赵亮; 印卧涛; 许欢
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2020-09-30
Filing date: 2020-09-30
Publication date: 2022-04-12

Abstract

The embodiment of the disclosure relates to a method and a device for acquiring performance information of an AB test algorithm, electronic equipment and a computer storage medium, comprising the following steps: acquiring a sample acquisition instruction, and acquiring sample data of an AB test based on the sample acquisition instruction, wherein the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme; acquiring an algorithm calling instruction, and calling at least two AB test algorithms based on the algorithm calling instruction; and respectively carrying out AB test based on the sample data and the at least two called AB test algorithms, and determining the performance information of the at least two AB test algorithms during the AB test. According to the technical scheme provided by the embodiment of the disclosure, the performance information of the AB test algorithm can be obtained, and the performance of the selected AB test algorithm on the performances such as test efficiency is more excellent.

Description

AB test algorithm performance information acquisition method and device and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for acquiring performance information of an AB test algorithm, an AB test method and an apparatus, an AB test system, an electronic device, and a computer storage medium.

Background

The AB test has been used for testing the drug treatment effect in the traditional medical field, and is gradually applied to the current Internet field for testing the version effect of algorithm online and product page revising. For example, when two (a/B) or more (a/B/n) schemes are made for a Web or App interface, it is necessary to randomly access the two or more test schemes by user groups having the same or similar components in the same time dimension, collect user experience data and/or business data of each user group when using each test scheme, and finally evaluate and analyze a version with a better effect.

In the AB test process, the quality of each test scheme is obtained by evaluating and analyzing the AB test algorithm, and to a certain extent, the performance of the AB test algorithm determines the test efficiency, the time cost and the like when performing the AB test. However, the AB test algorithms available in the prior art include various algorithm types, and the performance of the AB test algorithms cannot be effectively tested at present.

Disclosure of Invention

To solve the above technical problem or at least partially solve the above technical problem, the present disclosure provides a performance acquisition method and apparatus for an AB test algorithm, an AB test method and apparatus, an AB test system, an electronic device, and a computer storage medium.

The embodiment of the disclosure provides a method for acquiring performance information of an AB test algorithm, which comprises the following steps:

acquiring a sample acquisition instruction, and acquiring sample data of an AB test based on the sample acquisition instruction, wherein the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme;

acquiring an algorithm calling instruction, and calling at least two AB test algorithms based on the algorithm calling instruction;

and respectively carrying out AB test based on the sample data and the at least two called AB test algorithms, and determining the performance information of the at least two AB test algorithms during the AB test.

Optionally, the determining the performance information of the at least two AB test algorithms during the AB test includes:

obtaining test result data when the at least two AB test algorithms are used for AB test;

determining performance information of the at least two AB test algorithms based on the test result data.

Optionally, the test result data includes the number of samples, and the obtaining of the test result data when performing the AB test by using at least two AB test algorithms includes:

obtaining the number of samples used when the AB test is completed by utilizing the at least two AB test algorithms;

the determining performance information of the at least two AB test algorithms based on the test result data includes:

determining performance information of the AB test algorithm based on the number of samples.

Optionally, the test result data further includes at least one of a P-value, a confidence level, and a confidence interval.

Optionally, the method further includes:

acquiring operation parameters when the at least two AB test algorithms are used for respectively carrying out AB test;

and displaying the operating parameters of the AB test.

Optionally, the operation parameter includes at least one of a current number of used samples, a current P value, a current confidence level, a current confidence interval, and a current test conclusion.

Optionally, the obtaining sample data of the AB test based on the sample obtaining instruction includes:

and acquiring sample data of the AB test from a cloud database or a local database based on the sample acquisition instruction.

Optionally, the invoking of at least two AB test algorithms based on the algorithm call instruction includes:

and calling the at least two AB test algorithms from an AB test algorithm library based on the algorithm calling instruction, wherein the AB test algorithms comprise at least one of a hypothesis testing algorithm, a sequential detection algorithm, a non-parametrical testing algorithm, a variable control algorithm and a user-defined algorithm.

Optionally, the method further includes:

determining data characteristic information of the at least two sets of test data;

the determining the performance information of the at least two AB test algorithms during the AB test includes:

determining performance information of the at least two AB test algorithms corresponding to each data characteristic information.

Optionally, the data characteristic information includes at least one of data source information or data distribution information.

Optionally, the method is executed in a local host, and the method further includes:

displaying a program compiling window;

the get sample get instruction includes:

acquiring a sample acquisition instruction input by a user through the program compiling window;

the obtaining algorithm calling instruction comprises the following steps:

acquiring an algorithm calling instruction input by a user through the program compiling window;

the method further comprises the following steps:

and displaying the performance information of the at least two AB test algorithms in the program compiling window.

Optionally, the method is executed in a local host, and before the obtaining the sample obtaining instruction, the method further includes:

displaying the first interactive interface;

the get sample get instruction includes:

acquiring a sample acquisition instruction input by a user through the first interactive interface;

the obtaining algorithm calling instruction comprises the following steps:

acquiring an algorithm calling instruction input by a user through the first interactive interface;

the method further comprises the following steps:

and displaying the performance information of the at least two AB test algorithms in the first interactive interface.

Optionally, the method is executed on a cloud server, and the obtaining the sample obtaining instruction includes:

receiving a sample acquisition instruction sent by client equipment;

the obtaining algorithm calling instruction comprises the following steps:

receiving an algorithm calling instruction sent by client equipment;

the method further comprises the following steps:

and sending the performance information of the at least two AB test algorithms to the client device.

Optionally, the method further includes:

and sending the performance information of the at least two AB test algorithms to an AB test device.

The embodiment of the present disclosure further provides a method for acquiring performance information of an AB test algorithm, including:

acquiring a sample acquisition instruction input by a user, and sending the sample acquisition instruction to a cloud server so as to enable the cloud server to acquire sample data of an AB test based on the sample acquisition instruction, wherein the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme;

the method comprises the steps of obtaining an algorithm calling instruction input by a user, and sending the algorithm calling instruction to a cloud server so that the cloud server calls at least two AB test algorithms based on the algorithm calling instruction;

and receiving performance information of the at least two AB test algorithms during the AB test fed back by the cloud server, wherein the performance information is determined when the cloud server carries out the AB test respectively based on the sample data and the at least two called AB test algorithms.

Optionally, the method further includes:

displaying the second interactive interface;

the acquiring of the sample acquiring instruction input by the user comprises the following steps:

acquiring a sample acquisition instruction input by a user through the second interactive interface;

the obtaining of the algorithm call instruction input by the user includes:

acquiring an algorithm calling instruction input by a user through the second interactive interface;

the method further comprises the following steps:

and displaying the performance information of the at least two AB test algorithms in the second interactive interface.

The embodiment of the disclosure further provides an AB test method, including:

acquiring test data corresponding to at least two schemes to be tested;

acquiring performance information of at least two AB test algorithms;

determining an AB test algorithm used by the AB test based on the performance information of the at least two AB test algorithms;

and performing the AB test based on the test data corresponding to the at least two schemes to be tested and the determined AB test algorithm used by the AB test.

Optionally, the method further includes:

receiving the performance information of at least two AB test algorithms sent by the performance information acquisition device;

and updating the AB test algorithms included in the AB test algorithm library based on the performance information of the at least two AB test algorithms.

Optionally, the performance information of the at least two AB test algorithms corresponds to data characteristic information of test data, and the determining, based on the performance information of the at least two AB test algorithms, an AB test algorithm used in the AB test includes:

acquiring data characteristic information of the test data;

and determining the AB test algorithm used by the AB test based on the data characteristic information of the test data and the performance information of at least two AB test algorithms corresponding to the data characteristic information of the test data.

The embodiment of the present disclosure further provides a device for acquiring performance information of an AB test algorithm, including:

the sample data acquisition module is used for acquiring a sample acquisition instruction and acquiring sample data of an AB test based on the sample acquisition instruction, wherein the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme;

the algorithm calling module is used for acquiring an algorithm calling instruction and calling at least two AB test algorithms based on the algorithm calling instruction;

and the performance information acquisition module is used for respectively carrying out AB test on the basis of the sample data and the at least two called AB test algorithms and determining the performance information of the at least two AB test algorithms during the AB test.

the system comprises a first instruction acquisition module, a second instruction acquisition module and a third instruction acquisition module, wherein the first instruction acquisition module is used for acquiring a sample acquisition instruction input by a user and sending the sample acquisition instruction to a cloud server so that the cloud server can acquire sample data of an AB test based on the sample acquisition instruction, the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme;

the second instruction acquisition module is used for acquiring an algorithm calling instruction input by a user and sending the algorithm calling instruction to the cloud server so that the cloud server calls at least two AB test algorithms based on the algorithm calling instruction;

and the performance information receiving module is used for receiving the performance information of the at least two AB test algorithms during the AB test fed back by the cloud server, wherein the performance information is determined when the cloud server carries out the AB test based on the sample data and the called at least two AB test algorithms respectively.

The embodiment of the disclosure also provides an AB testing device, which comprises

The test data acquisition module is used for acquiring test data corresponding to at least two schemes to be tested;

the performance information acquisition module is used for acquiring the performance information of at least two AB test algorithms;

the algorithm determining module is used for determining an AB test algorithm used by the AB test based on the performance information of the at least two AB test algorithms;

and the AB testing module is used for carrying out AB testing based on the testing data corresponding to the at least two schemes to be tested and the determined AB testing algorithm used by the AB testing.

The embodiment of the disclosure also provides an AB test system, which comprises the performance information acquisition device of the AB test algorithm and the AB test device.

An embodiment of the present disclosure further provides an electronic device, which includes:

a processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instruction from the memory, and execute the instruction to implement any one of the performance information obtaining methods of the AB test algorithm or the AB test method.

An embodiment of the present disclosure further provides a computer-readable storage medium, where a computer program is stored, and the computer program is used to execute any one of the performance information obtaining methods of the AB test algorithm or the AB test method.

The embodiment of the disclosure provides a technical scheme capable of acquiring performance information of an AB test algorithm and performing an AB test based on the performance information of the AB test algorithm, and particularly, sample data required by the AB test can be acquired in an off-line mode, and at least two AB test algorithms are called. On the basis of the above steps, in the embodiment of the present disclosure, an AB test process is simulated, the AB test is performed by using the at least two AB test algorithms, and further, corresponding test result data is obtained, and then, performance information of the at least two AB test algorithms is determined based on the test result data. Through the technical scheme provided by the embodiment of the disclosure, the performance information of the AB test algorithm can be obtained, and the specific test algorithm used in the follow-up online AB test can be selected based on the performance information, so that the self characteristics of the AB test algorithm can be fully considered in the follow-up selection, the applicability of sample data is fully considered, and the performance of the selected AB test algorithm on the performances such as test efficiency and the like is more excellent.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.

Fig. 1 is a schematic flow chart of a method for acquiring performance information of an AB test algorithm according to an embodiment of the present disclosure;

FIG. 2 is a schematic flow chart illustrating the step 103 in the embodiment shown in FIG. 1;

fig. 3 is a schematic flow chart of another method for acquiring performance information of an AB test algorithm according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a parameter display page provided in the embodiment of the present disclosure;

fig. 5 is a schematic diagram of a parameter display page provided in the embodiment of the present disclosure;

fig. 6 is a schematic flow chart of a method for acquiring performance information of an AB test algorithm according to another embodiment of the present disclosure;

fig. 7 is a schematic flow chart of an AB test method provided in an embodiment of the present disclosure;

fig. 8 is a schematic flow chart of another AB test method provided by the embodiments of the present disclosure;

fig. 9 is a schematic structural diagram of a performance information obtaining apparatus of an AB test algorithm according to an embodiment of the present disclosure;

fig. 10 is a schematic structural diagram of the performance information obtaining module 93 provided in the embodiment shown in fig. 9;

fig. 11 is a schematic structural diagram of a performance information obtaining apparatus of another AB test algorithm provided in the embodiment of the present disclosure;

fig. 12 is a schematic structural diagram of a performance information obtaining apparatus for yet another AB test algorithm according to an embodiment of the present disclosure;

fig. 13 is a schematic structural diagram of an AB testing apparatus according to an embodiment of the present disclosure;

fig. 14 is a schematic structural diagram of another AB test apparatus provided in the embodiments of the present disclosure;

fig. 15 is a schematic structural diagram of an AB test system according to an embodiment of the present disclosure;

fig. 16 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

Aiming at the problems that in the prior art, when an algorithm is on line or a product page is modified, AB test is carried out based on the algorithm before and after the algorithm is on line or the page before and after the modification, test data corresponding to each test scheme in the operation process is obtained by carrying out on-line operation on the algorithm or the page, then the test data obtained by each test scheme is used as sample data and is input into the corresponding AB test algorithm for evaluation and analysis, and an optimal technical scheme is obtained; however, in the different test scenarios, the sample data input into the AB test algorithm has a large difference, and the applicability of different sample data and the AB test algorithm has a large difference; in addition, for the AB test algorithm, various algorithm types are also included, such as a hypothesis test algorithm, a sequential test algorithm, a non-parametric test algorithm, a variable control algorithm, and the like, and each type of AB test algorithm also has its own characteristics. However, in the prior art, when performing AB test, the existing AB test algorithm in the algorithm platform is usually called directly, and AB test is performed based on sample data, so it is difficult to select a proper AB test algorithm according to the characteristics of the AB test algorithm and considering its applicability to various sample data, which may result in low AB test efficiency and high time and cost.

In view of the above defects in the prior art, the embodiments of the present disclosure provide a technical solution capable of acquiring performance information of an AB test algorithm.

According to the technical scheme provided by the embodiment of the disclosure, the performance information of the AB test algorithm can be executed on a local host computer or a cloud server, and can be performed in an off-line mode, and can also be integrated in an on-line AB test system, so that the performance information of the AB test algorithm available in the AB test system can be evaluated; in either case, reference can be made to the selection of the AB test algorithm when performing the AB test. According to the technical scheme provided by the embodiment of the disclosure, for the various situations, sample data for performing the AB test is obtained, the sample data comprises at least two groups of test data, each group of test data corresponds to one test scheme, and at least two AB test algorithms are simultaneously called. On the basis of the steps, an AB test process is simulated in the embodiment of the disclosure, AB tests are performed by using at least two AB test algorithms, corresponding test result data is further obtained, and then performance information of the at least two AB test algorithms is determined based on the test result data. According to the technical scheme provided by the embodiment of the disclosure, the performance information of the AB test algorithm can be obtained, and the performance information is used as a basis for selecting the AB test algorithm in the subsequent online AB test, so that the self characteristics of the AB test algorithm are fully considered in the subsequent selection, the applicability of sample data is fully considered, and the performance of the selected AB test algorithm on the performances such as test efficiency is more excellent.

Fig. 1 is a schematic flowchart of a method for acquiring performance information of an AB test algorithm according to an embodiment of the present disclosure, where the method may be executed on a local host or a cloud server, as shown in fig. 1, the method includes the following steps:

step 101, obtaining a sample obtaining instruction, and obtaining sample data of an AB test based on the sample obtaining instruction, wherein the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme;

specifically, in this step, after the sample acquisition instruction is acquired, based on the sample acquisition instruction, sample data may be acquired from a cloud or locally, where the sample data may be in an AB test process, for example, when a new algorithm is online, the new online algorithm and an original algorithm are respectively used as two test schemes, and the online operation of the new algorithm is used to acquire corresponding test data, for example, a voice recognition algorithm, an identification result of the voice recognition algorithm that is online may be used as the test data, and for different test schemes, a set of test data may be respectively obtained; or when the shopping page or the advertisement page on the APP is modified, two modification schemes are provided, the two modification schemes can be operated on line at the moment, and the click data of the two modification schemes is obtained as the test data to obtain two groups of test data. The sample data includes two groups of test data, which is only an example, and in some cases, more groups of test data may be included, for example, when a shopping page or an advertisement page on the APP is modified, an original shopping page or an advertisement page may be used as a test scheme, and at this time, three groups of test data may be obtained as sample data. In some cases, the test data in the embodiments of the present disclosure may include one of user experience data and/or business data, for example, any one of user click data and exposure data.

In addition, the obtained test data is real operation data, and in some cases, besides the real operation data, one or more of operation data, actual explicitly distributed assumed data, actual explicitly non-analyzed assumed data, virtual explicitly distributed assumed data or virtual explicitly non-distributed assumed data can be simulated;

in addition, the sample data may be stored in a local database, or stored in the cloud, that is, the sample data may be one of local data and cloud data, and the local data may further include text form data or array form data.

102, acquiring an algorithm calling instruction, and calling at least two AB test algorithms from an AB test algorithm library;

in the disclosed embodiment, an AB test algorithm library may be provided, which may include a plurality of AB test algorithms, including but not limited to at least one of hypothesis testing algorithms, sequential testing algorithms, non-parametric testing algorithms, and variable control algorithms. In the embodiment of the disclosure, after the algorithm call instruction is obtained, the corresponding AB test algorithm is selected from the test algorithm library based on the algorithm call instruction, and the performance of the selected AB test algorithm is tested to obtain the performance information.

In some embodiments, besides adding the classic test algorithm to the algorithm library, the user may be allowed to customize the AB test algorithm, and add the customized AB test algorithm to the algorithm library to evaluate the performance information of the AB test algorithm created by the user, and for this step 102, at least two AB test algorithms may be simultaneously selected, and if the selected AB test algorithm includes a classic test algorithm and a customized test algorithm, the performance evaluation of the customized test algorithm may be implemented through subsequent steps.

103, respectively carrying out AB test based on the sample data and the called at least two AB test algorithms, and determining the performance information of the at least two AB test algorithms during the AB test;

in the step, an AB test algorithm is used for carrying out an AB test, the specific execution content of the AB test is the same as that of the traditional AB test, evaluation and analysis are carried out according to at least two groups of test data, the advantages and disadvantages of the test scheme generating the two groups of test data are obtained, and the scheme to be tested is determined to be a recommended scheme, namely when the two test schemes are a new online algorithm and an original algorithm, whether the new algorithm is more excellent than the original algorithm or not is determined; when the two test schemes are two versions of a shopping page or an advertisement page on the APP, determining which version is better, for example, the obtained conversion rate is higher.

In the embodiment shown in fig. 1 of the present disclosure, when only the above technical solution is executed, it may be regarded as a technical solution for completing a normal AB test, through which the evaluation of the quality of the test solution is achieved and a better solution is recommended. However, as a technical solution capable of evaluating the performance information of the AB test algorithms, in this step, not only one AB test algorithm is used for performing the AB test to evaluate each test scheme, but also at least two AB test algorithms are called, and the AB test is performed by using each called AB test algorithm to evaluate each test scheme, so as to further obtain the performance information of each AB test algorithm in this process. In some embodiments, the obtained performance information of the AB test algorithm may be determined based on the number of samples used by the AB test algorithm in the AB test process, and a smaller number of samples generally used means a higher test efficiency in the AB test, otherwise the lower the test efficiency; then, the performance information of the AB test algorithm is determined based on the above-mentioned number of samples, i.e. it can be set that the performance of the AB test algorithm is better when the number of samples is smaller, and the performance of the AB test algorithm is worse when the number of samples is larger.

In the embodiment of the present disclosure, the step 103 of obtaining the performance information of the AB test algorithm may be performed by a method as shown in fig. 2, which may include the following two steps:

step 201, obtaining test result data when performing AB test by using at least two AB test algorithms;

after performing AB tests on the AB test algorithms in the above steps and obtaining corresponding evaluation results, test result data during the AB tests is obtained in this step, and the test result data may be used to determine performance information of the AB test algorithms in the subsequent step 202. Specifically, the test result data may include the number of samples used; in addition, at least one of the parameters such as P value, confidence level or confidence interval may also be included.

The number of samples refers to the number of samples used in the process of running the test scheme, for example, when the test scheme is an algorithm, if the algorithm is a speech recognition algorithm, the number of samples may be the number of speech information input to the speech recognition algorithm; and when the test scheme is a page to be online, the number of the samples is the number of users allocated to each version to be tested. The sample data used in the AB test in the embodiment of the present disclosure may be divided into sample data generated by different samples according to different samples to which the sample data belongs, and different sample data is also successively provided to the AB test algorithm during the AB test, so that the number of samples used when each AB test process is finished, that is, when the evaluation analysis result is obtained, can be obtained.

For both the P value and the confidence level, it can be used to determine the trustworthiness of the evaluation result obtained by the AB test algorithm, for a certain AB test, a target value of the P value or the confidence level can be preset, and in the AB test process, if the P value or the confidence level in the test result data reaches the preset target value, it means that the AB test process can be ended.

The confidence interval is used as a statistical concept, and refers to a sample mean range for interval estimation of overall parameters of a probability sample, and can be used as an auxiliary parameter of a P value or confidence to evaluate whether the AB test process can be ended or not.

Step 202, determining performance information of at least two AB test algorithms based on the test result data.

When performing the AB test in step 202, after obtaining test result data corresponding to each AB test algorithm, that is, after obtaining parameters such as the number of samples, the P value, the confidence level, or the confidence interval, the performance information of the AB test algorithm may be determined based on the test result data, specifically, the P value or the confidence level may be obtained in the AB test process, and it may be determined whether the P value or the confidence level reaches a preset target value and the AB test process is ended, and further, when obtaining the AB test process, the number of samples used by at least two AB test algorithms is obtained.

Specifically, as described in the above embodiment, the two AB test algorithms called in the step 102 are the hypothesis testing algorithm and the sequential testing algorithm, where at the end of the AB test process, the number of samples used by the hypothesis testing algorithm is 10000, and the number of samples used by the sequential testing algorithm is 15000, and then it may be preferentially recommended to use the AB test algorithm with a small number of samples, that is, to determine that the performance information of the hypothesis testing algorithm is better than that of the sequential testing algorithm.

After the performance information of the AB test algorithm is determined according to the number of samples used for finishing the AB test process, a basis can be provided for selecting the AB test algorithm in the subsequent AB test. When the subsequent AB test is performed, if a hypothesis testing algorithm with a smaller number of samples is selected, the time required for performing the AB test can be shortened, and the performance such as the test efficiency is more excellent. Therefore, the self characteristics of the AB test algorithm are fully considered when the AB test algorithm is selected subsequently, and the applicability of sample data is fully considered.

Fig. 3 is a schematic flow chart of another method for acquiring performance information of an AB test algorithm according to an embodiment of the present disclosure, and as shown in fig. 3, in addition to steps 101 to 103 shown in fig. 1, the method may further include:

104, acquiring operation parameters when the AB test is respectively carried out by utilizing the at least two AB test algorithms;

specifically, in the embodiment of the present disclosure, the sample data is sequentially provided to an AB test algorithm, the AB test process may be monitored, and the operation parameters in the AB test process may be obtained in real time, where the operation parameters include but are not limited to at least one of the number of currently used samples, the current P value, the current confidence level interval, and the current test conclusion, and for the number of currently used samples, the current confidence level, and the current confidence level interval, the relevant description in the above embodiment may be referred to, when it is determined that a preset target value is met according to the P value or the confidence level, it may be determined that the AB test may be stopped, and an actual test conclusion may be obtained, and before there is no definite test conclusion, the parameters are null.

In the embodiment of the present disclosure, different operation parameters may also be obtained according to different AB test algorithms.

And 105, displaying the operation parameters during the AB test.

In this step, a parameter display page may be generated, and parameter display frames are displayed on the parameter display page, each parameter display frame corresponds to one operation parameter, for example, a corresponding parameter display frame is set for each of the number of currently used samples, the current P value, the current confidence level, the current confidence interval, and the current test conclusion, and a real-time value of the operation parameter is displayed in each parameter frame, which may specifically refer to fig. 4. Or, for some operation parameters, a corresponding parameter coordinate system may be set and a parameter curve may be displayed, where an abscissa may represent time, and an ordinate represents a corresponding real-time parameter value, and a variation curve of the corresponding operation parameter from the AB test start time to the current time is displayed through the parameter curve, which may specifically refer to fig. 5. Or in some cases, part of the operating parameters on the same reference display page are displayed by using the parameter display box shown in fig. 4, and part of the operating parameters are displayed by using the variation curve shown in fig. 5.

In the embodiment of the disclosure, the running parameters during the AB test are displayed in real time, so that the progress of the AB test can be known at any time, the AB test process is monitored, the running conditions of all AB test algorithms can be known more accurately, the performance information of the AB test algorithms can be analyzed more deeply, and especially for the user-defined AB test algorithms, the development of the AB test algorithms can be guided according to the running parameters obtained through monitoring.

The technical solution provided in the embodiment of the present disclosure may be used as a performance information obtaining scheme of an offline AB test algorithm, and for sample data required in the AB test, the sample data may also be stored in a database in advance, and when performance information evaluation of the AB test algorithm is required, the sample data is obtained from the database, specifically, the database may be a cloud database or a local database, and at this time, step 101 in the embodiment shown in fig. 1 may specifically include:

In addition, in the embodiment of the disclosure, no matter the sample data is stored in the cloud database or the local database, the sample data can be used for performing offline evaluation on the AB test algorithm, and compared with the case that the user traffic needs to be waited by using online data, the offline evaluation scheme of the embodiment of the disclosure can enable the performance evaluation speed to be faster, and can feed back the performance information of the AB test algorithm more timely.

In some embodiments, a rich quantity source is provided for the composition of the sample data, and the sample data can be obtained from various application scenarios, for example, the sample data may be real operating data, simulated operating data, or customized sample data according to requirements. Specifically, the data types of the sample data include real operation data, simulated operation data, actual explicitly distributed assumed data, actual explicitly non-analyzed assumed data, virtual explicitly distributed assumed data, or virtual explicitly non-distributed assumed data. Through the sample data of various data types, the requirement for quickly evaluating the performance of the AB test algorithm can be met, the application scene of the technical scheme is more flexible due to the function of customizing the sample data by a user, and the user can customize the corresponding sample data according to the test or evaluation requirement of the user.

Further, in the embodiment of the present disclosure, because the data may be stored in the cloud database or the local data database, where the data form may include multiple forms, for example, data in an array form, data in a text form, or other data forms suitable for being stored in the cloud database, where the obtained data forms are not uniform, in the embodiment of the present disclosure, the sample data obtained in step 101 may be further formatted to obtain sample data in a more uniform data form, which is favorable for being subsequently input to an AB test algorithm for an AB test.

Further, the invoking of at least two AB test algorithms based on the algorithm invoking instruction in step 102 in the embodiment shown in fig. 1 may include:

and calling at least two AB test algorithms from the AB test algorithm library based on the algorithm calling instruction, wherein the AB test algorithms comprise at least one of a hypothesis testing algorithm, a sequential detection algorithm, a non-parametrical testing algorithm, a variable control algorithm and a user-defined algorithm.

In the embodiment of the present disclosure, in the process of formatting the sample data, it is certainly considered that, in some cases, the function of formatting is added to each AB test algorithm, so that the sample data can be directly input into the AB test algorithm.

In addition, for the different AB test algorithms described above, it is common to first calculate statistics based on the sample data and then proceed with the AB test based on the statistics, e.g., for the hypothesis testing algorithm, the statistics obtained based on the sample data include the mean and variance. In addition, in the case of the user-defined AB test algorithm, if the user-defined AB test algorithm does not have the capability of obtaining statistics based on sample data, the user needs to obtain statistics based on sample data in addition to the above-mentioned formatting process for sample data, and the content included in the specific statistics needs to be defined differently from the user-defined AB test algorithm.

In the above embodiment of the present disclosure, the acquired performance information of the AB test algorithm may be distinguished according to characteristics of sample data, that is, data characteristic information of at least two groups of test data included in the sample data, where the characteristic information of the test data may include at least one of data source information or data distribution information, where the data source information may distinguish that the test data originates from different APPs, different websites, and the like; the data distribution information is a distribution type of the test data, such as at least one of a normal distribution, a gaussian distribution, a gamma distribution, a polynomial distribution, or a binomial distribution.

In the embodiment of the present disclosure, in order to determine the performance information of at least two AB test algorithms, the performance information corresponding to each piece of data characteristic information is determined, that is, for the test data with different pieces of data characteristic information, the performance information of the corresponding AB test algorithms is different, for example, for the test data in normal distribution, where the number of samples used by the first AB test algorithm is smaller than the number of samples used by the second AB test algorithm; and for the test data of the Gaussian distribution, the number of samples used by the second AB test algorithm is smaller than that used by the first AB test algorithm.

In the embodiment of the disclosure, the performance information of the AB test algorithm corresponding to each piece of data characteristic information is specifically obtained, so that when the AB test algorithm is selected in the subsequent AB test, a more matched AB test algorithm can be selected based on the data characteristic information of the test data, and the purposes of reducing the number of samples required by the AB test and improving the AB test efficiency are achieved.

According to the technical scheme provided by the embodiment of the disclosure, two operation modes can be provided, wherein one operation mode is executed in the local host, and the other operation mode is executed in the cloud server.

Corresponding to the first operation mode, when the software test system operates in the local host, the program compiling window may be displayed on the local host, or the software test system may interact with a user by setting the first interactive interface, where the user may be a software tester.

When the program compiling window is used for interacting with a user, the method further comprises the following steps:

and displaying a program compiling window, specifically, providing a compiling function for a user, and inputting an instruction in a command line mode.

Further, after the program compiling window is displayed, the sample obtaining instruction in step 101 shown in fig. 1 may specifically obtain a sample obtaining instruction input by a user through the program compiling window, that is, the sample obtaining instruction is input in the program compiling window in a user command line manner.

Further, the obtaining of the algorithm invoking instruction in step 102 shown in fig. 1 may specifically include: and acquiring an algorithm calling instruction input by a user through the program compiling window, namely inputting the algorithm calling instruction in the program compiling window in a user command line mode.

In this case, the method may further include:

and displaying the performance information of at least two AB test algorithms in a program compiling window.

Different from the above-mentioned interaction with the user by using the program compiling window, the embodiment of the present disclosure may further use the first interactive interface to interact with the user, and at this time, the following steps are further included:

the method includes the steps of displaying a first interactive interface, specifically, the first interactive interface is a mode that a user can operate on the interface, for example, various instructions are input, specifically, displaying a command button on the first interactive interface, and inputting the instructions by clicking the button.

Further, after the first interactive interface is displayed, the instruction for obtaining a sample in step 101 shown in fig. 1 may be specifically an instruction for obtaining a sample input by a user through the first interactive interface, that is, the user inputs the instruction for obtaining a sample through a command button on the first interactive interface.

Further, the obtaining of the algorithm invoking instruction in step 102 shown in fig. 1 may specifically be obtaining an algorithm invoking instruction input by a user through the first interactive interface, that is, the user inputs the algorithm invoking instruction through a command button on the first interactive interface.

In this case, the method may further include:

and displaying the performance information of at least two AB test algorithms on the first interactive interface.

For the second operation mode, the performance information obtaining method of the AB test algorithms provided in the embodiments is executed on the cloud server, and then the cloud server interacts with the client device to obtain an instruction input by a user through the client device, and after obtaining the performance information of the AB test algorithms, the performance information is sent to the client device to obtain the performance information of the at least two AB test algorithms.

In this operation mode, the instruction for obtaining a sample in step 101 shown in fig. 1 may specifically be a sample obtaining instruction sent by the receiving client device; the algorithm invoking instruction obtained in step 102 shown in fig. 1 may specifically be an algorithm invoking instruction sent by a receiving client device.

In the embodiment of the disclosure, the function of acquiring the performance information of the AB test algorithm is set in the cloud server, so that the service is more conveniently provided for a large number of users, the requirement on the client device of the user is reduced, and meanwhile, more diversified AB test algorithms can be set in the AB test algorithm library of the cloud server.

The performance information acquisition method for the technical AB test algorithm provided by the embodiment of the disclosure can be used as an independent operation platform for evaluating the performance of the AB test algorithm, and the performance information of the corresponding AB test algorithm can be acquired through the independent operation platform so as to be referred by users, particularly developers of the AB test algorithm. In some embodiments, the AB test system may further be used as a part of the AB test system, that is, the AB test system includes a performance information obtaining device capable of obtaining performance information of the AB test algorithm and an AB test device capable of performing the AB test.

Fig. 6 is a schematic flowchart of a method for acquiring performance information of an AB test algorithm according to an embodiment of the present disclosure, where the embodiment shown in fig. 6 is a method executed on a client device when the method in the embodiment shown in fig. 1 is executed on a cloud server, and as shown in fig. 6, the method includes the following steps:

601, obtaining a sample obtaining instruction input by a user, and sending the sample obtaining instruction to a cloud server so that the cloud server obtains sample data of an AB test based on the sample obtaining instruction, wherein the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme;

step 602, obtaining an algorithm calling instruction input by a user, and sending the algorithm calling instruction to a cloud server, so that the cloud server calls at least two AB test algorithms based on the algorithm calling instruction;

step 603, receiving performance information of the at least two AB test algorithms during the AB test fed back by the cloud server, wherein the performance information is determined when the cloud server performs the AB test based on sample data and the at least two called AB test algorithms respectively.

In the embodiment of the present disclosure, in a case that the performance information of the AB test algorithm is obtained in the cloud server, a user may input various instructions required by the scheme through the client device, for example, a sample obtaining instruction, an algorithm invoking instruction, and the like, specifically, the instructions may be obtained in an interactive interface manner, that is, before the step 601, the method further includes the following steps:

and displaying the second interactive interface.

Then, further, the obtaining of the sample obtaining instruction input by the user in step 601 may specifically include:

acquiring a sample acquisition instruction input by a user through a second interactive interface;

the obtaining of the algorithm invoking instruction input by the user in step 602 may specifically include:

acquiring an algorithm calling instruction input by a user through a second interactive interface;

after receiving the performance information of the at least two AB test algorithms during the AB test fed back by the cloud server in the step 603, the method may further include

And displaying the performance information of at least two AB test algorithms in the second interactive interface.

The client devices and the cloud server form a system capable of acquiring the performance information of the AB test algorithm, wherein a second interactive interface is developed, so that a user can access the cloud server by using each client device and display the second interactive interface, different users can access the cloud server by using different client devices, and the acquisition of the performance information of at least two AB test algorithms can be realized.

Based on the technical solutions provided by the embodiments of the present disclosure, it is able to obtain performance information of each AB test algorithm, and the embodiments of the present disclosure also provide a technical solution for performing AB test based on the performance information of the AB test algorithms, and fig. 7 is a schematic flow diagram of an AB test method provided by the embodiments of the present disclosure, where the test method may be run on an AB test apparatus, as shown in fig. 7, the AB test method includes the following steps:

701, acquiring test data corresponding to at least two schemes to be tested;

specifically, the two schemes to be tested in this step may include various situations, for example, two pages before and after the page is modified when the page is modified, or two algorithms before and after the new algorithm is on-line when the new algorithm is on-line, which may be used as two methods to be tested. By performing online operation on the algorithm or the page, test data corresponding to each test scheme in the operation process can be acquired.

Step 702, acquiring performance information of at least two AB test algorithms;

specifically, in the embodiment of the present disclosure, a plurality of AB test algorithms are preset in the AB test apparatus, and further, performance information of the plurality of AB test algorithms may be stored, where the performance information may be obtained by referring to the embodiment shown in any one of fig. 1 to 6 of the present disclosure, or may be obtained by other manners.

703, determining an AB test algorithm used by the AB test based on the performance information of at least two AB test algorithms;

after the performance information of each AB test algorithm is obtained in step 702, an AB test algorithm with a better performance can be obtained based on the performance information of each AB test algorithm, for example, when the performance information is based on the number of samples required for completing the AB test, the AB test algorithm with a better performance is an algorithm with a smaller number of samples required for completing the AB test.

And step 704, performing the AB test based on the test data corresponding to the at least two schemes to be tested and the determined AB test algorithm used by the AB test.

The test data is acquired in the step 701, and after the corresponding AB test algorithm is determined based on the performance information in the step 703, the AB test is performed in the step, and the algorithm selected based on the performance information of the AB test algorithm is used, so that the AB test can be completed by using a small number of samples, the AB test efficiency can be remarkably improved, and the time cost required by the AB test can be reduced.

The specific steps for performing the AB test in the embodiments of the present disclosure may also be referred to in the description and illustration of some of the embodiments shown in fig. 1 above.

As described above, in the embodiment of the present disclosure, the AB testing device and the associated performance information acquiring device in the embodiments shown in fig. 1 to 6 may jointly form an AB testing system, and the performance information of at least two AB testing algorithms acquired by the performance information acquiring device may be sent to the AB testing device. Specifically, as shown in fig. 8, on the basis of the embodiment shown in fig. 7, before the above steps 701 to 704, the following steps may be further included:

step 705, receiving performance information of at least two AB test algorithms sent by the performance information acquisition device;

specifically, after receiving the performance information, the performance information may be stored locally, for example, in a performance information list, where the performance information list includes each AB test algorithm and its corresponding performance information.

In addition, further, an AB test algorithm library may also be generally maintained in the AB test apparatus, and a plurality of AB test algorithms are stored in the AB test algorithm library for use in performing an AB test subsequently, in this embodiment of the disclosure, after receiving the performance information of the at least two AB test algorithms, it may be considered that the AB test algorithms included in the AB test algorithm library are updated based on the performance information, and the method may include the following steps:

and step 706, updating the AB test algorithms included in the AB test algorithm library based on the performance information of at least two AB test algorithms.

For example, the AB test algorithm with poor performance information is deleted, and a new AB test algorithm with better performance information is introduced, so that the update iteration of the AB test algorithm can be realized.

After the

steps

705 and 706 are performed, when the

steps

701 and 704 are performed, the AB test algorithm with better performance may be selected for the AB test.

In some embodiments, as described in the above embodiments of the present disclosure, the performance information of at least two AB test algorithms may correspond to the data characteristic information of the test data, that is, for the same AB test algorithm, the performance information of the same AB test algorithm may also be different for different data characteristic information, so that when the algorithm used in the AB test is selected, the data characteristic information of the test data may be obtained first; and then determining the AB test algorithm used by the AB test based on the data characteristic information of the test data and the performance information of at least two AB test algorithms corresponding to the data characteristic information of the test data, so that the AB test algorithm which is most matched with the data characteristic information of the test data is selected, and a better effect is achieved.

In the embodiment of the present disclosure, the data characteristic information may include at least one of data source information or data distribution information. The data source information can be used for distinguishing the situations that the test data come from different APPs or different websites and the like; the data distribution information may be a distribution type of the test data, such as at least one of a normal distribution, a gaussian distribution, a gamma distribution, a polynomial distribution, or a binomial distribution.

Fig. 9 is a schematic structural diagram of a performance information obtaining apparatus for an AB test algorithm according to an embodiment of the present disclosure, where the apparatus may be disposed on a local host or a cloud server, and the apparatus can execute the method shown in fig. 1, and the apparatus includes a sample data obtaining module 91, an algorithm calling module 92, and a performance information obtaining module 93.

The sample data acquiring module 91 is configured to acquire a sample acquiring instruction, and acquire sample data of an AB test based on the sample acquiring instruction, where the sample data includes at least two groups of test data, and each group of test data corresponds to one test scheme;

The algorithm calling module 92 is used for acquiring an algorithm calling instruction and calling at least two AB test algorithms from an AB test algorithm library;

In some embodiments, besides adding the classic test algorithm to the algorithm library, the user may be allowed to customize the AB test algorithm, and add the customized AB test algorithm to the algorithm library to evaluate the performance information of the AB test algorithm created by the user, and for the module, at least two AB test algorithms may be simultaneously selected, and if the selected AB test algorithm includes a classic test algorithm and a customized test algorithm, the performance evaluation of the customized test algorithm may be implemented through subsequent steps.

A performance information obtaining module 93, configured to perform AB tests on the basis of the sample data and the invoked at least two AB test algorithms, and determine performance information of the at least two AB test algorithms during the AB tests;

In the embodiment shown in fig. 9 of the present disclosure, when only the above technical solution is executed, it may be regarded as a technical solution of completing a normal AB test, through which the evaluation of the quality of the test solution is achieved and a better solution is recommended. However, as a technical solution capable of evaluating performance information of the AB test algorithm, in this step, not only one AB test algorithm is used for performing the AB test to evaluate each test scheme, but also at least two AB test algorithms are called, and the AB test is performed by using each called AB test algorithm to evaluate each test scheme. In the process, performance information of each AB test algorithm is further acquired. In some embodiments, the obtained performance information of the AB test algorithm may be determined based on the number of samples used by the AB test algorithm in the AB test process, and a smaller number of samples generally used means a higher test efficiency in the AB test, otherwise the lower the test efficiency; then, the performance information of the AB test algorithm is determined based on the above-mentioned number of samples, i.e. it can be set that the performance of the AB test algorithm is better when the number of samples is smaller, and the performance of the AB test algorithm is worse when the number of samples is larger.

In the embodiment of the present disclosure, the performance information obtaining module 93, as shown in fig. 10, may include the following two sub-modules: a first sub-module 931 and a second sub-module 932.

A first sub-module 931, configured to obtain test result data when performing an AB test by using at least two AB test algorithms;

after performing the AB test on each AB test algorithm and obtaining the corresponding evaluation result, in this step, test result data during the AB test is obtained, and the test result data may be used in the subsequent second sub-module to determine performance information of the AB test algorithm. Specifically, the test result data may include the number of samples used; in addition, at least one of the parameters such as P value, confidence level or confidence interval may also be included.

A second sub-module 932 for determining performance information for at least two AB test algorithms based on the test result data.

When performing the AB test in the second sub-module 932, after obtaining the test result data corresponding to each AB test algorithm, that is, after obtaining the parameters such as the number of samples, the P value, the confidence level, or the confidence interval, the performance information of the AB test algorithm may be determined based on the test result data, specifically, the P value or the confidence level may be obtained in the AB test process, and it may be determined whether the P value or the confidence level reaches a preset target value and the AB test process is ended, and further, when obtaining the AB test process, the number of samples used by at least two AB test algorithms is obtained.

Specifically, as described in the above embodiment, the two AB test algorithms invoked in the second sub-module 832 are the hypothesis testing algorithm and the sequential testing algorithm, wherein at the end of the AB test process, the number of samples used by the hypothesis testing algorithm is 10000, and the number of samples used by the sequential testing algorithm is 15000, and then the AB test algorithm with a small number of samples may be preferentially recommended, that is, the performance information of the hypothesis testing algorithm is determined to be better than that of the sequential testing algorithm.

Fig. 11 is a schematic structural diagram of another performance information obtaining apparatus for an AB test algorithm provided in the embodiment of the present disclosure, as shown in fig. 11, in addition to including the functional modules shown in fig. 9, the apparatus may further include:

a parameter acquisition module 94 and a parameter presentation module 95;

the parameter obtaining module 94 is configured to obtain operation parameters when performing AB tests respectively by using the at least two AB test algorithms;

The parameter display module 95 is used for displaying the operating parameters during the AB test.

In this module, a parameter display page may be generated, and parameter display frames are displayed on the parameter display page, each parameter display frame corresponds to one operation parameter, for example, a corresponding parameter display frame is set for each of the number of currently used samples, the current P value, the current confidence level, the current confidence interval, and the current test conclusion, and a real-time value of the operation parameter is displayed in each parameter frame, which may specifically refer to fig. 4. Or, for some operation parameters, a corresponding parameter coordinate system may be set and a parameter curve may be displayed, where an abscissa may represent time, and an ordinate represents a corresponding real-time parameter value, and a variation curve of the corresponding operation parameter from the AB test start time to the current time is displayed through the parameter curve, which may specifically refer to fig. 5. Or in some cases, part of the operating parameters on the same reference display page are displayed by using the parameter display box shown in fig. 4, and part of the operating parameters are displayed by using the variation curve shown in fig. 5.

The technical solution provided in the embodiment of the present disclosure may be used as a performance information obtaining scheme of an offline AB test algorithm, and for sample data required in the AB test, the sample data may also be stored in a database in advance, and when performance information evaluation of the AB test algorithm is required, the sample data may be obtained from the database, specifically, the database may be a cloud database or a local database, and at this time, the sample data obtaining module 91 in the embodiment shown in fig. 9 may be configured to obtain the sample data of the AB test from the cloud database or the local database based on a sample obtaining instruction.

Further, in the embodiment of the present disclosure, because the data may be stored in the cloud database or the local data database, where the data form may include multiple forms, for example, data in an array form, data in a text form, or other data forms suitable for being stored in the cloud database, where the obtained data forms are not uniform, in the embodiment of the present disclosure, the sample data obtained by the sample data obtaining module 91 may be further formatted first to obtain sample data in a more uniform data form, which is favorable for being subsequently input to an AB test algorithm to perform an AB test.

Further, the algorithm invoking module 92 in the embodiment shown in fig. 9 invokes at least two AB test algorithms based on the algorithm invoking instruction, which may include:

In the above embodiment of the present disclosure, the acquired performance information of the AB test algorithm may be distinguished according to characteristics of sample data, that is, data characteristic information of at least two groups of test data included in the sample data, where the characteristic information of the test data may include at least one of data source information or data distribution information, where the data source information may be used to distinguish that the test data originates from different APPs or different websites; the data distribution information is a distribution type of the test data, such as at least one of a normal distribution, a gaussian distribution, a gamma distribution, a polynomial distribution, or a binomial distribution.

The technical solution provided in the above embodiment of the present disclosure may have two operation modes, one operation mode is to operate in the local host, and correspondingly, the performance information obtaining device of the AB test algorithm is disposed in the local host; the other operation mode is that the device is operated on a cloud server, and correspondingly, the performance information acquisition device of the AB test algorithm is arranged on the cloud server.

Further, after the program compiling window is displayed, the sample obtaining instruction in the sample data obtaining module 91 shown in fig. 9 may specifically obtain a sample obtaining instruction input by the user through the program compiling window, that is, the sample obtaining instruction is input in the program compiling window in a user command line manner.

Further, the obtaining of the algorithm invoking instruction in the algorithm invoking module 92 shown in fig. 9 may specifically include: and acquiring an algorithm calling instruction input by a user through the program compiling window, namely inputting the algorithm calling instruction in the program compiling window in a user command line mode.

In this case, the apparatus may further include:

Different from the above-mentioned interaction with the user by using the program compiling window, in the embodiment of the present disclosure, the interaction with the user may also be performed by using the first interaction interface, and at this time, the following is also included:

Further, after the first interactive interface is displayed, the instruction for obtaining a sample in the sample data obtaining module 91 shown in fig. 9 may be specifically an instruction for obtaining a sample input by the user through the first interactive interface, that is, the user inputs the instruction for obtaining a sample on the first interactive interface through a command button.

Further, the obtaining of the algorithm invoking instruction in the algorithm invoking module 92 shown in fig. 9 may specifically be obtaining an algorithm invoking instruction input by the user through the first interactive interface, that is, the user inputs the algorithm invoking instruction through the command button on the first interactive interface.

In this case, the method may further include:

In this operation mode, the instruction for obtaining a sample in the sample data obtaining module 91 shown in fig. 9 may specifically be a sample obtaining instruction sent by the receiving client device; the algorithm obtaining instruction in the algorithm invoking module 92 shown in fig. 9 may specifically be an algorithm invoking instruction sent by a receiving client device.

The performance information acquisition device for the technical AB test algorithm provided by the embodiment of the disclosure can be used as an independent operation platform for evaluating the performance of the AB test algorithm, and the performance information of the corresponding AB test algorithm can be acquired through the independent operation platform so as to be referred by users, especially developers of the AB test algorithm. In some embodiments, the AB test system may further be used as a part of the AB test system, that is, the AB test system includes a performance information obtaining device capable of obtaining performance information of the AB test algorithm and an AB test device capable of performing the AB test.

Fig. 12 is a schematic structural diagram of a performance information obtaining apparatus for an AB test algorithm according to another embodiment of the present disclosure, where the embodiment shown in fig. 12 is a corresponding apparatus that is set on a client device when the apparatus shown in fig. 9 is set on a cloud server, and as shown in fig. 12, the apparatus includes:

the first instruction obtaining module 121 is configured to obtain a sample obtaining instruction input by a user, and send the sample obtaining instruction to the cloud server, so that the cloud server obtains sample data of an AB test based on the sample obtaining instruction, where the sample data includes at least two groups of test data, and each group of test data corresponds to one test scheme;

the second instruction obtaining module 122 is configured to obtain an algorithm calling instruction input by a user, and send the algorithm calling instruction to the cloud server, so that the cloud server calls at least two AB test algorithms based on the algorithm calling instruction;

the performance information receiving module 123 is configured to receive performance information of the at least two AB test algorithms during the AB test, where the performance information is determined when the AB test is performed by the at least two AB test algorithms respectively, and the performance information is based on sample data by the cloud server.

In the embodiment of the present disclosure, in a case that the performance information of the AB test algorithm is obtained in the cloud server, a user may input various instructions required by the scheme through the client device, for example, a sample obtaining instruction, an algorithm invoking instruction, and the like, specifically, the instructions may be obtained through an interactive interface, that is, the method may further include:

and the page display module is used for displaying the second interactive interface.

Further, the obtaining of the sample obtaining instruction input by the user in the first instruction obtaining module 121 may specifically include:

the obtaining of the algorithm invoking instruction input by the user by the second instruction obtaining module 122 may specifically include:

after the performance information receiving module 123 receives the performance information of at least two AB test algorithms during the AB test fed back by the cloud server, the method may further include:

and displaying the performance information of at least two AB test algorithms in the second interactive interface through a page display module.

Based on the technical solution provided by the foregoing embodiment of the present disclosure, it can obtain performance information of each AB test algorithm, an embodiment of the present disclosure further provides a technical solution for performing an AB test based on the performance information of the AB test algorithm, fig. 13 is a schematic structural diagram of an AB test apparatus provided by an embodiment of the present disclosure, and as shown in fig. 13, the AB test apparatus may include:

the test data acquisition module 131 is configured to acquire test data corresponding to at least two schemes to be tested;

specifically, the two schemes to be tested may include various situations, for example, two pages before and after the page is modified when the page is modified, or two algorithms before and after the new algorithm is on-line when the new algorithm is on-line, which may be used as two methods to be tested. By performing online operation on the algorithm or the page, test data corresponding to each test scheme in the operation process can be acquired.

A performance information obtaining module 132, configured to obtain performance information of at least two AB test algorithms;

An algorithm determining module 133, configured to determine an AB test algorithm used by the AB test based on the performance information of the at least two AB test algorithms;

after the performance information of each AB test algorithm is obtained in the performance information obtaining module 132, an AB test algorithm with a better performance can be obtained based on the performance information of each AB test algorithm, for example, when the performance information is based on the number of samples required for completing the AB test, the AB test algorithm with the better performance is an algorithm that requires fewer samples for completing the AB test.

And the AB testing module 134 is configured to perform an AB test based on the test data corresponding to the at least two schemes to be tested and the determined AB testing algorithm used by the AB test.

After the test data is acquired by the test data acquisition module 131 and the corresponding AB test algorithm is determined based on the performance information in the algorithm determination module 123, the AB test is performed in the module, and the algorithm selected based on the performance information of the AB test algorithm is used, so that the AB test can be completed by using a small number of samples, the AB test efficiency can be remarkably improved, and the time cost required for the AB test can be reduced.

As described above, in the embodiment of the present disclosure, the AB testing device and the associated performance information acquiring device in the embodiments shown in fig. 1 to 6 may jointly form an AB testing system, and the performance information of at least two AB testing algorithms acquired by the performance information acquiring device may be sent to the AB testing device. Specifically, as shown in fig. 14, on the basis of the embodiment shown in fig. 13, the method may further include:

a performance information receiving module 135, configured to receive performance information of at least two AB test algorithms sent by the performance information obtaining apparatus;

In addition, further, an AB test algorithm library may also be generally maintained in the AB test apparatus, and a plurality of AB test algorithms are stored in the AB test algorithm library for use in performing the AB test subsequently, in this embodiment of the disclosure, after receiving the performance information of the at least two AB test algorithms, it may be considered that the AB test algorithms included in the AB test algorithm library are updated based on the performance information, that is, the AB test algorithm library may include:

and an algorithm updating module 136, configured to update the AB test algorithms included in the AB test algorithm library based on the performance information of the at least two AB test algorithms.

In the embodiment of the present disclosure, the data characteristic information may include at least one of data source information or data distribution information. The data source information can be used for distinguishing the situations that the test data come from different APPs or different websites and the like; the data distribution information is a distribution type of the test data, such as at least one of a normal distribution, a gaussian distribution, a gamma distribution, a polynomial distribution, or a binomial distribution.

Fig. 15 is a schematic structural diagram of an AB test system according to an embodiment of the present disclosure, and as shown in fig. 15, the system includes a performance information obtaining device 151 of the AB test algorithm shown in fig. 9 or fig. 11, and an AB test device 152 shown in fig. 13 or fig. 14. As described above, the performance information of the AB test algorithm can obtain the performance information of at least two AB test algorithms, and the AB test apparatus can select the AB test algorithm used in the AB test based on the performance information, so that the AB test algorithm with better performance can be selected, and the test efficiency of the AB test can be improved.

Fig. 15 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure, and for a performance parameter information obtaining scheme of the AB test algorithm in the embodiment of the present disclosure, the performance parameter information obtaining scheme may be implemented on a cloud server or a local host, so that the electronic device may be a cloud server or a local host, or a client device in the embodiment.

Referring now in specific detail, there is shown a schematic diagram of a structure suitable for implementing the electronic device 500 in the embodiments of the present disclosure. The electronic device 500 in the disclosed embodiment may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The illustrated electronic device is merely an example and should not impose any limitations on the functionality or footprint of embodiments of the present disclosure.

As shown, the electronic device 500 may include a processing means (e.g., central processing unit, graphics processor, etc.) 501 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

Generally, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, and the like; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and a communication device 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While the electronic device 500 is illustrated with various means, it is not required that all illustrated means be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 509, or installed from the storage means 508, or installed from the ROM 502. The computer program performs the above-described functions defined in the performance information acquisition method of the AB test algorithm of the embodiment of the present disclosure when executed by the processing device 501.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a sample acquisition instruction, and acquiring sample data of an AB test based on the sample acquisition instruction, wherein the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme; acquiring an algorithm calling instruction, and calling at least two AB test algorithms based on the algorithm calling instruction; and respectively carrying out AB test based on the sample data and the at least two called AB test algorithms, and determining the performance information of the at least two AB test algorithms during the AB test.

Alternatively, the computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:

and receiving performance information of the at least two AB test algorithms during the AB test fed back by the cloud server, wherein the performance information is determined when the cloud server is used for performing the AB test based on the sample data and the at least two called AB test algorithms respectively.

acquiring test data corresponding to at least two schemes to be tested;

acquiring performance information of at least two AB test algorithms;

Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of an element does not in some cases constitute a limitation on the element itself.

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the performance information acquisition method of the AB test algorithm provided in any one of the present disclosure, or for executing the AB test method provided in any one of the present disclosure.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A performance information acquisition method of an AB test algorithm is characterized by comprising the following steps:

2. The method of claim 1, wherein determining performance information of the at least two AB test algorithms at the time of AB testing comprises:

3. The method of claim 2, wherein the test result data comprises a sample number, and wherein obtaining test result data for AB testing using at least two AB testing algorithms comprises:

4. The method of claim 3, wherein the test result data further comprises at least one of a P-value, a confidence level, and a confidence interval.

5. The method of claim 1, further comprising:

and displaying the operating parameters of the AB test.

6. The method of claim 5, wherein the operational parameters include at least one of a current number of samples used, a current P value, a current confidence level, a current confidence interval, and a current test conclusion.

7. The method of claim 1, wherein the obtaining sample data for the AB test based on the sample obtain instruction comprises:

8. The method of claim 1, wherein invoking at least two AB test algorithms based on the algorithm call instruction comprises:

9. The method of claim 1, further comprising:

10. The method of claim 9, wherein the data characteristic information comprises at least one of data source information or data distribution information.

11. The method of claim 1, wherein the method is performed in a local host, the method further comprising:

displaying a program compiling window;

the get sample get instruction includes:

the obtaining algorithm calling instruction comprises the following steps:

the method further comprises the following steps:

12. The method of claim 1, wherein the method is executed in a local host, and wherein the obtaining the sample capture instruction is preceded by:

displaying the first interactive interface;

the get sample get instruction includes:

the obtaining algorithm calling instruction comprises the following steps:

the method further comprises the following steps:

13. The method of claim 1, wherein the method is performed on a cloud server, and wherein obtaining the sample acquisition instructions comprises:

receiving a sample acquisition instruction sent by client equipment;

the obtaining algorithm calling instruction comprises the following steps:

receiving an algorithm calling instruction sent by client equipment;

the method further comprises the following steps:

14. The method of claim 1, further comprising:

15. A performance information acquisition method of an AB test algorithm is characterized by comprising the following steps:

16. The method of claim 15, further comprising:

displaying the second interactive interface;

the obtaining of the algorithm call instruction input by the user includes:

the method further comprises the following steps:

17. An AB test method, comprising:

acquiring test data corresponding to at least two schemes to be tested;

acquiring performance information of at least two AB test algorithms;

18. The method of claim 17, further comprising:

19. The method of claim 17 or 18, wherein the performance information of the at least two AB test algorithms corresponds to data characteristic information of test data, and wherein determining the AB test algorithm used for the AB test based on the performance information of the at least two AB test algorithms comprises:

acquiring data characteristic information of the test data;

20. The method of claim 19, wherein the data characteristic information comprises at least one of data source information or data distribution information.

21. An AB test algorithm performance information acquisition device, comprising:

22. An AB test algorithm performance information acquisition device, comprising:

the system comprises a first instruction acquisition module, a second instruction acquisition module and a third instruction acquisition module, wherein the first instruction acquisition module is used for acquiring a sample acquisition instruction input by a user and sending the sample acquisition instruction to a cloud server so that the cloud server acquires sample data of an AB test based on the sample acquisition instruction, the sample data comprises at least two groups of test data, and each group of test data corresponds to one test scheme;

23. An AB testing device, which is characterized by comprising

24. An AB test system comprising performance information acquisition means of the AB test algorithm of claim 22, and AB test means of claim 23.

25. An electronic device, characterized in that the electronic device comprises:

a processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instructions from the memory and execute the instructions to implement the AB test algorithm performance information acquisition method of any one of claims 1-14, or the AB test algorithm performance information acquisition method of claim 15 or 16, or the AB test method of any one of claims 17-20.

26. A computer-readable storage medium, characterized in that the storage medium stores a computer program for executing the method for obtaining performance information of an AB-test algorithm as claimed in any one of the preceding claims 1-14, or the method for obtaining performance information of an AB-test algorithm as claimed in claim 15 or 16, or the method for AB-test as claimed in any one of the claims 17 to 20.