CN116975520A - Reliability evaluation method, device, equipment and storage medium for AB experiment - Google Patents
Reliability evaluation method, device, equipment and storage medium for AB experiment Download PDFInfo
- Publication number
- CN116975520A CN116975520A CN202310917436.6A CN202310917436A CN116975520A CN 116975520 A CN116975520 A CN 116975520A CN 202310917436 A CN202310917436 A CN 202310917436A CN 116975520 A CN116975520 A CN 116975520A
- Authority
- CN
- China
- Prior art keywords
- data
- group
- index
- experimental
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000002474 experimental method Methods 0.000 title claims abstract description 111
- 238000011156 evaluation Methods 0.000 title abstract description 17
- 238000000034 method Methods 0.000 claims abstract description 43
- 238000012360 testing method Methods 0.000 claims abstract description 36
- 239000011159 matrix material Substances 0.000 claims description 42
- 230000009471 action Effects 0.000 claims description 39
- 238000004590 computer program Methods 0.000 claims description 30
- 239000013598 vector Substances 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 description 12
- 238000004458 analytical method Methods 0.000 description 8
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000010998 test method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000000546 chi-square test Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000000551 statistical hypothesis test Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012353 t test Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Operations Research (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Algebra (AREA)
- Evolutionary Biology (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application relates to a reliability evaluation method, device, equipment and storage medium for AB experiments. The method comprises the following steps: collecting experimental data of an AB experiment, wherein the experimental data comprise flow data and index data; carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data; when the data flow of each group is not different, carrying out hypothesis testing on each group of data according to the flow data and the index data, and determining the score corresponding to the index data; and determining the reliability of the AB experiment according to the score corresponding to the index data. By adopting the method, the reliability of the AB experiment can be obtained by combining the flow non-difference judgment and the hypothesis test, so that the reliability is more accurate.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for evaluating reliability of an AB experiment.
Background
The AB experiment platform is a common experiment design and analysis tool used for evaluating the influence of different variables on experiment results. At present, the experiment reliability in the AB experiment platform is mainly judged by a simple statistical method and is judged by manpower.
However, in the conventional experimental reliability judging method, the judgment is carried out manually, which easily results in the problem of inaccurate judgment.
Disclosure of Invention
Based on this, it is necessary to provide a reliability evaluation method for AB experiments in view of the above technical problems.
A reliability assessment method of AB experiment includes:
collecting experimental data of an AB experiment, wherein the experimental data comprise flow data and index data;
carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data;
when the data flow of each group is not different, carrying out hypothesis testing on each group of data according to the flow data and the index data, and determining the score corresponding to the index data;
and determining the reliability of the AB experiment according to the score corresponding to the index data.
In one embodiment, the flow data includes at least one data item, and the determining that the flow of each set of data in the experimental data is not different according to the flow data includes:
acquiring current A-group data and B-group data in experimental data;
determining standard deviation of each data item in the current A group data and the current B group data;
comparing the standard deviation of each data item in the current A group data with the standard deviation of each data item in the current B group data;
When the difference value of the standard deviation of each data item in the current A group data and the standard deviation of each data item in the current B group data is in a preset range, determining that the current A group data and the current B group data have no difference in flow.
In one embodiment, when the flow rates of the data sets are not different, performing hypothesis testing on the data sets according to the flow rate data and the index data to determine the score corresponding to the index data, including:
acquiring current A-group flow data and index data and current B-group flow data and index data in experimental data;
determining a P value corresponding to a primary assumption according to the current A group flow data and index data, the current B group flow data and index data and a preset primary assumption, wherein the primary assumption is that the current A group index data and the current B group index data have no difference;
comparing the P value with a preset significance level;
when the P value is smaller than the significance level, rejecting the original assumption, and determining that the current A group index data and the current B group index data have differences;
when the index data of the current A group and the index data of the current B group are different, determining the corresponding score of the index data according to the preset score.
In one embodiment, the method further comprises:
Determining a confidence interval corresponding to the original hypothesis according to the experimental data;
when the P value is smaller than the significance level, rejecting the original hypothesis, determining that the current a-group index data is different from the current B-group index data, including:
when the P value is smaller than the significance level and the values of the two endpoints of the confidence interval are the same in sign, rejecting the original assumption, and determining that the current A group index data and the current B group index data are different;
the determining the score corresponding to the index data according to the preset score includes:
and determining the score corresponding to the index data according to the preset score and the confidence interval.
In one embodiment, determining the credibility of the AB experiment according to the score corresponding to the index data includes:
determining the weight of target indexes according to a preset target matrix, wherein the target matrix is a matrix formed by the results of pairwise comparison of the importance of each index of experimental data;
normalizing the target matrix;
averaging each row of the normalized matrix to obtain a weight vector;
determining a target index and a weight corresponding to the target index according to the weight vector;
obtaining the score of the target index from the score corresponding to the index data;
and determining the credibility of the AB experiment according to the score and the weight of the target index.
In one embodiment, the experimental data further includes action data, and the method further includes:
obtaining a score corresponding to the action data according to the action data and weights preset for the action data;
the determining the credibility of the AB experiment according to the score corresponding to the index data comprises the following steps:
and determining the credibility of the AB experiment according to the weight of the target index, the score corresponding to the target index and the score corresponding to the action data.
In one embodiment, the experimental data further includes an experimental scene and a plurality of experimental schemes corresponding to the experimental scene, and the method further includes:
determining an optimal experimental scheme of the experimental scene from a plurality of experimental schemes according to the credibility and the experimental data;
pushing the optimal experimental scheme to a terminal for display.
A reliability evaluation device for AB experiments, said device comprising:
the acquisition module is used for acquiring experimental data of the AB experiment, wherein the experimental data comprise flow data and index data;
the judging module is used for judging whether the flow is different from each group of data in the experimental data according to the flow data;
the processing module is used for carrying out hypothesis testing on each group of data according to the flow data and the index data when the flow of each group of data is not different, and determining the score corresponding to the index data;
And the determining module is used for determining the credibility of the AB experiment according to the score corresponding to the index data.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
collecting experimental data of an AB experiment, wherein the experimental data comprise flow data and index data;
carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data;
when the data flow of each group is not different, carrying out hypothesis testing on each group of data according to the flow data and the index data, and determining the score corresponding to the index data;
and determining the reliability of the AB experiment according to the score corresponding to the index data.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
collecting experimental data of an AB experiment, wherein the experimental data comprise flow data and index data;
carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data;
when the data flow of each group is not different, carrying out hypothesis testing on each group of data according to the flow data and the index data, and determining the score corresponding to the index data;
And determining the reliability of the AB experiment according to the score corresponding to the index data.
The reliability evaluation method, the reliability evaluation device, the computer equipment and the storage medium for the AB experiment are characterized in that experimental data of the AB experiment are collected, and the experimental data comprise flow data and index data; carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data; when the data flow of each group is not different, carrying out hypothesis testing on each group of data according to the flow data and the index data, and determining the score corresponding to the index data; and determining the reliability of the AB experiment according to the score corresponding to the index data. According to the method, the flow data in the experimental data are subjected to flow non-difference judgment, and then the credibility of the experimental data is evaluated by combining the hypothesis testing method, so that the credibility evaluation of the AB experiment is more accurate.
Drawings
FIG. 1 is an application environment diagram of a reliability assessment method for an AB experiment in one embodiment;
FIG. 2 is a flow chart of a method of evaluating the reliability of an AB experiment in one embodiment;
FIG. 3 is a flow chart illustrating a flow non-difference determination step for each set of data in the experimental data according to the flow data in one embodiment;
FIG. 4 is a flowchart illustrating steps for determining scores corresponding to index data according to hypothesis testing of each group of data according to flow data and index data in one embodiment;
FIG. 5 is a block diagram of a reliability assessment device for an AB experiment in one embodiment;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
Referring to fig. 1, fig. 1 is a schematic view of an application environment of a reliability evaluation method for an AB experiment according to an exemplary embodiment of the present application. As shown in fig. 1, the application environment includes a server 100 and a terminal 101, and the server 100 and the terminal 101 can be connected in a communication manner through a network 102, so as to implement the reliability evaluation method of the AB experiment of the present application.
The server 100 is configured to collect experimental data of an AB experiment, where the experimental data includes flow data and index data; carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data; when the data flow of each group is not different, carrying out hypothesis testing on each group of data according to the flow data and the index data, and determining the score corresponding to the index data; determining the reliability of the AB experiment according to the score corresponding to the index data; the reliability of the AB experiment is sent to the terminal 101 for display. The server 100 may be implemented as a stand-alone server or as a server cluster including a plurality of servers.
The terminal 101 is configured to receive and display an evaluation result of the experiment result of the AB experiment sent by the server 100. The terminal 101 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices.
Network 102 is used to implement a network connection between terminal 101 and server 100. In particular, network 102 may comprise various types of wired or wireless networks.
In one embodiment, as shown in fig. 2, a method for evaluating the reliability of an AB experiment is provided, and the method is applied to the server in fig. 1 for illustration, and includes the following steps:
s11, collecting experimental data of an AB experiment, wherein the experimental data comprise flow data and index data.
In the present application, an AB experiment refers to an experiment for comparing two or more different protocols to determine which protocol is more effective. The AB experiment in the application can be an experiment performed on a trade type scheme, an experiment performed on an air control type scheme, and an experiment performed on a front-end page type scheme such as style improvement, U I change and the like. By way of example, if the experiment is performed on a transaction-like scheme, the experimental data may be transaction data, and the specific scheme of the experiment may be, for example, a coupon transmission scheme or a promotional scheme. If the scheme is a scheme of wind control type, for example, the scheme can be a comparison of different schemes of wind control rules.
The experimental data refer to data when two or more schemes participating in comparison are executed, wherein the data comprise data in the execution process and execution results. For example, in the comparison of the scheme for sending coupons, the first scheme is to send coupons of M1 elements, the second scheme is to send coupons of M2 elements, and the information of coupons received by each user side is collected respectively, so as to obtain the experimental data.
The flow data refers to the grouping information and the flow characteristic information of the experimental samples. The grouping information refers to the type of the scheme received by each sample in the experimental samples, i.e. which experimental samples receive the a scheme and which experimental samples receive the B scheme (if there are multiple schemes, the information of the crowd receiving each scheme can also be expanded). Taking the experiment of sending coupons as an example, if the experimental sample is 1 ten thousand people, the experimental scene is sending coupons, the experimental scheme is sending coupons 1 and sending coupons 2, then the group entering information refers to whether the coupon type of the page seen by each experimental sample (person) is 1 or 2.
The flow characteristic information refers to the characteristics of the experimental sample, such as sex, age, and daily habit data of the experimental sample. The index data refers to preset indexes and corresponding values of the indexes in the current experimental scene. The experimental scene in the application can comprise transaction class, wind control class, front page class and the like. For example, in the experimental scenario of the trade type, the index may be set as the trade amount, trade time, trade frequency, and the like.
S12, judging whether the flow is different from each group of data in the experimental data according to the flow data.
In the present application, experimental data is grouped in advance. In the grouping, it is necessary to group according to the experimental scheme adopted by the experimental data, specifically, the experimental data adopting the same experimental scheme are grouped into one group. For example, the experimental protocol involved in the experiment is to send coupon 1 and send coupon 2. Wherein, experiment scheme 1 is to send coupon 1, and experiment scheme 2 is to send coupon 2. The experimental samples are the population receiving the coupons, and when grouping, the experimental data of all the experimental samples receiving the coupons 1 are grouped, and the experimental data of all the experimental samples receiving the coupons 2 are grouped. In the present application, if there are several protocols, the AB experiments are performed in several groups.
The above-mentioned flow non-difference determination refers to analyzing flow data of two groups of experiments (if there are multiple schemes, the two groups are divided into multiple groups), and determining whether the flow of the two groups of experiments is different according to the analysis result.
When the flow data is analyzed, the average value of the flow data can be calculated for comparison, and if the comparison result is not different, the flow of the AB two groups of experiments is determined to be not different.
And S13, when the data flow rates of the groups are not different, carrying out hypothesis test on the data of the groups according to the flow rate data and the index data, and determining the score corresponding to the index data.
In the present application, the above hypothesis test is also called a statistical hypothesis test, and is a statistical inference method for determining whether the sample-to-sample, sample-to-total difference is caused by sampling error or intrinsic difference. Significance testing is one of the most common methods in hypothesis testing, and is also the most basic form of statistical inference, the basic principle of which is to make some assumption about the characteristics of the population first, and then make an inference as to whether the assumption should be rejected or accepted by statistical reasoning of sampling studies. The usual hypothesis test methods include Z test, t test, chi-square test, F test, etc.
In the application, an original hypothesis is preset, the original hypothesis can be an experimental result or an experimental result of the hypothesis A group data and the hypothesis B group data without difference, then calculation is carried out according to the experimental data, and an inference is made on whether the hypothesis is rejected or accepted according to the calculation result.
Illustratively, it is assumed that the above index data includes a conversion rate. Then an assumption is made regarding the conversion rate of the data of group a and the conversion rate of the data of group B, assuming that there is no difference between the two, and then calculation is performed based on the data of group a and the data of group B, and an inference is made as to whether the assumption should be rejected or accepted based on the calculation result. The results of the above hypothesis test include the results of calculation based on experimental data and the results of whether the original hypothesis is received. The result of calculation according to the experimental data needs to be specifically based on the adopted test method of hypothesis test.
S14, determining the reliability of the AB experiment according to the score corresponding to the index data.
In the present application, determining the credibility of the AB experiment according to the score corresponding to the index data may include:
acquiring the weight of each index in the index data;
and determining the credibility of the AB experiment according to the weight of each index and the corresponding score.
In one embodiment, referring to fig. 3, the flow data includes at least one data item, and the determining that the flow of each set of data in the experimental data is not different according to the flow data may include:
s21, acquiring current A-group data and B-group data in the experimental data.
S22, determining standard deviation of each data item in the current A group data and the current B group data.
S23, comparing the standard deviation of each data item in the current A group data with the standard deviation of each data item in the current B group data.
S24, when the difference value of the standard deviation of each data item in the current A group data and the standard deviation of each data item in the current B group data is in a preset range, determining that the current A group data and the current B group data have no difference in flow.
In the present application, the above-mentioned data of group a refers to experimental data corresponding to samples of the a protocol in the AB experiment. The above-mentioned group B data refer to experimental data of experimental samples taken in the B protocol in the AB experiment. It should be noted that the number of the protocols in the AB experiment in the present application may be two or more. When the experimental protocols are multiple, it is necessary to perform a pairwise comparison of experimental data of the multiple protocols, i.e., perform AB experiments pairwise. For example, when three schemes, A, B and C, are included, AB experiments need to be compared for AB, AC and BC, respectively.
The above-mentioned data items refer to respective items included in the flow data. For example, the sample age is one item, the sample gender is one item, the sample preference is one item, and so on.
The server calculates standard deviations of the respective items corresponding to the respective items, and compares the standard deviations of the respective items of the group a with the standard deviations of the respective items corresponding to the group B. For example, the standard deviation of the ages of the group a samples is compared with the standard deviation of the ages of the group B samples, and when the difference obtained by the comparison is within a preset range, it is determined that the item of the ages of the two groups AB is not different. And similarly, comparing the items in turn, and determining that the flow rates of the two groups AB are not different when the results of the items are not different. The preset ranges corresponding to the respective items may be different or the same.
In one embodiment, the method further comprises:
when the standard deviation of one item in the current A group data is different from the standard deviation of the corresponding item in the B group data, determining that the current A group data is different from the B group data;
when the current A group data and the current B group data are different, generating prompt information, wherein the prompt information can comprise prompt contents of problems of the experimental samples.
In another embodiment, the application can also calculate the average value of each item and judge that the flow is not different by comparing the average values.
According to the method, the distribution of flow data is analyzed by calculating the sample mean value, standard deviation and the like of the experimental data, the flow non-difference judgment is carried out among the AB groups, and the reliability of the experimental result of the AB experiment can be evaluated by combining the factors of the experimental flow distribution, so that the evaluation of the experimental result is more accurate.
In one embodiment, referring to fig. 4, when the flow rates of the data sets are not different, performing hypothesis testing on the data sets according to the flow rate data and the index data to determine the score corresponding to the index data, including:
s31, acquiring current A-group flow data and index data and current B-group flow data and index data in the experimental data.
S32, determining a P value corresponding to a primary assumption according to the current A group flow data and index data, the current B group flow data and index data and a preset primary assumption, wherein the primary assumption is that the current A group index data and the current B group index data are not different.
And S33, comparing the P value with a preset significance level.
And S34, rejecting the original assumption when the P value is smaller than the significance level, and determining that the current A group index data and the current B group index data are different.
S35, when the index data of the current A group and the index data of the current B group are different, determining the corresponding score of the index data according to the preset score.
In the application, when the schemes participating in the experiment comprise a plurality of schemes, experimental data corresponding to two schemes are selected from the schemes as current A-group data and current B-group data, and comparison is carried out. Meanwhile, the index data in the application comprises data corresponding to a plurality of indexes, the index data is required to be screened from the current A group data and the current B group data, and the data of the current index is further screened from the index data, so that the current A group index data and the current B group index data are obtained, and the current hypothesis test is carried out. Illustratively, the hypothetical metrics include metric 1, metric 2, and metric 3. The data of index 1 corresponding to experiment scheme 1 may be first obtained to form the current a group index data, and the data of index 1 corresponding to experiment scheme 2 may be obtained to form the current B group index data, so as to perform the first round of hypothesis testing. After the completion, the data of the index 2 corresponding to the experimental scheme 1 is obtained to form the current A-group index data, the data of the index 2 corresponding to the experimental scheme 2 is obtained to form the current B-group index data, so that a second round of hypothesis testing is performed, and the results of the hypothesis testing corresponding to the indexes of the experimental scheme 1 and the experimental scheme 2 are finally obtained.
In the application, the P value analysis method is adopted for the detection when the hypothesis test is carried out. Specifically, the above-mentioned P value refers to a probability value calculated from sample data in hypothesis testing, representing the probability of occurrence of an observed sample result or more extreme results. Typically, we compare the P value with a predetermined significance level to determine if the sample data supports the original hypothesis. The significance level in the present application may be set in advance, and for example, the significance level may be set to 0.05 in general.
In performing the P-value analysis, it is necessary to first determine the hypotheses to be examined, including the original hypothesis and the alternative hypothesis. The original assumption in the present application is that there is no significant difference between the two samples or populations to be compared. The alternative assumption in the present application is that there is a significant difference between the two samples or populations to be compared.
Further, a corresponding test statistic is calculated according to the sample data, and a P value is calculated according to the requirements of the hypothesis test method. Finally, the P value is compared to a significance level (preset threshold), if the P value is less than the significance level, the original hypothesis is rejected, and if not, the original hypothesis is accepted, and the original hypothesis is accepted.
In one possible design, p=m/n here, where n represents the total number of all possible base results in the trial. m represents the number of samples in which the group a index data differs from the group B index data.
In the present application, the preset score may be set according to the actual requirement, for example, may be 0 score and 100 score. When the result of the calculation of the P value is that the original assumption is refused, that is, when the experimental data of the AB group have significant differences, the preset score is determined to be 0 score. The score corresponding to the target index is equal to the preset score of 0 score. Otherwise, when the result of the calculation of the P value is that the original assumption is received, the preset score is 100 points, and the score corresponding to the target index is equal to the preset score which is 100 points.
In the application, the score corresponding to the target index is the score which needs to be deducted when the reliability is calculated later or the evaluation result is obtained, namely, the higher the score corresponding to the target index is, the lower the subsequent evaluation result is.
According to the method, the P value analysis is carried out on the experimental data, and the reliability of the AB experimental data is determined according to the P value analysis result, so that the experiment evaluation is more comprehensive.
In one embodiment, the method may further include:
Determining a confidence interval corresponding to the original hypothesis according to the experimental data;
rejecting the original hypothesis when the P value is smaller than the significance level, determining that the current A group index data is different from the current B group index data, including:
when the P value is smaller than the significance level and the values of the two endpoints of the confidence interval are the same in sign, rejecting the original assumption, and determining that the current A group index data and the current B group index data are different;
determining the score corresponding to the index data according to the preset score, wherein the score comprises the following steps:
and determining the score corresponding to the index data according to the preset score and the confidence interval.
In the application, the hypothesis test can also be tested by calculating the confidence interval, and whether the AB group experimental result (AB group index data) has a difference or not is analyzed by calculating the confidence interval.
The confidence interval in the present application refers to an estimated interval of the overall parameters constructed from the sample statistics. In statistics, the confidence interval of a probability sample is an interval estimate of some overall parameter of the sample. The confidence interval shows the degree to which the true value of this parameter falls with a certain probability around the measurement, which gives the degree of confidence of the measured value of the measured parameter.
In one embodiment, the confidence interval corresponding to the original hypothesis is determined according to the experimental data, and may be calculated by the following formula:
confidence interval = (sample mean a-sample mean b±t distribution critical value standard error)
Where standard error = sqrt (sample variance a/nA + sample variance B/nB)
the t distribution value can be obtained by looking up a table. nA and nB are the sample sizes of samples a and B, respectively.
Illustratively, assume that there are two sets of samples A and B, each containing 100 samples, with sample index means of 10 and 12, respectively, sample index variances of 4 and 5, respectively, and a confidence level of 95%. According to the formula, the standard error is sqrt (4/100+5/100) =0.3, the degree of freedom is 198, and the t distribution threshold is 1.972. Thus, the confidence interval for the index mean difference for samples A and B is (-2.59, -1.41).
Further, when the confidence interval obtains the values of the two endpoints to be the same sign, the original assumption is refused, otherwise, the original assumption is received. For example: the confidence interval is [30,50] or [ -50, -30], then this indicates that the original hypothesis is rejected. If the confidence interval is [ -30, 50], then the original hypothesis is accepted.
In the present application, the determining the score corresponding to the index data according to the preset score may include:
obtaining a score corresponding to the confidence interval according to the confidence interval;
And determining the score corresponding to the index data according to the score corresponding to the confidence interval and the preset score.
Specifically, obtaining the score corresponding to the confidence interval according to the confidence interval may include:
when the confidence interval is different between the left value and the right value, the score corresponding to the confidence interval is a preset score, and is generally set to 0 score. Otherwise, the score corresponding to the confidence interval can be calculated according to the following formula:
score＝100-(w/x)-(|m-x|/x)-(1-α)×100
where w represents the width of the confidence interval, x represents the estimated value, m represents the center of the confidence interval, and α represents the confidence level.
For example, the estimated value is 80, the confidence interval is 70 to 90, and the confidence level is 95%. The confidence interval width is 20 and the center is 80, and the score can be calculated as:
score＝100-(w/x)-(|m-x|/x)-(1-α)×100＝100-(20/80)-(|80-80|/80)-(1-0.95)×100＝94.75
according to the embodiment of the application, the reliability of the AB experiment can be judged by combining the P value analysis and the confidence interval analysis, so that the reliability is more accurate.
In one embodiment, the determining the credibility of the AB experiment according to the score corresponding to the index data may include:
determining the weight of target indexes according to a preset target matrix, wherein the target matrix is a matrix formed by the results of pairwise comparison of the importance of each index of experimental data;
Normalizing the target matrix;
averaging each row of the normalized matrix to obtain a weight vector;
determining a target index and a weight corresponding to the target index according to the weight vector;
obtaining the score of the target index from the score corresponding to the index data;
and determining the credibility of the AB experiment according to the score and the weight of the target index.
In the present application, the target matrix is a predetermined matrix. The values in the matrix are the values of the comparison results obtained by pairwise comparison of the importance of the indexes. That is, given a target matrix, the value a in the target matrix is used to measure the importance of the corresponding index B as compared to the index C.
Specifically, the results of the pairwise comparison of the importance of the respective indexes may be evaluated by an expert, or obtained by a user investigation, and the obtained results may be used to construct the target matrix described above, where the values in the target matrix are a scale representing the importance of two factors, for example, the values 1, 3, 5, and 7 are included in the target matrix. Then 1 indicates that two indices are of equal importance, 3 indicates that one index is slightly more important than the other index, and 7 indicates that one index is more important than the other index.
Illustratively, assuming there are three indices A, B, C, they need to be compared to obtain a target matrix between them:
column normalization is carried out on the target matrix, and the normalized matrix is obtained as follows:
each row of the normalized matrix is averaged to obtain a weight vector w= [0.587,0.324,0.089], and each value in the weight vector is the weight corresponding to the index A, B and the index C sequentially from left to right.
The determining the target index and the weight corresponding to the target index according to the weight vector may include:
and selecting the value with the highest value from the values of the weight vectors as the weight corresponding to the target index, wherein the corresponding index is the target index.
Taking the above example as an example, the target index is an a index, and the corresponding weight is 0.587.
In the present application, the determining the credibility of the AB experiment according to the score and the weight of the target index may include:
multiplying the score of the target index by the weight to obtain a first score;
subtracting the first score from the preset score to obtain the reliability of the AB experiment.
According to the method and the device, the index can be given with the weight, the reliability is evaluated by combining the weight of the index, and the accuracy of the reliability is improved.
In one embodiment, the experimental data further includes motion data, and the method further includes:
obtaining a score corresponding to the action data according to the action data and weights preset for the action data;
the reliability of the AB experiment is determined according to the score corresponding to the index data, and the method comprises the following steps:
and determining the credibility of the AB experiment according to the weight of the target index, the score corresponding to the target index and the score corresponding to the action data.
In the present application, the above-mentioned operation data refers to the operations of the operator on the platform, the number of times, time, frequency, etc. of each operation, for example, the flow rate changing operation, policy replacement, and experimental lock-in group information. Policy replacement such as actions to change coupons, etc. The action data also comprises
The application can preset basic scores for each preset action data, and obtain the corresponding scores of each action according to the basic scores, the sending times, the time and the frequency of each action and the weight of each action.
Further, determining the reliability of the AB experiment according to the weight of the target index, the score corresponding to the target index, and the score corresponding to the action data may include:
And subtracting the product of the weight of the target index and the score corresponding to the target index from the preset total score, and subtracting the score corresponding to the action data to obtain the reliability of the AB experiment.
According to the embodiment of the application, certain actions are taken as the deduction items, and the deduction information of the actions is combined for evaluation when the reliability is evaluated, so that the reliability is more accurate.
In one embodiment, the experimental data further includes an experimental scene and a plurality of experimental schemes corresponding to the experimental scene, and the method further includes:
determining an optimal experimental scheme of the experimental scene from a plurality of experimental schemes according to the credibility and the experimental data;
pushing the optimal experimental scheme to a terminal for display.
In the application, the experimental scene can comprise a transaction scene, a wind control scene and a front-end interface scene. The experimental scheme can be a specific experimental scheme corresponding to an experimental scene. For example, the transaction-type scenario described above may be a specific scheme for issuing different coupons.
In the present application, the determining, according to the reliability and the experimental data, the optimal experimental scheme of the experimental scenario from the multiple experimental schemes may include:
And when the reliability is higher than a preset threshold value and each experimental index in the experimental data is higher than a corresponding preset threshold value, determining the optimal experimental scheme.
In one embodiment, the method further comprises:
pushing calculation results corresponding to the indexes to a terminal for display, wherein the calculation results comprise calculation results of the P value and calculation results of the confidence interval.
In one embodiment, as shown in fig. 5, there is provided a reliability evaluation device for AB experiments, including: the device comprises an acquisition module 11, a judgment module 12, a processing module 13 and a determination module 14, wherein:
the acquisition module 11 is used for acquiring experimental data of the AB experiment, wherein the experimental data comprises flow data and index data;
the judging module 12 is used for judging whether the flow is different from each group of data in the experimental data according to the flow data;
the processing module 13 is configured to perform hypothesis testing on each set of data according to the flow data and the index data when the flow of each set of data is not different, and determine a score corresponding to the index data;
and the determining module 14 is used for determining the credibility of the AB experiment according to the score corresponding to the index data.
In one embodiment, the flow data includes at least one data item, the determining module 12 may obtain current a-group data and B-group data in the experimental data, determine standard deviations of the data items in the current a-group data and the B-group data, compare the standard deviations of the data items in the current a-group data with the standard deviations of the data items in the current B-group data, and determine that the current a-group data and the current B-group data have no difference when the differences of the standard deviations of the data items in the current a-group data and the standard deviations of the data items in the current B-group data are within a preset range.
In one embodiment, the processing module 13 may obtain current a-group flow data and index data and current B-group flow data and index data in the experimental data, determine a P value corresponding to the original hypothesis according to the current a-group flow data and index data and the current B-group flow data and index data and a preset original hypothesis, compare the P value with a preset significance level, reject the original hypothesis when the P value is less than the significance level, determine a difference between the current a-group index data and the current B-group index data, and determine a score corresponding to the index data according to the preset score when the current a-group index data and the current B-group index data are different.
In one embodiment, the processing module 13 may further determine a confidence interval corresponding to the original hypothesis according to the experimental data, reject the original hypothesis when the P value is smaller than the significance level and the values of the two endpoints of the confidence interval are the same in sign, determine that the current a-group index data and the current B-group index data have differences, and determine the score corresponding to the index data according to the preset score and the confidence interval.
In one embodiment, the determining module 14 may determine the weight of the target index according to a preset target matrix, where the target matrix is a matrix formed by comparing the importance of each index of the experimental data in pairs, normalize the target matrix, average each row of the normalized matrix to obtain a weight vector, determine the target index and the weight corresponding to the target index according to the weight vector, obtain the score of the target index from the scores corresponding to the index data, and determine the reliability of the AB experiment according to the scores and the weights of the target index.
In one embodiment, the experimental data further includes action data, and the determining module 14 may further obtain a score corresponding to the action data according to the action data and a weight set for the action data in advance, and determine the reliability of the AB experiment according to the weight of the target index, the score corresponding to the target index, and the score corresponding to the action data.
In one embodiment, the above-mentioned experimental data further includes an experimental scene and a plurality of experimental schemes corresponding to the experimental scene, and the above-mentioned device further includes a pushing module (not shown), where the pushing module may determine an optimal experimental scheme of the experimental scene from the plurality of experimental schemes according to the reliability and the experimental data, and push the optimal experimental scheme to the terminal for display.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer equipment is used for storing data such as operation data of the intelligent household equipment. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method of reliability assessment for an AB experiment.
In one embodiment, a computer device is provided comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of when executing the computer program: collecting experimental data of an AB experiment, wherein the experimental data comprise flow data and index data; carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data; when the data flow of each group is not different, carrying out hypothesis testing on each group of data according to the flow data and the index data, and determining the score corresponding to the index data; and determining the reliability of the AB experiment according to the score corresponding to the index data.
In one embodiment, the flow data includes at least one data item, and when the processor executes the computer program to implement the step of determining that there is no difference in flow rate between each group of data in the experimental data according to the flow data, the specific experiment includes the following steps:
acquiring current A-group data and B-group data in experimental data;
determining standard deviation of each data item in the current A group data and the current B group data;
comparing the standard deviation of each data item in the current A group data with the standard deviation of each data item in the current B group data;
when the difference value of the standard deviation of each data item in the current A group data and the standard deviation of each data item in the current B group data is in a preset range, determining that the current A group data and the current B group data have no difference in flow.
In one embodiment, when the processor executes the computer program to implement the step of performing hypothesis testing on each group of data according to the flow data and the index data and determining the score corresponding to the index data when the flow of each group of data is not different, the following steps are specifically implemented:
acquiring current A-group flow data and index data and current B-group flow data and index data in experimental data;
determining a P value corresponding to a primary assumption according to the current A group flow data and index data, the current B group flow data and index data and a preset primary assumption, wherein the primary assumption is that the current A group index data and the current B group index data have no difference;
Comparing the P value with a preset significance level;
when the P value is smaller than the significance level, rejecting the original assumption, and determining that the current A group index data and the current B group index data have differences;
when the index data of the current A group and the index data of the current B group are different, determining the corresponding score of the index data according to the preset score.
In one embodiment, the processor, when executing the computer program, specifically further implements the steps of:
determining a confidence interval corresponding to the original hypothesis according to the experimental data;
the processor executes the computer program to realize the steps of rejecting the original hypothesis when the P value is smaller than the significance level and determining the difference between the current A group index data and the current B group index data, and concretely realizes the following steps:
when the P value is smaller than the significance level and the values of the two endpoints of the confidence interval are the same in sign, rejecting the original assumption, and determining that the current A group index data and the current B group index data are different;
when the processor executes the computer program to realize the above-mentioned step of determining the score corresponding to the index data according to the preset score, the following steps are specifically realized:
and determining the score corresponding to the index data according to the preset score and the confidence interval.
In one embodiment, when the processor executes the computer program to implement the above step of determining the reliability of the AB experiment according to the score corresponding to the index data, the following steps are specifically implemented:
determining the weight of target indexes according to a preset target matrix, wherein the target matrix is a matrix formed by the results of pairwise comparison of the importance of each index of experimental data;
normalizing the target matrix;
averaging each row of the normalized matrix to obtain a weight vector;
determining a target index and a weight corresponding to the target index according to the weight vector;
obtaining the score of the target index from the score corresponding to the index data;
and determining the credibility of the AB experiment according to the score and the weight of the target index.
In one embodiment, the experimental data further includes action data, and the processor executes the computer program to implement the following steps:
obtaining a score corresponding to the action data according to the action data and weights preset for the action data;
the reliability of the AB experiment is determined according to the score corresponding to the index data, and the method comprises the following steps:
and determining the credibility of the AB experiment according to the weight of the target index, the score corresponding to the target index and the score corresponding to the action data.
In one embodiment, the above experimental data further includes an experimental scene and a plurality of experimental schemes corresponding to the experimental scene, and the processor implements the following steps when executing the computer program:
determining an optimal experimental scheme of the experimental scene from a plurality of experimental schemes according to the credibility and the experimental data;
pushing the optimal experimental scheme to a terminal for display.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of: collecting experimental data of an AB experiment, wherein the experimental data comprise flow data and index data; carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data; when the data flow of each group is not different, carrying out hypothesis testing on each group of data according to the flow data and the index data, and determining the score corresponding to the index data; and determining the reliability of the AB experiment according to the score corresponding to the index data.
In one embodiment, the flow data includes at least one data item, and the computer program is executed by the processor to implement the step of determining that the flow of each set of data in the experimental data is not different according to the flow data, where the specific experiment includes the following steps:
Acquiring current A-group data and B-group data in experimental data;
determining standard deviation of each data item in the current A group data and the current B group data;
comparing the standard deviation of each data item in the current A group data with the standard deviation of each data item in the current B group data;
when the difference value of the standard deviation of each data item in the current A group data and the standard deviation of each data item in the current B group data is in a preset range, determining that the current A group data and the current B group data have no difference in flow.
In one embodiment, the computer program is executed by the processor to perform the steps of performing hypothesis testing on each group of data according to the flow data and the index data when the flow of each group of data is not different, and determining the score corresponding to the index data, and specifically implementing the following steps:
acquiring current A-group flow data and index data and current B-group flow data and index data in experimental data;
determining a P value corresponding to a primary assumption according to the current A group flow data and index data, the current B group flow data and index data and a preset primary assumption, wherein the primary assumption is that the current A group index data and the current B group index data have no difference;
comparing the P value with a preset significance level;
When the P value is smaller than the significance level, rejecting the original assumption, and determining that the current A group index data and the current B group index data have differences;
when the index data of the current A group and the index data of the current B group are different, determining the corresponding score of the index data according to the preset score.
In one embodiment, the computer program when executed by the processor performs the steps of:
determining a confidence interval corresponding to the original hypothesis according to the experimental data;
the computer program is executed by the processor to reject the original assumption when the P value is smaller than the significance level, and when the difference step between the current A group index data and the current B group index data is determined, the following steps are specifically realized:
when the P value is smaller than the significance level and the values of the two endpoints of the confidence interval are the same in sign, rejecting the original assumption, and determining that the current A group index data and the current B group index data are different;
when the computer program is executed by the processor to realize the above-mentioned step of determining the score corresponding to the index data according to the preset score, the following steps are specifically realized:
and determining the score corresponding to the index data according to the preset score and the confidence interval.
In one embodiment, when the computer program is executed by the processor to implement the above step of determining the reliability of the AB experiment according to the score corresponding to the index data, the following steps are specifically implemented:
Determining the weight of target indexes according to a preset target matrix, wherein the target matrix is a matrix formed by the results of pairwise comparison of the importance of each index of experimental data;
normalizing the target matrix;
averaging each row of the normalized matrix to obtain a weight vector;
determining a target index and a weight corresponding to the target index according to the weight vector;
obtaining the score of the target index from the score corresponding to the index data;
and determining the credibility of the AB experiment according to the score and the weight of the target index.
In one embodiment, the experimental data further includes action data, and the computer program when executed by the processor specifically implements the steps of:
obtaining a score corresponding to the action data according to the action data and weights preset for the action data;
the reliability of the AB experiment is determined according to the score corresponding to the index data, and the method comprises the following steps:
and determining the credibility of the AB experiment according to the weight of the target index, the score corresponding to the target index and the score corresponding to the action data.
In one embodiment, the above experimental data further includes an experimental scene and a plurality of experimental schemes corresponding to the experimental scene, and the computer program when executed by the processor specifically implements the following steps:
Determining an optimal experimental scheme of the experimental scene from a plurality of experimental schemes according to the credibility and the experimental data;
pushing the optimal experimental scheme to a terminal for display.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
1. A method for evaluating the reliability of an AB experiment, the method comprising:
collecting experimental data of an AB experiment, wherein the experimental data comprise flow data and index data;
carrying out flow non-difference judgment on each group of data in the experimental data according to the flow data;
when the flow rate of each group of data is not different, carrying out hypothesis testing on each group of data according to the flow rate data and the index data, and determining the score corresponding to the index data;
And determining the credibility of the AB experiment according to the score corresponding to the index data.
2. The method according to claim 1, wherein the flow data includes at least one data item, and the determining that there is no difference in flow between the sets of data in the experimental data according to the flow data includes:
acquiring current A-group data and B-group data in the experimental data;
determining standard deviation of each data item in the current A group data and the current B group data;
comparing the standard deviation of each data item in the current A group data with the standard deviation of each data item in the current B group data;
and when the difference value of the standard deviation of each data item in the current A group data and the standard deviation of each data item in the current B group data is in a preset range, determining that the current A group data and the current B group data have no difference in flow.
3. The method according to claim 1, wherein when the flow rates of the respective sets of data are not different, performing a hypothesis test on the respective sets of data according to the flow rate data and the index data, and determining the score corresponding to the index data includes:
acquiring current A-group flow data and index data and current B-group flow data and index data in the experimental data;
Determining a P value corresponding to a primary assumption according to the current A group flow data and index data, the current B group flow data and index data and the preset primary assumption, wherein the primary assumption is that the current A group index data and the current B group index data are not different;
comparing the P value with a preset significance level;
rejecting the original hypothesis when the P value is smaller than the significance level, and determining that the current A group index data and the current B group index data have differences;
and when the current A group index data and the current B group index data are different, determining the score corresponding to the index data according to a preset score.
4. A method according to claim 3, characterized in that the method further comprises:
determining a confidence interval corresponding to the original hypothesis according to the experimental data;
and rejecting the original hypothesis when the P value is smaller than the significance level, and determining that the current A group index data is different from the current B group index data comprises the following steps:
when the P value is smaller than the significance level and the values of the two endpoints of the confidence interval are the same in sign, rejecting the original assumption, and determining that the current A group index data and the current B group index data are different;
The determining the score corresponding to the index data according to the preset score comprises the following steps:
and determining the score corresponding to the index data according to the preset score and the confidence interval.
5. The method of claim 1, wherein determining the confidence level of the AB experiment based on the score corresponding to the metric data comprises:
determining the weight of target indexes according to a preset target matrix, wherein the target matrix is a matrix formed by the results of pairwise comparison of the importance of each index of the experimental data;
normalizing the target matrix;
averaging each row of the normalized matrix to obtain a weight vector;
determining a target index and a weight corresponding to the target index according to the weight vector;
obtaining the score of the target index from the score corresponding to the index data;
and determining the credibility of the AB experiment according to the score and the weight of the target index.
6. The method of claim 5, wherein the experimental data further comprises action data, the method further comprising:
obtaining a score corresponding to the action data according to the action data and a weight preset for the action data;
The determining the credibility of the AB experiment according to the score corresponding to the index data comprises the following steps:
and determining the credibility of the AB experiment according to the weight of the target index, the score corresponding to the target index and the score corresponding to the action data.
7. The method of claim 6, wherein the experimental data further comprises an experimental scenario and a plurality of experimental protocols corresponding to the experimental scenario, the method further comprising:
determining an optimal experimental scheme of the experimental scene from the plurality of experimental schemes according to the credibility and the experimental data;
pushing the optimal experimental scheme to a terminal for display.
8. A reliability assessment device for AB experiments, the device comprising:
the acquisition module is used for acquiring experimental data of the AB experiment, wherein the experimental data comprise flow data and index data;
the judging module is used for judging whether the flow is different from each group of data in the experimental data according to the flow data;
the processing module is used for carrying out hypothesis testing on each group of data according to the flow data and the index data when the flow of each group of data is not different, and determining the score corresponding to the index data;
And the determining module is used for determining the credibility of the AB experiment according to the score corresponding to the index data.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 7 when the computer program is executed by the processor.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310917436.6A CN116975520A (en) | 2023-07-25 | 2023-07-25 | Reliability evaluation method, device, equipment and storage medium for AB experiment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310917436.6A CN116975520A (en) | 2023-07-25 | 2023-07-25 | Reliability evaluation method, device, equipment and storage medium for AB experiment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116975520A true CN116975520A (en) | 2023-10-31 |
Family
ID=88480937
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310917436.6A Pending CN116975520A (en) | 2023-07-25 | 2023-07-25 | Reliability evaluation method, device, equipment and storage medium for AB experiment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116975520A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117370215A (en) * | 2023-12-05 | 2024-01-09 | 智者四海（北京）技术有限公司 | Optimizing sampling method, optimizing sampling device, electronic equipment and storage medium |
-
2023
- 2023-07-25 CN CN202310917436.6A patent/CN116975520A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117370215A (en) * | 2023-12-05 | 2024-01-09 | 智者四海（北京）技术有限公司 | Optimizing sampling method, optimizing sampling device, electronic equipment and storage medium |
| CN117370215B (en) * | 2023-12-05 | 2024-02-09 | 智者四海（北京）技术有限公司 | Optimizing sampling method, optimizing sampling device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI639824B (en) | Method, apparatus, and non-transitory computer readable storage medium for integration of automatic and manual defect classification | |
| CN111475680A (en) | Method, device, equipment and storage medium for detecting abnormal high-density subgraph | |
| Kočišová et al. | Discriminant analysis as a tool for forecasting company's financial health | |
| TW201350836A (en) | Optimization of unknown defect rejection for automatic defect classification | |
| CN110991474A (en) | Machine learning modeling platform | |
| WO2017220140A1 (en) | Life insurance system with fully automated underwriting process for real-time underwriting and risk adjustment, and corresponding method thereof | |
| Haraksim et al. | Measuring coherence of computer-assisted likelihood ratio methods | |
| US9235463B2 (en) | Device and method for fault management of smart device | |
| CN116975520A (en) | Reliability evaluation method, device, equipment and storage medium for AB experiment | |
| CN111338972A (en) | Machine learning-based software defect and complexity incidence relation analysis method | |
| CN114298176A (en) | Method, device, medium and electronic equipment for detecting fraudulent user | |
| CN110928859A (en) | Model monitoring method and device, computer equipment and storage medium | |
| CN107274025B (en) | System and method for realizing intelligent identification and management of power consumption mode | |
| CN113344469B (en) | Fraud identification method and device, computer equipment and storage medium | |
| CN113837481B (en) | Financial big data management system based on block chain | |
| CN113392086B (en) | Medical database construction method, device and equipment based on Internet of things | |
| CN115630708A (en) | Model updating method and device, electronic equipment, storage medium and product | |
| KR102336462B1 (en) | Apparatus and method of credit rating | |
| Nikolikj et al. | Sensitivity Analysis of RF+ clust for Leave-one-problem-out Performance Prediction | |
| US20090037415A1 (en) | Data analysis | |
| CN112133447A (en) | Construction method of colloid screening model and colloid screening method | |
| Olsen et al. | Predicting dactyloscopic examiner fingerprint image quality assessments | |
| CN111258788A (en) | Disk failure prediction method, device and computer readable storage medium | |
| CN112685324B (en) | Method and system for generating test scheme | |
| CN116577685B (en) | Health detection method, data processing method, related device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |