CN112579422A - Scheme testing method and device, server and storage medium - Google Patents
Scheme testing method and device, server and storage medium Download PDFInfo
- Publication number
- CN112579422A CN112579422A CN201910926314.7A CN201910926314A CN112579422A CN 112579422 A CN112579422 A CN 112579422A CN 201910926314 A CN201910926314 A CN 201910926314A CN 112579422 A CN112579422 A CN 112579422A
- Authority
- CN
- China
- Prior art keywords
- document list
- document
- scheme
- user
- test
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 158
- 238000000034 method Methods 0.000 claims abstract description 38
- 238000005070 sampling Methods 0.000 claims description 18
- 238000012163 sequencing technique Methods 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 13
- 238000002474 experimental method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000013401 experimental design Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of computers, in particular to a scheme testing method, a device, a server and a storage medium, wherein the method comprises the following steps: acquiring a test task, and determining a scheme set to be tested and a test user set according to the test task; generating a document list set according to the schemes in the scheme set to be tested based on preset operation instructions of the users for each user in the test user set, wherein the document lists in the document list set correspond to the schemes in the scheme set to be tested one by one; merging the document lists in the document list set to generate a target document list; sending the target document list to the user, and acquiring click data of the user aiming at the target document list; and analyzing the click data of all the users in the test user set to determine a target scheme. The scheme testing method provided by the invention can improve the testing efficiency, ensure the testing timeliness and improve the accuracy of the testing result.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a scheme testing method, a scheme testing device, a server and a storage medium.
Background
With the development of internet technology, the attention degree of an application program service provider to a user is increased, in the application of providing information service for the user, in order to improve the user experience in the using process, products and systems can be frequently adjusted through various methods and strategies, user feedback needs to be obtained quickly, the effect of a new scheme can be adjusted in time, the scheme is updated iteratively according to feedback data of the user, if the result of the user on the scheme feedback of the products and the systems is not ideal, even the current new scheme needs to be abandoned in time, and the AB test is very important at the moment and is the core basis of data-driven products.
The traditional AB test is to divide users into A, B groups by random sampling, apply different product solutions to different groups of users, and analyze A, B the behavior of the two groups of users by statistical methods to determine a more optimal product solution. The AB test aims to obtain a representative experimental conclusion through scientific experimental design, sample representativeness, flow segmentation and the like, and the conclusion is ensured to be credible when popularized to all flows.
However, the conventional AB test requires a large number of samples for obtaining a large statistical Power (Power) and accuracy, and if the flow rate allocated by the test is small, it takes a long time to accumulate enough samples. For the time-sensitive test of news, the traditional AB test has poor effect.
Disclosure of Invention
In view of the foregoing problems in the prior art, an object of the present invention is to provide a method, an apparatus, a server, and a storage medium for testing a scenario, which can improve the testing efficiency and the accuracy of the testing result.
In order to solve the above problem, the present invention provides a method for testing a scenario, including:
acquiring a test task, and determining a scheme set to be tested and a test user set according to the test task;
generating a document list set according to the schemes in the scheme set to be tested based on preset operation instructions of the users for each user in the test user set, wherein the document lists in the document list set correspond to the schemes in the scheme set to be tested one by one;
merging the document lists in the document list set to generate a target document list;
sending the target document list to the user, and acquiring click data of the user aiming at the target document list;
and analyzing the click data of all the users in the test user set to determine a target scheme.
Another aspect of the present invention provides a scenario testing apparatus, including:
the acquisition module is used for acquiring a test task and determining a scheme set to be tested and a test user set according to the test task;
a generating module, configured to generate, for each user in the test user set, a document list set according to a scheme in the to-be-tested scheme set based on a preset operation instruction of the user, where document lists in the document list set correspond to schemes in the to-be-tested scheme set one to one;
the merging module is used for merging the document lists in the document list set to generate a target document list;
the sending module is used for sending the target document list to the user and acquiring click data of the user aiming at the target document list;
and the analysis module is used for analyzing the click data of all the users in the test user set to determine a target scheme.
Another aspect of the invention provides a server comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by the processor to implement the method as described above.
Another aspect of the invention provides a computer readable storage medium having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by a processor to implement a method as described above.
Due to the technical scheme, the invention has the following beneficial effects:
according to the scheme testing method, the device, the server and the storage medium, a plurality of document lists are generated based on the preset operation instructions of the users according to the scheme of the scheme set to be tested, the document lists generated by different schemes are combined into one target document list to be sent to the users, so that the click data of the users aiming at the target document list are obtained, and finally the optimal scheme is determined by counting and analyzing the click data of all the users in the test user set. In the test process, users concentrated by test users do not need to be grouped according to different schemes, the whole test shares the same flow, and the required flow is small, so that more schemes and tests can be supported to be carried out simultaneously, the test efficiency can be improved, the test timeliness can be ensured, and the accuracy of test results can be improved.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings used in the description of the embodiment or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a schematic illustration of an implementation environment provided by an embodiment of the invention;
FIG. 2 is a flow chart of a scenario testing method provided by one embodiment of the present invention;
FIG. 3 is a schematic diagram of a scenario test system provided by an embodiment of the present invention;
FIG. 4 is a flow chart of a protocol testing method provided by another embodiment of the present invention;
FIG. 5 is a flow chart of a protocol testing method provided by another embodiment of the present invention;
FIG. 6 is a flow chart of a scenario testing method provided by another embodiment of the present invention;
FIG. 7 is a flow chart of a scenario testing method provided by another embodiment of the present invention;
FIG. 8 is a schematic diagram of a scenario testing method under an AB test scenario according to an embodiment of the present invention;
FIG. 9 is a schematic structural diagram of a scenario testing apparatus provided in an embodiment of the present invention;
fig. 10 is a schematic structural diagram of a server according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or device that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or device.
In order to make the objects, technical solutions and advantages disclosed in the embodiments of the present invention more clearly apparent, the embodiments of the present invention are described in further detail below with reference to the accompanying drawings and the embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of the invention and are not intended to limit the embodiments of the invention. First, the embodiments of the present invention explain the following concepts:
and (4) AB test: the users are divided into A, B two groups by random sampling and different strategies are applied, giving A, B a statistical method whether there is a significant difference in the user behavior of the two groups, thereby concluding which strategy is superior.
Statistical efficacy (Power): when the actual effect is better for group a than group B, the experimental data shows the probability that group a is better than group B.
The Interleaving algorithm: the two lists are combined according to a certain rule and rearranged into a new list.
Referring to fig. 1 of the specification, a schematic diagram of an implementation environment of a scenario testing method according to an embodiment of the present invention is shown, and as shown in fig. 1, the implementation environment may include at least a client 110 and a server 120.
The client 110 may include a physical device such as a smart phone, a tablet computer, an e-book reader, a laptop portable computer, a desktop computer, etc., and may also include software running on the physical device, such as a web page provided by a service provider to a user, and applications provided by the service provider to the user. The client 110 may specifically be an application providing an information service, and may include a search engine, a news application, a video playing application, and the like.
The server 120 may comprise a server operating independently, or a distributed server, or a server cluster composed of multiple servers. The server 120 may establish a communication connection with the client 110 through a wireless or wired network. The server 120 may be specifically configured to implement the scheme testing method provided in the embodiment of the present invention.
Referring to fig. 2 of the specification, a flow of a scenario testing method provided by an embodiment of the present invention is shown, where the scenario testing method is applicable to a scenario testing apparatus provided by an embodiment of the present invention, and the scenario testing apparatus may be configured in a server. Although the present specification provides method steps as described in the examples or flowcharts, additional or fewer steps may be included based on routine or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. In actual system or product execution, sequential execution or parallel execution (e.g., parallel processor or multi-threaded environment) may be possible according to the embodiments or methods shown in the figures. As shown in fig. 2, the method may include the steps of:
s210: and acquiring a test task, and determining a scheme set to be tested and a test user set according to the test task.
In the embodiment of the invention, the test task can be designed and issued by a tester, the test task can comprise a scheme set to be tested, the scheme set to be tested can comprise a plurality of schemes, and the optimal scheme can be determined from the plurality of schemes by the scheme test method of the embodiment of the invention. Specifically, the test task may include an AB test, and the number of solutions in the solution set to be tested may be two.
In one specific embodiment, referring to FIG. 3 in conjunction with the description herein, a tester may design experiments, configure experiments, observe experiments, and manage experiments via a world Wide Web system (Web System). After a tester designs an experiment, the experimental configuration can be submitted to a flow management module, a reasonable random flow distribution algorithm is provided, the gray-level issuing experiment configuration is sent to an agent service module of a multi-Internet Data Center (IDC), and a test task is sent to an experiment access module through the agent service. And the test task is completed by the interaction of the experiment access module and the user client.
In another possible embodiment, the obtaining a test task, and determining a to-be-tested solution set and a test user set according to the test task may include:
determining flow configuration information according to the test task;
and determining a plurality of users through random sampling based on the flow configuration information, and generating the test user set.
In the embodiment of the invention, the testThe test task may include traffic configuration information for executing the test task, and the test users may be sampled in a random sampling manner according to the traffic configuration information. For example, the whole traffic may be divided into ten thousand buckets, each test task is allocated a certain number of buckets, and the user is sampled according to the proportion of the number of buckets, for example, the number of buckets of a test task may be set to 8, and random sampling may be performed
To perform the test task.
In another possible embodiment, the obtaining a test task, and determining a to-be-tested solution set and a test user set according to the test task may further include:
determining the limiting conditions of the users in the test user set according to the test task;
acquiring user figures of all users;
and screening the users determined by random sampling based on the user portrait so that the users in the test user set meet the limiting conditions.
In an embodiment of the present invention, the limiting condition may include a special requirement set by a tester for a user performing the test task, for example, a requirement for the age, occupation, and the like of the user, and may further include a requirement for the test user set, for example, a requirement for the number of users in the test user set. The user image can be used for distinguishing different users, the user image can comprise information such as identification, gender, age, occupation and social relation of the user, and in the process of determining the user executing the test task, screening can be carried out according to the user image, and the user meeting the limiting conditions is selected.
Illustratively, if the limiting condition is girl with age greater than 24 years old, a user may be randomly sampled, and then it is determined whether the user's age is greater than 24 years old and gender is female according to the user's representation, if the user's age is greater than 24 years old and gender is female, the user is added to a test user set, otherwise, the sampling is performed again until the number of sampled users meets the set traffic configuration information. And if the limiting condition is that the number of the users in the test user set is not more than 10 ten thousand, limiting the number of the sampled users to be not more than 10 ten thousand in the sampling process.
S220: and aiming at each user in the test user set, generating a document list set according to the scheme in the scheme set to be tested based on a preset operation instruction of the user, wherein the document lists in the document list set correspond to the schemes in the scheme set to be tested one by one.
In the embodiment of the present invention, after determining the users in the test user set, a preset operation instruction of the user may be obtained through the user client, where the preset operation instruction may be an operation instruction generated by a user clicking a preset operation control (for example, clicking an operation icon of the client or clicking a preset control displayed by the client), and may also be a voice operation instruction obtained after the user clicks the preset operation control. After the preset operation instruction of the user is obtained, a document list can be generated according to each scheme in the scheme set to be tested, and the document list set is formed by combining the schemes.
In the embodiment of the invention, the preset operation instruction can be different according to different schemes in the scheme set to be tested. For example, if the scheme is a search policy, the preset operation instruction may be a search instruction issued after a user inputs a search word, and if the scheme is a recommendation policy, the preset operation instruction may be a recommendation instruction issued by the user. It is understood that the preset operation instruction may also be another type of operation instruction, and is not described herein again.
Exemplarily, assuming that the solution set to be tested includes two solutions a and B, after the preset operation instruction of the user is obtained, a document list a may be generated according to the solution a, a document list B may be generated according to the solution B, and the document list a and the document list B are combined into a document list set.
S230: and combining the document lists in the document list set to generate a target document list.
In the embodiment of the present invention, a merging process of the document lists may be implemented by a real-time stream processing technology, and the target document list may include all documents of the document lists in the document list set.
In one possible embodiment, as shown in fig. 4, the merging the document lists in the document list set to generate the target document list may include:
s2311: an initially empty first document list is constructed.
S2321: and randomly sequencing the document lists in the document list set to obtain a sequencing result.
S2331: sequentially selecting a target document from the document list according to the sorting result; and if the target document is different from the documents in the first document list, adding the target document into the first document list.
S2341: and when all the documents in the document list are added into the first document list, taking the first document list as a target document list.
In the embodiment of the invention, the document lists in the document list set can be merged through a Balanced Interleaving algorithm to obtain a target document list, and when the target document is selected from the document list, the target document can be sequentially selected according to the sequence of the documents in the document list. In practical applications, the step S2331 may be performed multiple times so that all documents in the document list are added to the first document list.
In one example, assuming that the document list set includes document list a and document list B, document list a and document list B may be first randomly ordered to determine which document list to select from first to join the first document list. If the sequencing result is A and B, selecting a target document from the document list A to be added into the first document list, then selecting a target document from the document list B to be added into the first document list, and repeating the steps until all documents are included in the first document list. Before adding the selected target document into the first document list, it may be determined whether the target document is different from the documents in the first document list, if so, the target document is added into the first document list, and if not, the target document is not added into the first document list. For example, if document list a includes documents d1, d3, d5, and document list B includes documents d2, d1, d4, and if documents d1, d2, d3 are already included in the first document list, then a target document d1 needs to be selected from document list B and added to the first document list, and since document d1 is already included in the first document list, document d1 may not be added to the first document list any more, and then a target document d5 continues to be selected from document list a and added to the second document list, and the finally obtained first document list may include documents d1, d2, d3, d5, d 4.
In another possible embodiment, as shown in fig. 5, the merging the document lists in the document list set to generate the target document list may include:
s2312: an initially empty second document list is constructed.
S2322: and randomly sequencing the document lists in the document list set to obtain a sequencing result.
S2332: according to the sorting result, sequentially selecting a target document from the document list, wherein the target document is different from the documents in the second document list; and adding the target document into the second document list, so that the difference value between the number of the documents belonging to any two document lists in the second document list is smaller than or equal to a preset threshold value.
S2342: and when all the documents in the document list are added into the second document list, taking the second document list as a target document list.
In the embodiment of the invention, the document lists in the document list set can be merged through a Team-draft Interleaving algorithm to obtain a target document list, and when the target document is selected from the document list, the target document can be sequentially selected according to the sequence of the documents in the document list. In the merging process, a difference between the numbers of documents belonging to any two document lists in the second document list may be made smaller than or equal to a preset threshold (for example, may be set to 1). In practical applications, the step S2332 may be performed multiple times so that all documents in the document list are added to the second document list.
In one example, assuming that the document list set includes document list a and document list B, document list a and document list B may be first randomly ordered to determine which document list to select from first to join the second document list. If the sequencing result is A and B, selecting a target document from the document list A to be added into the second document list, then selecting a target document from the document list B to be added into the second document list, and repeating the steps until all documents are included in the second document list. In the merging process, the target documents selected from the document list a and the document list B are different from the documents in the second document list, so as to ensure that the difference between the number of the documents belonging to any two document lists in the second document list is less than or equal to 1. For example, if document list a includes documents d1, d3, d5, and document list B includes documents d2, d1, d4, and it is assumed that documents d1, d2, d3 are already included in the second document list, then document d1 needs to be selected from document list B as a target document, since document d1 is already included in the second document list, document d4 needs to be selected from document list B as a target document again, since document d4 is not included in the second document list, document d4 may be added to the second document list, and then one target document d5 continues to be selected from document list a to be added to the second document list, and the resulting second document list may include documents d1, d2, d3, d4, d 5. In the merging process, the number of the documents belonging to the document list A and the document list B in the second document list is the same or the difference value is 1.
S240: and sending the target document list to the user, and acquiring click data of the user aiming at the target document list.
In the embodiment of the present invention, the target document list may be sent to a user client, so that the user client displays the target document list to a user, and the user client obtains and returns click data of the user for the target document list, where the click data may include document information clicked by the user.
S250: and analyzing the click data of all the users in the test user set to determine a target scheme.
In the embodiment of the invention, after the click data of the user for the target document list is acquired, the target scheme can be determined by analyzing the click data in an off-line manner. The target scheme may be a scheme with the most clicks by the users in the scheme set to be tested, and after the target scheme is obtained, the target scheme may be used as a scheme to be finally adopted. In a specific embodiment, referring to fig. 3 in the specification, the target scheme may be determined by analyzing click data of all users in the test user set through an index analysis module.
In one possible embodiment, as shown in fig. 6, the analyzing the click data of all users in the test user set to determine the target scheme may include:
s610: and counting the number of document clicks of each user in the test user set aiming at the document list in the document list set.
S620: and calculating the score of the scheme in the to-be-tested scheme set corresponding to the document list according to the number of the document clicks.
S630: and determining candidate schemes according to the scores of the schemes in the scheme set to be tested.
S640: and determining a target scheme according to the candidate scheme.
In the embodiment of the invention, the number of documents clicked by users in each document list in the document list set can be respectively counted, the number of times that the scheme corresponding to each document list wins is determined according to the number of the clicked documents, the score of each scheme is calculated, the candidate scheme wins is determined, the confidence coefficient of the conclusion that the candidate scheme wins is finally judged, and the final target scheme is determined.
In one possible embodiment, as shown in fig. 7, the determining the target solution according to the candidate solution may include:
s641: and randomly sampling click data of all users in the test user set for multiple times to obtain a sample data set.
In the embodiment of the invention, the number of samples subjected to multiple random sampling can be half of the total number of samples, and one sample data can be obtained by performing random sampling once. It should be noted that the number of randomly sampled samples may also take other statistically significant values.
S642: and analyzing the sample data in the sample data set, and determining a confidence interval.
In the embodiment of the invention, each sample data obtained by sampling can be analyzed, the sample score of a candidate scheme can be calculated by the method, a plurality of sample scores obtained by sampling for multiple times are ranked, and then two preset quantiles are selected as the lower bound and the upper bound of the confidence interval.
S643: and if the score of the candidate scheme is within the confidence interval, the candidate scheme is taken as a target scheme.
In the embodiment of the invention, the confidence of the selected candidate scheme can be determined by judging whether the score of the candidate scheme is located in the confidence interval, if the score of the candidate scheme is located in the confidence interval, the confidence of the conclusion that the candidate scheme is superior is shown, and the candidate scheme can be used as the target scheme; if the score of the candidate scheme is outside the confidence interval, the conclusion that the candidate scheme wins is low in confidence and cannot be adopted, and the test can be carried out again at the moment.
The following describes a scheme testing method provided by an embodiment of the present invention by taking a specific application scenario as an example. With reference to fig. 8 of the specification, when a user searches for information, different search strategies (for example, a strategy a and a strategy B, respectively) may be designed for the same search behavior, and click data of search results obtained by the user for the different search strategies may be analyzed through an AB test, so as to determine an optimal search strategy as a search strategy to be finally adopted.
In the AB test process, it is necessary to first determine traffic configuration information for testing, for example, 2% of overall traffic may be used for testing, and a plurality of users are determined by random sampling according to the traffic configuration information to generate a test user set. After the search instruction of the user in the test user set is obtained, a document list A can be generated based on an A strategy according to a search word input by the user, a document list B can be generated based on a B strategy, then the generated document list A and the generated document list B can be combined into a target document list and sent to the user, and click data of the user on the target document list is obtained. And finally, analyzing the click data of all the users in the test user set to determine the optimal search strategy.
Specifically, assuming that the test user set includes 100 ten thousand users, counting click data of the 100 ten thousand users, if the number of times (a) that a certain user clicks a document in the document list a generated according to the policy a is greater than the number of times (B) that a user clicks a document in the document list B generated according to the policy B, adding 1 to the number of times (a) that the policy a wins, if the number (a) is less than the number (B), adding 1 to the number (win) (B) of times that the policy B wins, and if the number (a) is equal to the number (B), adding 1 to the number of times (ab) that the policy a and the policy B are leveled. Counting the click data of the 100 ten thousand users to obtain the final values of the times win (A) of strategy A winning, the times win (B) of strategy B winning and the times tie (AB) of strategy A and strategy B keeping equal, and calculating the score difference value delta (AB) of strategy A and strategy B according to the following formula:
determining a winning strategy according to the value of delta (AB), if the value of delta (AB) is larger than zero, winning the strategy A, and taking the strategy A as a candidate strategy; if the value of delta (AB) is less than zero, then strategy B wins, and strategy B is taken as a candidate strategy.
After the candidate strategy is determined, 1 ten thousand times of sampling can be performed on the click data of 100 ten thousand users, the number of samples sampled each time can be taken as the click data of 50 ten thousand users, so that 1 ten thousand sample data can be obtained, the value of 1 ten thousand deltas (ab) can be obtained by analyzing the 1 ten thousand sample data, 5% quantiles and 95% quantiles are respectively taken as the lower bound lower (delta) and the upper bound upper (delta) of a confidence interval, if the delta (ab) calculated by the formula (1) is greater than lower (delta) and less than upper (delta), the conclusion confidence coefficient is high, and the candidate strategy can be taken as the target strategy.
In summary, according to the scheme testing method provided by the invention, a plurality of document lists are generated based on the preset operation instruction of the user according to the scheme in the to-be-tested scheme set, the document lists generated by different schemes are merged into one target document list to be sent to the user, so as to obtain the click data of the user aiming at the target document list, and finally, the optimal scheme is determined by counting and analyzing the click data of all users in the testing user set. In the test process, users concentrated by test users do not need to be grouped according to different schemes, the whole test shares the same flow, and the required flow is small, so that more schemes and tests can be supported to be carried out simultaneously, the test efficiency can be improved, the test timeliness can be ensured, and the accuracy of test results can be improved.
Referring to the specification fig. 9, which shows a schematic structural diagram of a scenario testing apparatus 900 according to an embodiment of the present invention, the apparatus 900 may include:
an obtaining module 910, configured to obtain a test task, and determine a to-be-tested scheme set and a test user set according to the test task;
a generating module 920, configured to generate, for each user in the test user set, a document list set according to a scheme in the to-be-tested scheme set based on a preset operation instruction of the user, where document lists in the document list set correspond to schemes in the to-be-tested scheme set one to one;
a merging module 930, configured to merge the document lists in the document list set to generate a target document list;
a sending module 940, configured to send the target document list to the user, and obtain click data of the user for the target document list;
an analysis module 950, configured to analyze click data of all users in the test user set, and determine a target scheme.
In one possible embodiment, the merging module 930 may include:
the first construction unit is used for constructing an initially empty first document list;
the first selecting unit is used for randomly selecting one document from all document lists in the document list set and adding the selected document into the first document list;
and the first determining unit is used for taking the first document list as a target document list after all documents in the document list are added into the first document list.
In another possible embodiment, the merging module 930 may include:
the second construction unit is used for constructing an initially empty second document list;
the sorting unit is used for randomly sorting the document lists in the document list set to obtain a sorting result;
the second selecting unit is used for sequentially selecting one document from the document lists and adding the document into the second document list according to the sorting result, so that the difference value between the number of the documents belonging to any two document lists in the second document list is smaller than or equal to a preset threshold value;
and the second determining unit is used for taking the second document list as a target document list after the documents in all the document lists are added into the second document list.
In another possible embodiment, the analysis module 950 may include:
the statistical unit is used for counting the document clicking number of each user in the test user set aiming at the document list in the document list set;
the calculation determining unit is used for calculating the scores of the schemes in the to-be-tested scheme set corresponding to the document list according to the number of the document clicks;
a third determining unit, configured to determine candidate schemes according to scores of the schemes in the scheme set to be tested;
a fourth determining unit, configured to determine a target scheme according to the candidate scheme.
It should be noted that, when the apparatus provided in the foregoing embodiment implements the functions thereof, only the division of the functional modules is illustrated, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the apparatus may be divided into different functional modules to implement all or part of the functions described above.
An embodiment of the present invention further provides a server, where the scenario testing apparatus is disposed in the server, and the server includes a processor and a memory, where the memory stores at least one instruction, at least one program, a code set, or a set of instructions, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by the processor to implement the scenario testing method provided in the above method embodiment.
The memory may be used to store software programs and modules, and the processor may execute various functional applications and protocol tests by executing the software programs and modules stored in the memory. The memory can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs needed by functions and the like; the storage data area may store data created according to use of the apparatus, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory may also include a memory controller to provide the processor access to the memory.
In a specific embodiment, please refer to fig. 10 of the specification, which shows a schematic structural diagram of a server provided in an embodiment of the present invention. The server is used for implementing the scheme testing method provided in the above embodiment. As shown in fig. 10, the server 1000 may have a relatively large difference due to different configurations or performances, and may include one or more Central Processing Units (CPUs) 1010 (the processor 1010 may include but is not limited to a Processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1030 for storing data, one or more storage media 1020 (e.g., one or more mass storage devices) for storing applications 1023 or data 1022. Memory 1030 and storage media 1020 may be, among other things, transient or persistent storage. The program stored in the storage medium 1020 may include one or more modules, each of which may include a series of instruction operations for a server. Still further, the central processor 1010 may be configured to communicate with the storage medium 1020 and execute a series of instruction operations in the storage medium 1020 on the server 1000. The server 1000 may also include one or more power supplies 1060, one or more wired or wireless network interfaces 1050, one or more input-output interfaces 1040, and/or one or more operating systems 1021, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so forth.
Input-output interface 1040 may be used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the server 1000. In one example, i/o Interface 1040 includes a Network adapter (NIC) that may be coupled to other Network devices via a base station to communicate with the internet. In one example, the input/output interface 1040 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
It will be understood by those skilled in the art that the structure shown in fig. 10 is merely illustrative and is not intended to limit the structure of the electronic device. For example, server 1000 may also include more or fewer components than shown in FIG. 10, or have a different configuration than shown in FIG. 10.
Embodiments of the present invention also provide a computer-readable storage medium, which may be disposed in a server to store at least one instruction, at least one program, a code set, or a set of instructions related to implementing a scenario testing method, where the at least one instruction, the at least one program, the code set, or the set of instructions are loaded and executed by a processor to implement the scenario testing method provided by the above method embodiments.
Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device, apparatus and server embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to the partial description of the method embodiments for relevant points.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A method for scenario testing, comprising:
acquiring a test task, and determining a scheme set to be tested and a test user set according to the test task;
generating a document list set according to the schemes in the scheme set to be tested based on preset operation instructions of the users for each user in the test user set, wherein the document lists in the document list set correspond to the schemes in the scheme set to be tested one by one;
merging the document lists in the document list set to generate a target document list;
sending the target document list to the user, and acquiring click data of the user aiming at the target document list;
and analyzing the click data of all the users in the test user set to determine a target scheme.
2. The method of claim 1, wherein analyzing click data of all users in the set of test users to determine a target solution comprises:
counting the number of document clicks of each user in the test user set on the document list in the document list set;
calculating the score of the scheme in the to-be-tested scheme set corresponding to the document list according to the number of the document clicks;
determining candidate schemes according to scores of the schemes in the scheme set to be tested;
and determining a target scheme according to the candidate scheme.
3. The method of claim 2, wherein the determining a target solution according to the candidate solution comprises:
randomly sampling click data of all users in the test user set for multiple times to obtain a sample data set;
analyzing the sample data in the sample data set, and determining a confidence interval;
and if the score of the candidate scheme is within the confidence interval, the candidate scheme is taken as a target scheme.
4. The method according to claim 1 or 2, wherein said merging the document lists in the document list set to generate a target document list comprises:
constructing a first document list which is empty initially;
randomly sequencing the document lists in the document list set to obtain a sequencing result;
sequentially selecting a target document from the document list according to the sorting result; if the target document is different from the documents in the first document list, adding the target document into the first document list;
and when all the documents in the document list are added into the first document list, taking the first document list as a target document list.
5. The method according to claim 1 or 2, wherein said merging the document lists in the document list set to generate a target document list comprises:
constructing a second document list which is empty initially;
randomly sequencing the document lists in the document list set to obtain a sequencing result;
according to the sorting result, sequentially selecting a target document from the document list, wherein the target document is different from the documents in the second document list; adding the target document into the second document list, so that the difference value between the number of the documents belonging to any two document lists in the second document list is smaller than or equal to a preset threshold value;
and when all the documents in the document list are added into the second document list, taking the second document list as a target document list.
6. The method of claim 1 or 2, wherein obtaining a test task, determining a set of solutions to be tested and a set of test users from the test task comprises:
determining flow configuration information according to the test task;
and determining a plurality of users through random sampling based on the flow configuration information, and generating the test user set.
7. The method of claim 6, wherein obtaining a test task, determining a set of solutions to be tested and a set of test users from the test task further comprises:
determining the limiting conditions of the users in the test user set according to the test task;
acquiring user figures of all users;
and screening the users determined by random sampling based on the user portrait so that the users in the test user set meet the limiting conditions.
8. A scenario testing apparatus, comprising:
the acquisition module is used for acquiring a test task and determining a scheme set to be tested and a test user set according to the test task;
a generating module, configured to generate, for each user in the test user set, a document list set according to a scheme in the to-be-tested scheme set based on a preset operation instruction of the user, where document lists in the document list set correspond to schemes in the to-be-tested scheme set one to one;
the merging module is used for merging the document lists in the document list set to generate a target document list;
the sending module is used for sending the target document list to the user and acquiring click data of the user aiming at the target document list;
and the analysis module is used for analyzing the click data of all the users in the test user set to determine a target scheme.
9. A server, comprising a processor and a memory, wherein at least one instruction, at least one program, set of codes, or set of instructions is stored in the memory, and wherein the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded and executed by the processor to implement the method according to any one of claims 1-7.
10. A computer readable storage medium having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910926314.7A CN112579422A (en) | 2019-09-27 | 2019-09-27 | Scheme testing method and device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910926314.7A CN112579422A (en) | 2019-09-27 | 2019-09-27 | Scheme testing method and device, server and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112579422A true CN112579422A (en) | 2021-03-30 |
Family
ID=75110004
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910926314.7A Pending CN112579422A (en) | 2019-09-27 | 2019-09-27 | Scheme testing method and device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579422A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113742577A (en) * | 2021-08-10 | 2021-12-03 | 深圳市东信时代信息技术有限公司 | AB test scheme processing method, device, equipment and storage medium based on SaaS |
| CN117093503A (en) * | 2023-10-17 | 2023-11-21 | 杭银消费金融股份有限公司 | Client test management method and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105610654A (en) * | 2016-03-02 | 2016-05-25 | 合一网络技术(北京)有限公司 | Server, and policy online test method and system |
| CN107402881A (en) * | 2017-04-14 | 2017-11-28 | 阿里巴巴集团控股有限公司 | The choosing method and device of a kind of project testing |
| CN107562632A (en) * | 2017-09-12 | 2018-01-09 | 北京奇艺世纪科技有限公司 | For the A/B method of testings and device of Generalization bounds |
| CN108510311A (en) * | 2018-02-28 | 2018-09-07 | 阿里巴巴集团控股有限公司 | A kind of method, apparatus and electronic equipment of determining marketing program |
| CN109614515A (en) * | 2018-10-30 | 2019-04-12 | 北京奇艺世纪科技有限公司 | Video search evaluation method and system |
| CN109948016A (en) * | 2017-10-31 | 2019-06-28 | 北京嘀嘀无限科技发展有限公司 | Application message method for pushing, device, server and computer readable storage medium |
-
2019
- 2019-09-27 CN CN201910926314.7A patent/CN112579422A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105610654A (en) * | 2016-03-02 | 2016-05-25 | 合一网络技术(北京)有限公司 | Server, and policy online test method and system |
| CN107402881A (en) * | 2017-04-14 | 2017-11-28 | 阿里巴巴集团控股有限公司 | The choosing method and device of a kind of project testing |
| CN107562632A (en) * | 2017-09-12 | 2018-01-09 | 北京奇艺世纪科技有限公司 | For the A/B method of testings and device of Generalization bounds |
| CN109948016A (en) * | 2017-10-31 | 2019-06-28 | 北京嘀嘀无限科技发展有限公司 | Application message method for pushing, device, server and computer readable storage medium |
| CN108510311A (en) * | 2018-02-28 | 2018-09-07 | 阿里巴巴集团控股有限公司 | A kind of method, apparatus and electronic equipment of determining marketing program |
| CN109614515A (en) * | 2018-10-30 | 2019-04-12 | 北京奇艺世纪科技有限公司 | Video search evaluation method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113742577A (en) * | 2021-08-10 | 2021-12-03 | 深圳市东信时代信息技术有限公司 | AB test scheme processing method, device, equipment and storage medium based on SaaS |
| CN117093503A (en) * | 2023-10-17 | 2023-11-21 | 杭银消费金融股份有限公司 | Client test management method and system |
| CN117093503B (en) * | 2023-10-17 | 2024-02-06 | 杭银消费金融股份有限公司 | Client test management method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7343568B2 (en) | Identifying and applying hyperparameters for machine learning | |
| CN107463704B (en) | Search method and device based on artificial intelligence | |
| CN108804641B (en) | Text similarity calculation method, device, equipment and storage medium | |
| US10621493B2 (en) | Multiple record linkage algorithm selector | |
| US11915104B2 (en) | Normalizing text attributes for machine learning models | |
| WO2018210440A1 (en) | Improved data clustering | |
| CN110413867B (en) | Method and system for content recommendation | |
| CN112800095A (en) | Data processing method, device, equipment and storage medium | |
| CN112579422A (en) | Scheme testing method and device, server and storage medium | |
| CN114490923A (en) | Training method, device and equipment for similar text matching model and storage medium | |
| US10853429B2 (en) | Identifying domain-specific accounts | |
| CN111651666A (en) | User theme recommendation method and device, computer equipment and storage medium | |
| CN116089367A (en) | Dynamic barrel dividing method, device, electronic equipment and medium | |
| CN111078972B (en) | Questioning behavior data acquisition method, questioning behavior data acquisition device and server | |
| CN110852078A (en) | Method and device for generating title | |
| CN114691875A (en) | Data classification and classification processing method and device | |
| CN115131058A (en) | Account identification method, device, equipment and storage medium | |
| CN113918796A (en) | Information searching method, device, server and storage medium | |
| CN112541069A (en) | Text matching method, system, terminal and storage medium combined with keywords | |
| US20220358152A1 (en) | Model performance through text-to-text transformation via distant supervision from target and auxiliary tasks | |
| CN110856253B (en) | Positioning method, positioning device, server and storage medium | |
| CN109635286B (en) | Policy hotspot analysis method and device, computer equipment and storage medium | |
| EP4109353A2 (en) | Method and apparatus of generating pre-training model, electronic device, and storage medium | |
| CN110457122B (en) | Task processing method, task processing device and computer system | |
| US20230195734A1 (en) | Machine learning enabled real time query handling system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |