CN108845936B - AB testing method and system based on massive users - Google Patents
AB testing method and system based on massive users Download PDFInfo
- Publication number
- CN108845936B CN108845936B CN201810547534.4A CN201810547534A CN108845936B CN 108845936 B CN108845936 B CN 108845936B CN 201810547534 A CN201810547534 A CN 201810547534A CN 108845936 B CN108845936 B CN 108845936B
- Authority
- CN
- China
- Prior art keywords
- group
- user access
- experimental group
- experimental
- product strategy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Abstract
The embodiment of the specification discloses an AB testing method and system based on massive users, which comprises the following steps: dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation; sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy; sending the successful product strategy to the second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group; and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of the successful product strategy.
Description
Technical Field
The embodiment scheme of the specification belongs to the technical field of webpage testing, and particularly relates to an AB testing method and system based on massive users.
Background
With the development of the internet, the attention degree on users is also increased, whether preferential activities or page revisions are carried out, an A/B test (namely AB test) is generally needed to verify whether a product strategy is feasible for the users, but the traditional AB test product strategy is to split the flow in half, and the product strategy is carried out on all users after the experimental effect is successful.
AB test is essentially a separate type inter-group experiment, and when performing the abest, a test page (defined as a B page) needs to be established, which may be different from the original page (defined as an a page) in the aspects of title font, background color, layout setting or wording, and the like, then the a page and the B page are pushed to all browsing users simultaneously in a random manner, and then the percentage of users who reach the inner page of the website through the a page and the percentage of users who reach the inner page through the B page are counted. Assuming that 6% of A and 20% of B, it means that the new page is preferred by the user. If the 20% result is not satisfactory, the new page can be modified further until the conversion cannot be increased any more. Therefore, the A/B test is actually a prior experiment system, belongs to a prediction conclusion, and is greatly different from the inductive conclusion of the posterior. The A/B test aims to obtain a representative experimental conclusion through scientific experimental design, sample representativeness, flow segmentation, small-flow test and the like, and the conclusion is ensured to be credible when being popularized to all flows.
However, the traditional A/B test method has shunt errors, the actual income generated by the project when 100% of flow is started after the experiment is successful is difficult to explain, and if 50% of flow is continuously reserved for comparison, the waste is caused.
Therefore, there is a need in the art for a solution that can avoid shunting errors and monitoring project revenues.
Disclosure of Invention
The embodiment of the specification aims to provide an AB testing method and system based on massive users, and the method and system aim to solve the problems of sampling errors, group traffic waste and difficulty in explaining actual income generated by a project when the traffic is fully opened by performing refined operation on user traffic.
In order to solve the above technical problem, an AB testing method and system based on massive users provided in an embodiment of the present specification are implemented in the following manner:
an AB testing method based on massive users comprises the following steps:
dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation;
sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy;
sending the successful product strategy to the second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group;
and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of the successful product strategy.
An AB testing system based on massive users, comprising:
the flow dividing unit is used for dividing the user access flow into a first experiment group, a second experiment group and a control group according to a preset proportional relation;
the first experiment unit is used for sending at least one product strategy to be tested to the first experiment group for cross validation and outputting a successful product strategy;
the second experiment unit is used for sending the successful product strategy to the second experiment group and supplementing the user access flow of the first experiment group to the second experiment group;
and the comparison unit is used for comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of the successful product strategy.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation;
sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy;
sending the successful product strategy to the second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group;
and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of the successful product strategy.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation;
sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy;
sending the successful product strategy to the second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group;
and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of the successful product strategy.
The AB testing method and system based on massive users can apply the cross validation method in statistical modeling to ABtest shunting, and solve the problems of sampling error, group traffic waste and difficulty in explaining actual income generated by projects when traffic is fully opened in the prior art by finely operating user traffic.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical product strategy in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without inventive efforts.
FIG. 1 is a process flow diagram of an embodiment of the method provided herein;
FIG. 2 is a schematic process flow diagram of another embodiment of the method provided herein;
FIG. 3 is a flowchart illustrating the processing of step S202 in one embodiment provided in the present specification;
FIG. 4 is a process flow diagram of multiple parallel experiments of product strategies in an embodiment provided herein;
FIG. 5 is a flowchart illustrating the processing of step S205 in one embodiment provided in the present specification;
FIG. 6 is a schematic structural diagram of an embodiment of an AB test system based on mass users provided in the present specification;
FIG. 7 is a schematic structural diagram of a first experimental unit in an embodiment provided in the present specification;
fig. 8 is a schematic structural diagram of a determining module in an embodiment provided in the present specification;
FIG. 9 is a schematic structural diagram of an AB test system based on a large number of users in an embodiment provided in the present specification;
fig. 10 is a schematic structural diagram of a verification unit in an embodiment provided in the present specification.
Detailed Description
In order to make those skilled in the art better understand the technical product strategy in the present application, the technical product strategy in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
One embodiment provided by the present description may be applied to a client/server system architecture. The client can comprise terminal equipment with a shooting function (at least with a shooting function), such as a smart phone, a tablet computer, an intelligent wearable device, a special shooting device and the like. The client can be provided with a communication module and can be in communication connection with a remote server to realize data transmission with the server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed system.
Technical term interpretation:
and (3) cross validation: methods commonly used in statistical modeling. The initial sample is divided into K sub-samples, one individual sub-sample is retained as data for the verification model, and the other K-1 samples are used for training. Cross validation is repeated K times, each sub-sample is validated once, the K results are averaged or other combinations are used, and a single estimate is obtained. The advantage of this method is that training and validation are performed repeatedly using randomly generated subsamples at the same time, with results validated once each time, with 10-fold cross validation being the most common.
Product strategy: the appearance of the product is referred to collectively as the appearance, embodiment, product concept, and the like. For example, the product policy may be: the product promotion method includes the following steps of product promotion activities, webpage appearance design, a new product and the like, and the application is not limited to the above.
The following describes an embodiment of the present disclosure with a specific application scenario as an example. Specifically, fig. 1 is a schematic flow chart of an embodiment of the AB testing method based on massive users provided in this specification. Although the present specification provides the method steps or apparatus structures as shown in the following examples or figures, more or less steps or modules may be included in the method or apparatus structures based on conventional or non-inventive efforts. In the case of steps or structures which do not logically have the necessary cause and effect relationship, the execution order of the steps or the block structure of the apparatus is not limited to the execution order or the block structure shown in the embodiments or the drawings of the present specification. When the described method or module structure is applied to a device, a server or an end product in practice, the method or module structure according to the embodiment or the figures may be executed sequentially or in parallel (for example, in a parallel processor or multi-thread processing environment, or even in an implementation environment including distributed processing and server clustering).
Of course, the following description of the embodiments does not limit other extensible solutions based on the present description. For example, in other implementation scenarios, the embodiments provided in this specification can also be applied to other implementation scenarios requiring AB testing. In a specific embodiment, as shown in fig. 1, in an embodiment of an AB testing method based on massive users provided in this specification, the method may include:
s101: and dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation. The user access flow refers to the number of user accesses required by a preset experiment.
S102: and sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy.
S103: and sending the successful product strategy to a second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group.
S104: and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of a successful product strategy.
According to the AB testing method based on the massive users, the user access flow is divided into a first experiment group, a second experiment group and a comparison group according to a preset proportional relation; sending at least one product strategy to be tested to a first experimental group for cross validation, and outputting a successful product strategy; sending the successful product strategy to a second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group; and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of a successful product strategy. As can be seen from the flow shown in fig. 1, the AB testing method based on massive users provided in the foregoing embodiment of the present description may avoid accidental errors through designing a cross validation step in the first experimental group, and determine that an experiment succeeds after an experiment significance rate of an experiment effect of multiple validation reaches a predetermined value; the idea of short-term experiment advance is adopted, the successful product strategy in the first experiment group is sent to the second experiment group, and waste of user access flow is effectively avoided; by setting the comparison group, the actual income condition of the project can be obtained by the comparison group at any time.
In order to make those skilled in the art better understand the present application, a more detailed embodiment is listed below, as shown in fig. 2, the above embodiment of the present specification provides an AB test method based on massive users, which includes:
s201: and dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation. The user access flow refers to the number of user accesses required by a preset experiment.
In specific implementation, the user access flow rate of the first experimental group is set to be the largest. Assuming that the preset user access flow is 100 thousands, the user access flow is divided into a first experiment group, a second experiment group and a control group according to the proportional relationship of 6:2:2, and then the user access flow of the first experiment group is 60 thousands, the user access flow of the second experiment group is 20 thousands and the user access flow of the control group is 20 thousands. The method is not only suitable for the split-flow mode of 6:2:2, but also has good effect on other split-flow ratios, and the method is not limited by the method. The first experimental group is mainly used for short-term experiments, the second experimental group is used for long-term experiments, and the control group is used for long-term control.
The method adopts the advanced idea of short-term experiment, the successful product strategy in the short-term experiment group is sent to the long-term experiment group, the flow opening of the long-term control group is only 20% or less, and the waste of power is avoided.
S202: and sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy.
As shown in fig. 3, S202 specifically includes:
s301: the first experimental group was divided into several groups, one of which was optionally a control group, and the remaining groups were experimental groups.
In specific implementation, for example, when a benefit needs to be done, 60% of the flow is started to enter a first experimental group (that is, the user access flow of the first experimental group is 60 ten thousand), the user access flow is uniformly divided into 6 groups in the first experimental group, the 1 st group to the 5 th group are experimental groups, the user access flow of each experimental group is 10 ten thousand, the last group is a control group, and the user access flow of the control group is 10 ten thousand.
S302: and sending the product strategy to be tested to an experimental group.
The number of the product strategies to be tested can be one or more, and if the product strategies to be tested are simultaneously sent to the experiment group, the product strategies to be tested are subjected to parallel experiments.
In specific implementation, as shown in fig. 4, n different product strategy experiments may be performed in parallel in the first experiment group, that is, each experiment group receives at least one product strategy for performing the experiment.
S303: and comparing the user access result of each experimental group with the user access result of the control group respectively, and outputting the experimental result of each experimental group.
In specific implementation, as shown in fig. 4, each experimental group compares the user access result with the user access result of the control group, and outputs the experimental result corresponding to the product policy received by each experimental group. For example, a parallel test experiment is performed on the preferential activity of the product policy 1-A and the adjustment of the product policy 2-page layout, the preferential activity of the product policy 1-A and the adjustment of the product policy 2-page layout are simultaneously sent to the users in the groups 1 to 5 (namely, experimental groups) to enable the users to see the preferential activity and the adjustment of the page layout of the product A, and the users in the group 6 (namely, a comparison group) cannot see the preferential activity and the adjustment of the page layout of the product A because the users do not receive the preferential activity and the adjustment of the page layout of the product policy 1-A. Next, the user access results of the 1 st to 5 th groups were compared with the user access results of the 6 th group-the control group, and the test results of the product policy 1 and the product policy 2 of each test group were output, as shown in table 1.
Wherein the experimental results comprise: significant (i.e., reaching the predetermined index of the product strategy to be tested) and insignificant (i.e., not reaching the predetermined index of the product strategy to be tested).
For the preferential activity of the product policy 1-A, the user access result of the 1 st group in the experimental group is as follows: the purchase rate of the product A reaches 80%, and the user visit result of the control group, namely the 6 th group, is as follows: the purchase rate of the product A is 20%, and the experimental result of the preferential activity of the product strategy 1-product A in the group 1 is that the purchase rate is improved by 60%. The experimental result is considered to be significant if the experimental result of the preferential activity of the product strategy 1-a is improved by 30%, so the experimental result of the 1 st group in the experimental group is significant. And the other experimental groups output the experimental results in turn as obvious or not according to the product strategies.
For the product policy 2-page layout adjustment, the user access results for group 1 in the experimental group are: the rate of the users entering the inner page reaches 80%, and the access result of the users of the control group, namely the 6 th group, is as follows: the rate of the user entering the inner page reaches 30%, and the experimental result of the product strategy 2-page layout adjustment in the group 1 is that the rate of the user entering the inner page is improved by 50%. And setting that if the experimental result of the product strategy 2-page layout adjustment is that the ratio of the user entering the inner page is improved by 20%, the experimental result is considered to be obvious, so that the experimental result of the 1 st group in the experimental groups is obvious. And the other experimental groups output the experimental results in turn as obvious or not according to the product strategies.
TABLE 1
S304: generating the significance rate of the strategy of the product to be tested according to the experimental result of each experimental group;
in specific implementation, as shown in fig. 4, the significance rate is set to X%, and the significance rates of the product strategy 1 and the product strategy 2 are calculated from the experimental results of the experimental groups in table 1.
The significance rate of product strategy 1 was: the experimental result of the product strategy 1 among the 5 experimental groups is the ratio of 5 significant experimental groups, i.e., (5/5) 100% to 100%, and thus the significant rate of the experimental result of the product strategy 1 is X% to 100%.
The significance of product strategy 2 was: the product strategy 2 among the 5 experimental groups had a significant ratio of 2 experimental groups, (2/5) of 100% to 40%, and thus the product strategy 2 had a significant ratio of X% to 40%.
S305: and judging whether the product strategy to be tested is successful according to the significance rate, and outputting the successful product strategy.
In specific implementation, judging whether the significance rate is greater than or equal to a preset value; and outputting the product strategy with the significance rate more than or equal to the preset value, namely a successful product strategy. When the significance rate of the experimental result of the product strategy to be tested is greater than the preset value, the product strategy is a successful product strategy, and the successful product strategy is output; when the significance rate of the experimental result of the product strategy to be tested is smaller than the preset value, the product strategy fails and needs to be improved.
The preset value is set to 90%, and according to the results in table 1 and S304, it can be known that the significance rate of the product strategy 1 is 100% greater than the preset 90%, that is, the experimental result of the product strategy 1 of the experimental group is significant, the product strategy 1 is a successful product strategy, and the first experimental group outputs the product strategy 1. The significance rate of the product strategy 2 is 40% less than 90%, namely, only 40% of the experimental results of the product strategy 2 of the experimental group are significant, the product strategy 2 is a failed product strategy, and the product strategy 2 is to be improved.
In the embodiment, the cross validation step is designed in a short-term experimental group (namely, a first experimental group), multiple times of validation are carried out to avoid accidental errors, namely, the user access flow is uniformly divided into n parts, n-1 parts are used as the experimental group, the rest is used as a control group, the experimental effect is verified for multiple times, and the product strategy to be tested is judged to be successful after the experimental significance rate reaches the preset value.
S203: and sending the successful product strategy to a second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group.
In specific implementation, the successful product policy 1 is sent to the second experimental group, and 60% of the user access flow of the first experimental group is supplemented to the second experimental group, so that the user access flow of the second experimental group is 20% + 60% ═ 80%, that is, 80 ten thousand, and at this time, 80 ten thousand users can receive the preferential activity of the product policy 1-a product.
S204: and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of a successful product strategy.
In specific implementation, the user access flow allocated to the control group is 20%, namely 20 ten thousand, and the user access flow is used for forming a control with the user access result of the first experimental group and explaining the actual benefit of a successful product strategy. The 80 general users of the second experimental group receive the preferential activities of the product policy 1-a product, the access result of the user is set as the product purchase rate a in this embodiment, and the access results of the 80 general users of the second experimental group are: the purchase rate of the product A reaches 80%, and the user access results of 20 ten thousand users in the control group are as follows: the purchase rate of the product a is 20%, the yield of the preferential activity of the product receiving the successful product strategy 1-a is increased by 60%, and the application is not limited to this.
The method and the device have the advantages that through the second experiment group and the contrast group, the situation that the actual income of the project cannot be explained due to the influence of light and busy seasons of the business is avoided, and the waste that the traditional ABtest opens 50% of user access flow to serve as the contrast group is avoided. Meanwhile, the long-term control group is arranged, namely only a small part of flow is reserved as a control after the experiment is successful, so that the actual income condition of the project where the product strategy is located can be obtained through the long-term control group (namely the control group) at any time.
S205: and checking whether the flow division of the user access flow is uniform.
As shown in fig. 5, S205 specifically includes the following steps:
s401: the control component is divided into two flow check groups with equal user access flow.
In specific implementation, the 20 universal user access flows of the control group are divided into a first flow test group and a second flow test group, and the user access flows of the first flow test group and the second flow test group are 10 ten thousand.
S402: and judging whether the flow division of the user access flow is uniform or not according to the user access results of the two flow detection groups.
In specific implementation, an A-A inspection method is adopted, and the specific inspection process is as follows:
taking the successful product policy 1-A product preferential activity as an example, counting the purchase rates of the product A of 10 ten thousand users of the first traffic test group and the product A of 10 ten thousand users of the second traffic test group, wherein the user access results of the first traffic test group are as follows: the purchase rate of the product A is 30.3 percent; the user access results of the first traffic verification group are: the purchase rate of the product A is 30.3 percent. Obviously, the user access results of the first traffic check group and the second traffic check group are substantially consistent, and the division of the user access traffic in step S201 is considered to be uniform, which is not limited in this application.
According to the embodiment, the method and the device for testing the access flow of the user ensure the uniformity of the access flow of the user and the accuracy of the test effect through A-A inspection.
In the present specification, each embodiment of the method is described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. Reference is made to the description of the method embodiments.
The method adopts the cross validation method in the statistical modeling to be applied to the shunting of the AB test, and solves the accidental errors caused by shunting in the shunting of the traditional AB test. Meanwhile, the idea of short-term experiment foreward is adopted, the successful product strategy in the short-term experiment group is sent to the long-term experiment group, the flow opening of the long-term comparison group is only 20% or less, waste is avoided, the long-term comparison group is arranged, the actual income condition of the project can be obtained through the long-term comparison group at any time, and the influence of seasonal fluctuation is avoided.
Based on the same application concept as the AB test method based on massive users, the present application also provides an AB test system based on massive users, as described in the following embodiments. Because the principle of solving the problems of the AB test system based on the mass users is similar to that of the AB test method based on the mass users, the implementation of the AB test system based on the mass users can refer to the implementation of the AB test method based on the mass users, and repeated parts are not described again.
Fig. 6 is a schematic structural diagram of an AB testing system based on mass users according to an embodiment of the present application, and as shown in fig. 6, the AB testing system based on mass users includes: a flow dividing unit 101, a first experiment unit 102, a second experiment unit 103, and a comparison unit 104.
The traffic dividing unit 101 is configured to divide the user access traffic into a first experiment group, a second experiment group, and a control group according to a preset proportional relationship. Wherein the user access traffic of the first experimental group is a ratio of maximum.
The first experiment unit 102 is configured to send at least one product policy to be tested to the first experiment group for cross validation, and output a successful product policy.
And the second experiment unit 103 is configured to send the successful product policy to the second experiment group, and supplement the user access traffic of the first experiment group to the second experiment group.
And the comparing unit 104 is configured to compare the user access result of the control group with the user access result of the second experimental group, so as to obtain the benefit of a successful product policy.
In one embodiment, as shown in FIG. 7, the first experiment unit 102 includes: the system comprises a group division module 201, a product strategy sending module 202, a comparison module 203, an output module 204 and a judgment module 205.
A group division module 201, configured to divide the first experimental group into a plurality of groups, wherein any one group is a control group, and the rest groups are experimental groups;
the product strategy sending module 202 is used for sending the product strategy to be tested to an experimental group;
the comparison module 203 is used for comparing the user access results of the experimental groups with the user access results of the control group respectively and outputting the experimental results of the experimental groups;
the output module 204 is used for generating the significance rate of the strategy of the product to be tested according to the experimental result of each experimental group;
the judging module 205 is configured to judge whether the product policy to be tested is successful according to the significance rate, and output a successful product policy.
In one embodiment, as shown in FIG. 8, the determining module 205 comprises: judgment submodule and output submodule
The judgment submodule 301 is configured to judge whether the significance rate is greater than or equal to a preset value;
and the output sub-module 302 is used for outputting the product strategy with the significance rate being greater than or equal to the preset value.
In one embodiment, as shown in fig. 9, the system further comprises: a checking unit 105 for checking whether the traffic division of the user access traffic is uniform.
In one embodiment, as shown in fig. 10, the inspection unit 105 includes: a traffic division module 401 and a result judgment module 402.
And the flow dividing module 401 is used for dividing the comparison component into two flow check groups with equal user access flow.
And the result judging module 402 is configured to judge whether the traffic division of the user access traffic is uniform according to the user access results of the two traffic inspection groups.
Based on the same application concept as the above-mentioned AB test method based on massive users, the present application provides a computer device, as described in the following embodiments. Because the principle of solving the problems of the computer equipment is similar to the AB testing method based on the mass users, the implementation of the computer equipment can refer to the implementation of the AB testing method based on the mass users, and repeated parts are not described again.
In one embodiment, a computer device comprises: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program, as shown in fig. 1:
s101: and dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation.
S102: and sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy.
S103: and sending the successful product strategy to a second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group.
S104: and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of a successful product strategy.
Based on the same application concept as the above-mentioned AB test method based on massive users, the present application provides a computer-readable storage medium, as described in the following embodiments. Because the principle of solving the problems of the computer-readable storage medium is similar to the AB test method based on the mass users, the implementation of the computer-readable storage medium can refer to the implementation of the AB test method based on the mass users, and repeated parts are not described again.
In one embodiment, a computer readable storage medium has stored thereon a computer program, as shown in FIG. 1, which when executed by a processor, performs the steps of:
s101: and dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation.
S102: and sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy.
S103: and sending the successful product strategy to a second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group.
S104: and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of a successful product strategy.
According to the AB testing method and system based on massive users, the user access flow is divided into a first experimental group, a second experimental group and a comparison group according to a preset proportional relation; sending at least one product strategy to be tested to a first experimental group for cross validation, and outputting a successful product strategy; sending the successful product strategy to a second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group; and comparing the user access result of the control group with the user access result of the second experimental group to obtain the income of a successful product strategy. As can be seen from the flow shown in fig. 1, the cross validation step is designed in the first experimental group, so that accidental errors are avoided through multiple validation, and the success of the experiment is judged after the experiment significance rate of the multiple validation experiment effect reaches a predetermined value; the idea of short-term experiment advance is adopted, the successful product strategy in the first experiment group is sent to the second experiment group, and waste of user access flow is effectively avoided; by setting the comparison group, the actual income condition of the project can be obtained by the comparison group at any time.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the hardware + program class embodiment, since it is substantially similar to the method embodiment, the description is simple, and the relevant points can be referred to the partial description of the method embodiment.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Although the present application provides method steps as described in an embodiment or flowchart, additional or fewer steps may be included based on conventional or non-inventive efforts. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an actual apparatus or client product executes, it may execute sequentially or in parallel (e.g., in the context of parallel processors or multi-threaded processing) according to the embodiments or methods shown in the figures.
In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually making an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development and writing, but the original code before compiling is also written by a specific Programming Language, which is called Hardware Description Language (HDL), and HDL is not only one but many, such as abel (advanced Boolean Expression Language), ahdl (alternate Hardware Description Language), traffic, pl (core universal Programming Language), HDCal (jhdware Description Language), lang, Lola, HDL, laspam, hardward Description Language (vhr Description Language), vhal (Hardware Description Language), and vhigh-Language, which are currently used in most common. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. An AB testing method based on massive users, the method comprising:
dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation;
sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy;
sending the successful product strategy to the second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group;
comparing the user access result of the control group with the user access result of the second experimental group to obtain the benefit of the successful product strategy;
the user access flow of the first experimental group is the largest;
the sending of the at least one product strategy to be tested to the first experimental group for cross validation and output of a successful product strategy comprises:
dividing the first experimental group into a plurality of groups, wherein one group is selected as a control group, and the rest groups are experimental groups;
sending the product strategy to be tested to the experimental group;
comparing the user access result of each experimental group with the user access result of the control group respectively, and outputting the experimental result of each experimental group;
generating the significance rate of the product strategy to be tested according to the experimental result of each experimental group;
and judging whether the product strategy to be tested is successful according to the significance rate, and outputting the successful product strategy.
2. The AB testing method based on massive users according to claim 1, judging whether the product strategy to be tested is successful according to the significance rate, and outputting a successful product strategy, including;
judging whether the significance rate is greater than or equal to a preset value or not;
and outputting the product strategy with the significance rate more than or equal to the preset value.
3. The AB testing method based on massive users according to claim 1, further comprising: and checking whether the flow division of the user access flow is uniform or not.
4. The AB testing method based on massive users according to claim 3, wherein the step of checking whether the flow division of the user access flow is uniform comprises the steps of:
the control component is two flow detection groups with equal user access flow;
and judging whether the flow division of the user access flow is uniform or not according to the user access results of the two flow detection groups.
5. An AB testing system based on a large number of users, the system comprising:
the flow dividing unit is used for dividing the user access flow into a first experiment group, a second experiment group and a control group according to a preset proportional relation;
the first experiment unit is used for sending at least one product strategy to be tested to the first experiment group for cross validation and outputting a successful product strategy;
the second experiment unit is used for sending the successful product strategy to the second experiment group and supplementing the user access flow of the first experiment group to the second experiment group;
a comparison unit, configured to compare the user access result of the control group with the user access result of the second experimental group, so as to obtain the benefit of the successful product policy;
the user access flow of the first experimental group is the largest;
the first experimental unit includes:
the group dividing module is used for dividing the first experimental group into a plurality of groups, wherein any one group is a control group, and the rest groups are experimental groups;
the product strategy sending module is used for sending the product strategy to be tested to the experimental group;
the comparison module is used for comparing the user access results of the experimental groups with the user access results of the control group respectively and outputting the experimental results of the experimental groups;
the output module is used for generating the significance rate of the strategy of the product to be tested according to the experimental result of each experimental group;
and the judging module is used for judging whether the product strategy to be tested is successful according to the significance rate and outputting the successful product strategy.
6. The AB testing system based on massive users according to claim 5, wherein the judging module comprises:
the judgment submodule is used for judging whether the significance rate is greater than or equal to a preset value or not;
and the output submodule is used for outputting the product strategy with the significance rate more than or equal to the preset value.
7. The mass user-based AB test system as claimed in claim 5, said system further comprising: and the checking unit is used for checking whether the flow division of the user access flow is uniform or not.
8. The AB mass users based test system of claim 7, the verification unit comprising:
the flow dividing module is used for dividing the comparison component into two flow detection groups with equal user access flow;
and the result judging module is used for judging whether the flow division of the user access flow is uniform or not according to the user access results of the two flow detection groups.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation;
sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy;
sending the successful product strategy to the second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group;
comparing the user access result of the control group with the user access result of the second experimental group to obtain the benefit of the successful product strategy;
the user access flow of the first experimental group is the largest;
the sending of the at least one product strategy to be tested to the first experimental group for cross validation and output of a successful product strategy comprises:
dividing the first experimental group into a plurality of groups, wherein one group is selected as a control group, and the rest groups are experimental groups;
sending the product strategy to be tested to the experimental group;
comparing the user access result of each experimental group with the user access result of the control group respectively, and outputting the experimental result of each experimental group;
generating the significance rate of the product strategy to be tested according to the experimental result of each experimental group;
and judging whether the product strategy to be tested is successful according to the significance rate, and outputting the successful product strategy.
10. A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
dividing the user access flow into a first experimental group, a second experimental group and a control group according to a preset proportional relation;
sending at least one product strategy to be tested to the first experimental group for cross validation, and outputting a successful product strategy;
sending the successful product strategy to the second experimental group, and supplementing the user access flow of the first experimental group to the second experimental group;
comparing the user access result of the control group with the user access result of the second experimental group to obtain the benefit of the successful product strategy;
the user access flow of the first experimental group is the largest;
the sending of the at least one product strategy to be tested to the first experimental group for cross validation and output of a successful product strategy comprises:
dividing the first experimental group into a plurality of groups, wherein one group is selected as a control group, and the rest groups are experimental groups;
sending the product strategy to be tested to the experimental group;
comparing the user access result of each experimental group with the user access result of the control group respectively, and outputting the experimental result of each experimental group;
generating the significance rate of the product strategy to be tested according to the experimental result of each experimental group;
and judging whether the product strategy to be tested is successful according to the significance rate, and outputting the successful product strategy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810547534.4A CN108845936B (en) | 2018-05-31 | 2018-05-31 | AB testing method and system based on massive users |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810547534.4A CN108845936B (en) | 2018-05-31 | 2018-05-31 | AB testing method and system based on massive users |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108845936A CN108845936A (en) | 2018-11-20 |
| CN108845936B true CN108845936B (en) | 2021-10-01 |
Family
ID=64210285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810547534.4A Active CN108845936B (en) | 2018-05-31 | 2018-05-31 | AB testing method and system based on massive users |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108845936B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109739757A (en) * | 2018-12-28 | 2019-05-10 | 微梦创科网络科技（中国）有限公司 | A kind of AB test method and device |
| CN109829757B (en) * | 2019-01-18 | 2021-02-26 | 泰康保险集团股份有限公司 | Optimal marketing scheme selection method and device |
| CN110034953A (en) * | 2019-03-12 | 2019-07-19 | 平安城市建设科技(深圳)有限公司 | Analysis method, device, equipment and the storage medium of tactful chain in being tested between group |
| CN110046296A (en) * | 2019-03-12 | 2019-07-23 | 平安城市建设科技(深圳)有限公司 | Processing method, device, equipment and the storage medium of experimental data |
| CN110324252A (en) * | 2019-05-28 | 2019-10-11 | 成都美美臣科技有限公司 | A kind of multivariable modification A B test flow allocation method |
| CN112148582B (en) * | 2019-06-27 | 2023-10-27 | 腾讯科技（深圳）有限公司 | Policy testing method and device, computer readable medium and electronic equipment |
| CN112749076B (en) * | 2019-10-30 | 2023-10-20 | 腾讯科技（深圳）有限公司 | Test method and device and electronic equipment |
| CN111324533B (en) * | 2020-02-17 | 2022-10-18 | 支付宝(杭州)信息技术有限公司 | A/B test method and device and electronic equipment |
| CN111311336B (en) * | 2020-03-17 | 2023-07-04 | 北京嘀嘀无限科技发展有限公司 | Method and system for testing and tracking policy execution |
| CN111460384B (en) * | 2020-03-31 | 2023-09-08 | 北京百度网讯科技有限公司 | Policy evaluation method, device and equipment |
| CN111784173B (en) * | 2020-07-09 | 2022-06-03 | 支付宝(杭州)信息技术有限公司 | AB experiment data processing method, device, server and medium |
| CN112465604A (en) * | 2020-12-14 | 2021-03-09 | 深圳依时货拉拉科技有限公司 | Order distribution method and device, computer equipment and computer readable storage medium |
| CN112734502B (en) * | 2021-03-29 | 2021-06-18 | 腾讯科技（深圳）有限公司 | Testing method and device for multimedia information directional delivery and electronic equipment |
| CN113254882B (en) * | 2021-06-07 | 2023-02-10 | 广州市百果园网络科技有限公司 | Method, device and equipment for determining experimental result and storage medium |
| CN113298573A (en) * | 2021-06-18 | 2021-08-24 | 北京有竹居网络技术有限公司 | Content delivery strategy comparison method and device, readable medium and electronic equipment |
| CN113791978B (en) * | 2021-09-13 | 2023-07-11 | 网易（杭州）网络有限公司 | Target control sample acquisition method and strategy detection method |
| CN115048284B (en) * | 2022-03-24 | 2024-01-23 | 上海任意门科技有限公司 | Method, computing device and storage medium for testing applications of a system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095081A (en) * | 2015-08-06 | 2015-11-25 | 小米科技有限责任公司 | Application program function testing method and device |
| CN105373480A (en) * | 2015-12-15 | 2016-03-02 | 广州唯品会信息科技有限公司 | ABTest system shunt optimizing method and system |
| CN105913145A (en) * | 2016-04-08 | 2016-08-31 | 北京吆喝科技有限公司 | Data driving-based AB test method |
| CN107045673A (en) * | 2017-03-31 | 2017-08-15 | 杭州电子科技大学 | Public bicycles changes in flow rate amount Forecasting Methodology based on heap Model Fusion |
| CN107451020A (en) * | 2017-06-28 | 2017-12-08 | 北京五八信息技术有限公司 | A kind of AB test systems and method of testing |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150012852A1 (en) * | 2013-07-08 | 2015-01-08 | Kobo Incorporated | User interface tool for planning an ab type of test |
| CN106713015B (en) * | 2016-12-07 | 2019-06-04 | 武汉斗鱼网络科技有限公司 | A kind of method and server of scheme test |
-
2018
- 2018-05-31 CN CN201810547534.4A patent/CN108845936B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095081A (en) * | 2015-08-06 | 2015-11-25 | 小米科技有限责任公司 | Application program function testing method and device |
| CN105373480A (en) * | 2015-12-15 | 2016-03-02 | 广州唯品会信息科技有限公司 | ABTest system shunt optimizing method and system |
| CN105913145A (en) * | 2016-04-08 | 2016-08-31 | 北京吆喝科技有限公司 | Data driving-based AB test method |
| CN107045673A (en) * | 2017-03-31 | 2017-08-15 | 杭州电子科技大学 | Public bicycles changes in flow rate amount Forecasting Methodology based on heap Model Fusion |
| CN107451020A (en) * | 2017-06-28 | 2017-12-08 | 北京五八信息技术有限公司 | A kind of AB test systems and method of testing |
Non-Patent Citations (1)
| Title |
|---|
| 基于TAM分组策略的SoC测试多目标优化设计;谈恩民 等;《微电子学与计算机》;20131030(第10期);69-72 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108845936A (en) | 2018-11-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108845936B (en) | AB testing method and system based on massive users | |
| CN107450979B (en) | Block chain consensus method and device | |
| CN108305158B (en) | Method, device and equipment for training wind control model and wind control | |
| CN108537568B (en) | Information recommendation method and device | |
| CN109062782B (en) | Regression test case selection method, device and equipment | |
| CN107577697B (en) | Data processing method, device and equipment | |
| CN110378400B (en) | Model training method and device for image recognition | |
| CN110674188A (en) | Feature extraction method, device and equipment | |
| CN110635962B (en) | Abnormity analysis method and device for distributed system | |
| CN115981870A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN107391564B (en) | Data conversion method and device and electronic equipment | |
| CN113079201B (en) | Information processing system, method, device and equipment | |
| CN109376988B (en) | Service data processing method and device | |
| CN112035341A (en) | Automatic testing method and device | |
| CN108804563B (en) | Data labeling method, device and equipment | |
| CN107368281B (en) | Data processing method and device | |
| CN107943923B (en) | Telegram code database construction method, telegram code identification method and device | |
| CN110874322B (en) | Test method and test server for application program | |
| CN107562533B (en) | Data loading processing method and device | |
| CN114710318B (en) | Method, device, equipment and medium for limiting high-frequency access of crawler | |
| CN107544753B (en) | Data processing method and device and server | |
| CN115130621A (en) | Model training method and device, storage medium and electronic equipment | |
| CN109903165B (en) | Model merging method and device | |
| CN110096477B (en) | Data storage method and device | |
| CN114817212A (en) | Database optimization method and optimization device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20201022 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20201022 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: Greater Cayman, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |