CN108415845A - AB tests computational methods, device and the server of system index confidence interval - Google Patents
AB tests computational methods, device and the server of system index confidence interval Download PDFInfo
- Publication number
- CN108415845A CN108415845A CN201810266136.5A CN201810266136A CN108415845A CN 108415845 A CN108415845 A CN 108415845A CN 201810266136 A CN201810266136 A CN 201810266136A CN 108415845 A CN108415845 A CN 108415845A
- Authority
- CN
- China
- Prior art keywords
- value
- global
- difference value
- numerical value
- division
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to field of computer technology, disclose a kind of computational methods, device and the server of AB tests system index confidence interval.The computational methods of AB test system index confidence intervals include the following steps：The global division of flow progress the first preset value time that AB is tested to system administration, obtains each global lower corresponding several groups being divided into of division, and each overall situation is mutually parallel between dividing；It calculates each corresponding several of global division and organizes corresponding index values, obtain each global several numerical value for dividing respective numbers；According to the numerical value randomly selected from each global several numerical value divided, the corresponding difference value of the second preset value packet size in each global division is obtained；Merge the corresponding difference value of the second preset value packet size described in each global division；According to the difference value after merging, confidence interval is obtained.The present invention improves the accuracy of confidence interval itself, and more reliable support is provided to the confidence level of AB test system index variations.
Description
Technical field
The present invention relates to field of computer technology, and the calculating side of system index confidence interval is tested more particularly to a kind of AB
Method, device and server.
Background technology
With the deep development of the Internet, applications, the mankind have gradually strided forward the new of information huge explosion from the epoch of absence of information
In epoch, the consulting of vast as the open sea news and the tinkling of jades video information met the eye on every side allow the people to roam leisurely in network world on internet
It has no way of selecting.Under the historical background of information huge explosion, effective and invalid information is simultaneously deposited, this causes people to obtain to oneself beneficial to letter
The cost of breath increases, thus how more accurately help user get advantageous information at Internet era core technology it
One, also exactly under this overall background, commending system comes into being.The core mission of commending system be exactly as much as possible to
User provides information that it is liked or beneficial to its, and user is avoided to be submerged in the ocean of invalid information.Commending system exists
Internet news, short-sighted frequency, the fields such as e-commerce are widely used, and are a kind of systems of dynamic change, by obtaining user couple
The behavioural information of system institute recommendation continues to optimize inner strategy and algorithm, is promoted and recommends quality and user experience, makes user
It is benefited.Wherein, commending system recommend quality optimization realized by continual algorithm and Policy iteration, and algorithm and
The iteration of strategy is completed by AB tests again.AB tests refer to algorithm A and B in commending system, it is desirable to assess A and B
Two algorithm which effects are more preferable, need first to define a series of indexs, such as clicking rate etc., then by different user groups
It uses algorithm A and algorithm B into commending contents on line respectively, and assesses the index value in the group for having used special algorithm, use
To judge algorithm A and algorithm B, which is better and which is worse.AB tests are a kind of assessment algorithm or tactful effect, realize algorithm or Policy iteration
Core methed, AB test system be exactly implement AB testing schemes a kind of system.The experiment implementer of system is tested using AB
Quality of the reliable achievement data for judging used algorithm in different user groups is provided.Since different user groups are using
In AB test systems, there may be antipodal variation tendencies for the index value in different user groups, then so that can not be direct
Changed come for judging algorithm A and algorithm B, which is better and which is worse according to the index value in different user groups.
For the above problem, in the prior art, confidence interval is applied in the index analysis that AB tests system, is passed through
The confidence interval of the change rate of each index is calculated separately, the confidence level of the change rate of each index can be obtained, by each
The confidence level of indicator difference is just more credible to judge the change rate of which index, so to judge algorithm A and algorithm B who
It is excellent that who is bad.And there are two types of the computational methods of confidence interval, one is in actual AB tests, due to needing to seek confidence interval
The distribution of index is unknown under normal circumstances, then firstly the need of an overall distribution is first assumed, is then obtained by statistical method
The complex distributions met to an index, reconstruct statistic, obtain relevant parameter, but in actual conditions, the finger respectively obeyed
Mark distribution may be variant, therefore the confidence interval of all indexs can not be solved with unified solution, and needs to refer to one by one
Mark is calculated according to specific condition, and cost is very big, and use is also inconvenient；Secondly using bootstrap methods, gone with experience distribution close
Like really distribution, but this method only carries out a global traffic for AB tests when starting and divides, and is then divided into this
Standard carries out the calculating of follow-up confidence interval so that the confidence interval acquired using the experience distribution of the index of simulation is partially narrow, leads
It causes difference of the index value in improved two orthogonal groupings without any experimental strategy more notable, is easy to have misled experiment
Judgement of the implementer to indicator difference credibility.
In this regard, there is an urgent need for propose a kind of method for seeking high accuracy confidence interval.
Invention content
The present invention provides AB computational methods, device and the server of test system index confidence interval, improves confidence area
Between the accuracy of itself, to AB test system index variation confidence level more reliable support is provided.
In a first aspect, the present invention provides the computational methods that a kind of AB tests system index confidence interval, including following step
Suddenly：
Flow progress the first preset value time overall situation that AB is tested to system administration divides, and obtains each global lower correspondence of division
Several groups being divided into are mutually parallel between described each time global division；
It calculates each corresponding several of global division and organizes corresponding index values, obtain each overall situation and divide respective numbers
Several numerical value；
According to the numerical value randomly selected from each global several numerical value divided, obtain each time it is global divide in second in advance
If the corresponding difference value of value packet size；
Merge the corresponding difference value of the second preset value packet size described in each global division；
According to the difference value after merging, confidence interval is obtained.
With reference to first aspect, the present invention is described to merge each global division in the first embodiment of first aspect
Described in the corresponding difference value of the second preset value packet size, include the following steps：
The corresponding difference value of the second preset value packet size described in each global division is integrated, third preset value is obtained
The corresponding difference value of a packet size；
By the corresponding difference value ascending order arrangement of the third preset value packet size, ordered series of numbers is obtained；
Wherein, the third preset value is first the second preset values of preset value *.
The first embodiment with reference to first aspect, the present invention are described in second of embodiment of first aspect
Numerical value after being merged according to the corresponding difference value of the second preset value packet size described in each global division, obtains confidence area
Between, include the following steps：
Confidence level is multiplied by the third preset value, the numerical value obtained after rounding is as index value；
The difference value that will be found in the ordered series of numbers according to the index value, as the corresponding difference of the confidence level
Value；
According to the corresponding difference value of the confidence level, confidence interval is obtained.
With reference to first aspect, the present invention in the third embodiment of first aspect, from each overall situation draw by the basis
The numerical value randomly selected in several numerical value divided obtains the corresponding difference of the second preset value packet size in each global division
Value, includes the following steps：
According to the numerical value randomly selected from each global several numerical value divided, obtain single in each global division
The corresponding difference value of packet size；
The numerical value that the basis is randomly selected from each global several numerical value divided is recycled, each overall situation is calculated and draws
In point the step of single packet size corresponding difference value, until obtaining the second preset value packet size in each global division
Corresponding difference value.
The third embodiment with reference to first aspect, the present invention are described in the 4th kind of embodiment of first aspect
According to the numerical value randomly selected from each global several numerical value divided, single packet size in each global division is calculated
Corresponding difference value, includes the following steps：
To randomly selecting the summation of the 4th preset value numerical value from each global several numerical value divided, each time is obtained entirely
Office divide in first sample and；
To randomly selecting the summation of the 4th preset value numerical value from each global several remaining numerical value divided, obtain
Each global the second sample divided and；
1 is subtracted the value of (the second sample with divided by first sample with) as each time it is global divide in single packet size it is corresponding
Difference value.
With reference to first aspect, the present invention is described that AB is tested system administration in the 5th kind of embodiment of first aspect
Flow carry out during time parallel global of the first preset value divide, whole users are divided into different groups by each global division
In, and each user is pertaining only to a group.
Second aspect, the present invention provide a kind of computing device of AB tests system index confidence interval, comprise the following modules：
Division module, the flow for AB to be tested to system administration carry out the first preset value time overall situation and divide, obtain each time
The overall situation divides lower corresponding several groups being divided into, and each overall situation is mutually parallel between dividing；
First computing module is organized corresponding index values for calculating each corresponding several of global division, is obtained each time
The overall situation divides several numerical value of respective numbers；
Second computing module, for according to the numerical value randomly selected from each global several numerical value divided, obtaining each
The corresponding difference value of second preset value packet size in secondary global division；
Merging module, for merging the corresponding difference value of the second preset value packet size described in each global division；
Third computing module, for according to the difference value after merging, obtaining confidence interval.
In conjunction with second aspect, for the present invention in the first embodiment of second aspect, the merging module includes following
Unit：
Integral unit, for integrating the corresponding difference value of the second preset value packet size described in each global division,
Obtain the corresponding difference value of third preset value packet size；
Sequencing unit obtains ordered series of numbers for arranging the corresponding difference value ascending order of the third preset value packet size；
Wherein, the third preset value is first the second preset values of preset value *.
In conjunction with the first embodiment of second aspect, the present invention is described in second of embodiment of second aspect
Third computing module includes with lower unit：
Indexing units, for confidence level to be multiplied by the third preset value, the numerical value obtained after rounding is as index value；
Searching unit, the difference value for will be found in the ordered series of numbers according to the index value, as the confidence
Horizontal corresponding difference value；
Computing unit, for according to the corresponding difference value of the confidence level, obtaining confidence interval.
In conjunction with second aspect, in the third embodiment of second aspect, second computing module includes the present invention
With lower unit：
Difference value computing unit, for according to the numerical value randomly selected from each global several numerical value divided, obtaining
To the corresponding difference value of single packet size in each global division；
Cycling element, the numerical value randomly selected from each global several numerical value divided for recycling the basis,
Calculate each time it is global divide in single packet size corresponding difference value the step of, until obtain each time it is global divide in second in advance
If the corresponding difference value of value packet size.
In conjunction with the third embodiment of second aspect, the present invention is described in the 4th kind of embodiment of second aspect
Difference value computing unit includes following subelement：
First summation subelement, for randomly selecting the 4th preset value from each global several numerical value divided
Numerical value is summed, obtain each time it is global divide in first sample with；
Second summation subelement, for pre- to randomly selecting the 4th from each global several remaining numerical value divided
If value numerical value summation, obtain each global the second sample divided with；
Difference value computation subunit, the value for subtracting (the second sample and divided by first sample and) using 1 are global as each time
The corresponding difference value of single packet size in division.
In conjunction with second aspect, for the present invention in the 5th kind of embodiment of second aspect, the division module includes being used for
The global division of flow progress the first preset value time that AB tests system is managed, wherein each global division is by whole users
It is divided into different groups, and each user is pertaining only to a group.
The third aspect, the present invention provide a kind of server, including memory and processor, and the memory is for storing
Computer program realizes that first aspect any one of them AB test system indexs are set when the computer program is executed by processor
Believe the computational methods in section.
Fourth aspect, the present invention provide a kind of computer readable storage medium, are deposited on the computer readable storage medium
Computer program is contained, realizes that first aspect any one of them AB tests system index confidence when which is executed by processor
The computational methods in section.
Compared with prior art, the present invention has at least the following advantages：
1. in computational methods, device and the server that AB provided by the invention tests system index confidence interval, by AB
The global division of flow progress the first preset value time for testing system administration obtains each global several lower corresponding being divided into of division
Group is mutually parallel between described each time global division, and using the grouping situation as the calculating of subsequent confidence interval basis.
Wherein, the flow progress repeatedly global division of system administration is tested AB, and is mutually parallel between each global division, is reduced
The randomness that the single overall situation divides alleviates influence of the single overall situation division to confidence interval computation result under actual conditions, carries
The accuracy that high confidence interval calculates.
2., will be each in computational methods, device and the server that AB provided by the invention tests system index confidence interval
The corresponding difference value of the second preset value packet size merges described in secondary global division, and according to the difference value after merging, obtains
To confidence interval.Wherein, the corresponding difference value of the second preset value packet size described in each global division is merged, is increased
While sample number, by each time it is global divide in be mutually parallel independent index value, produce contact, this method by merging
It is simple and practicable, while alleviating the single overall situation under actual conditions and dividing the lower independent index value that is respectively mutually parallel for dividing randomness
Influence to confidence interval computation result improves the accuracy of confidence interval calculating, and to AB test system index variations
Confidence level provides more reliable support.Specifically, present invention simulation allochoric to index error is more nearly in practical application
The case where so that the confidence interval obtained based on the simulation distribution is more in line with reality, overcomes and does not carry out any strategy change
Two orthogonal groupings between difference value break through confidence interval conspicuousness Decision boundaries the problem of so that confidence interval itself
It is more accurate.
The additional aspect of the present invention and advantage will be set forth in part in the description, these will become from the following description
Obviously, or practice through the invention is recognized.
Description of the drawings
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, wherein：
Fig. 1 is the first method flow chart of the embodiment of the present invention one；
Fig. 2 is the second method flow chart of the embodiment of the present invention one；
Fig. 3 is the third method flow diagram of the embodiment of the present invention one；
Fig. 4 is the fourth method flow chart of the embodiment of the present invention one；
Fig. 5 is the first module frame chart of the embodiment of the present invention two；
Fig. 6 is second of module frame chart of the embodiment of the present invention two；
Fig. 7 is the third module frame chart of the embodiment of the present invention two；
Fig. 8 is the 4th kind of module frame chart of the embodiment of the present invention two；
Fig. 9 is the 5th kind of module frame chart of the embodiment of the present invention two.
Specific implementation mode
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, and is only used for explaining the present invention, and is not construed as limiting the claims.
Those skilled in the art of the present technique are appreciated that unless expressly stated, singulative " one " used herein, " one
It is a ", " described " and "the" may also comprise plural form.It is to be further understood that is used in the specification of the present invention arranges
It refers to there are the feature, integer, step, operation, element and/or component, but it is not excluded that presence or addition to take leave " comprising "
Other one or more features, integer, step, operation, element, component and/or their group.It should be understood that when we claim member
Part is " connected " or when " coupled " to another element, it can be directly connected or coupled to other elements, or there may also be
Intermediary element.In addition, " connection " used herein or " coupling " may include being wirelessly connected or wirelessly coupling.It is used herein to arrange
Diction "and/or" includes that the whole of one or more associated list items or any cell are combined with whole.
Those skilled in the art of the present technique are appreciated that unless otherwise defined, all terms used herein (including technology art
Language and scientific terminology), there is meaning identical with the general understanding of the those of ordinary skill in fields of the present invention.Should also
Understand, those terms such as defined in the general dictionary, it should be understood that have in the context of the prior art
The consistent meaning of meaning, and unless by specific definitions as here, the meaning of idealization or too formal otherwise will not be used
To explain.
Those skilled in the art of the present technique are appreciated that remote network devices used herein above comprising but be not limited to count
The cloud that calculation machine, network host, single network server, multiple network server collection or multiple servers are constituted.Here, Yun Youji
It is constituted in a large amount of computers or network server of cloud computing (Cloud Computing), wherein cloud computing is Distributed Calculation
One kind, a super virtual computer being made of the computer collection of a group loose couplings.In the embodiment of the present invention, distal end
It can be realized and be communicated by any communication modes between the network equipment, terminal device and WNS servers, including but not limited to, is based on
The mobile communication of 3GPP, LTE, WIMAX, based on TCP/IP, the computer network communication of udp protocol and based on bluetooth, infrared
The low coverage wireless transmission method of transmission standard.
Computational methods, the device kimonos of system index confidence interval are tested AB provided by the present invention below in conjunction with the accompanying drawings
The specific implementation mode of business device describes in detail.Here, the illustrative embodiments and their description of the present invention are for explaining this hair
It is bright but not as a limitation of the invention.
Embodiment one
In conjunction with Fig. 1, the embodiment of the present invention provides a kind of computational methods of AB tests system index confidence interval, including following
Step：
AB is tested the global division of flow progress the first preset value time of system administration by S01, is obtained under each global division
Corresponding several groups being divided into are mutually parallel between described each time global division；
Wherein, AB tests system and is mainly responsible for the tasks such as management assignment of traffic and the calculating of corresponding index, assignment of traffic
It is mainly realized by multiple orthogonal, multiple orthogonal realizes that the flow progress for dividing and being to be managed AB tests system is repeatedly global
It divides.In the present embodiment, the flow for testing AB system administration carries out the global division of the first preset value time, wherein complete each time
Office divides and is divided into whole users in different groups, and each user is pertaining only to a group, is not done mutually between the interior user of group
It disturbs；It is mutually parallel between each global division, i.e., identical user can exist simultaneously in repeatedly parallel division, great Liang Yong
The statistical result of family behavior is not interfere with each other in multiple parallel divisions.
Specifically, the flow that AB is tested to system administration carries out the parallel overall situation of the first preset value time and divides it, in
Each time whole users are divided into different groups by global divide, and each user is pertaining only to a group.
In this example, it is assumed that the first preset value is M, and the flow that AB tests system is managed is carried out primary global
It divides, several groups being divided into preferably are divided into 100*10nA group (n is non-negative even number, generally takes 0 or 2), will be for the first time
Global division mark is M1, the global division mark of last time is MM, then right under each global division obtained in step S01
The relationship for several groups that should be divided into can be stated such as table 1：
| The overall situation divides number | The quantity for the group that each correspondence is divided into |
| M1 | 100*10nIt is a |
| M2 | 100*10nIt is a |
| … | … |
| MM | 100*10nIt is a |
1 each global division of table and the numerical relation list between corresponding each group be divided into
S02 calculates each corresponding several of global division and organizes corresponding index values, obtains each overall situation and divides respective counts
Several numerical value of amount；
Wherein, AB tests the index calculating of system mainly by the index value calculating in the user group for using algorithms of different
Out.In the present embodiment, index value is calculated as in the case where pre-defining a series of indexs, such as clicking rate passes through
It uses algorithm A and algorithm B into commending contents on line respectively in different user groups, and calculates the user group of corresponding special algorithm
Index value.The index value is characterized in different user groups in AB test systems and the corresponding tests of algorithm A and algorithm B is used to tie respectively
Fruit.Each corresponding several of global division of the calculating are organized in corresponding index values, if since each global division is corresponding
Include the user group for using A algorithm in dry group, also there is the user group for using B algorithms, so what correspondence was calculated
Index value includes using A algorithm and the calculated numerical value of B algorithms.Wherein, user group one index value of correspondence, one
Index value corresponds to a numerical value.
The example for continuing to continue to use in step S01 illustrates, and global divide corresponds to 100*10 each timenA group, and
A corresponding index value is calculated in one group correspondence, i.e., the lower correspondence of global division obtains 100*10 each timenA numerical value closes
System such as table 2：
| The overall situation divides number | The quantity for the group that each correspondence is divided into | The quantity of corresponding index value |
| M1 | 100*10nIt is a | 100*10nIt is a |
| M2 | 100*10nIt is a | 100*10nIt is a |
| … | … | … |
| MM | 100*10nIt is a | 100*10nIt is a |
Under 2 each global divisions of table, each grouping and the numerical relation list between corresponding index value
Further, in the present embodiment, for ease of citing, the corresponding numerical value of above-mentioned each index value is passed through into X1, X2,
X3…Xm(m=1,2,3 ... 100*10n) record.
S03 obtains in each global division the according to the numerical value randomly selected from each global several numerical value divided
The corresponding difference value of two preset value packet sizes；
Specifically, in conjunction with Fig. 2, according to the number randomly selected from several numerical value of each global division described in step S03
Value obtains the corresponding difference value of the second preset value packet size in each global division, includes the following steps：
S031 is obtained according to the numerical value randomly selected from each global several numerical value divided in each global division
The corresponding difference value of single packet size；
Further, it in conjunction with Fig. 3, is randomly selected according to from several numerical value of each global division described in step S031
Numerical value, obtain each time it is global divide in the corresponding difference value of single packet size, include the following steps：
S0311 is obtained to randomly selecting the summation of the 4th preset value numerical value from each global several numerical value divided
Each time global divide in first sample and；Preferably, the 4th preset value is less than described each time several global divided
Numerical value.
Wherein, the 4th preset value numerical value of being randomly selected from each global several numerical value divided is in step
Each overall situation divides corresponding X in S021, X2, X3…Xm(m=1,2,3 ... 100*10n) the 4th preset value is extracted in each numerical value
Numerical value.In the present embodiment, for ease of citing, the 4th preset value is used into k*10n(n takes the numerical value n, k when traffic partition
Take natural number, and k<100) it records.
Assuming that in certain primary global division, several numerical value are randomly selected in step S0311, are drawn into
X1…Xx(1<x<M), then it sums to its each numerical value, forms first sample and S1。
S0312 is asked randomly selecting the 4th preset value numerical value from each global several remaining numerical value divided
With, obtain each global the second sample divided and；
Specifically, described to randomly select the 4th preset value numerical value from each global several remaining numerical value divided
After excluding numerical value decimated in step S0311, the 4th preset value is randomly selected from several remaining numerical value
Numerical value.Wherein, global divide is randomly selected out in corresponding step S0311 and step S0312 in several numerical value each time
Quantity it is identical.Also that is, in step S0312, k*10 still can be used in the 4th preset valuen(n takes numerical value when traffic partition
N, k take natural number, and k<100) it records.The 4th that it is randomly selected from each global several remaining numerical value divided
Preset value numerical value is summed to obtain the second sample and S2。
S0313 1 is subtracted the value of (the second sample and divided by first sample and) as each time it is global divide in individually grouping is greatly
Small corresponding difference value.
Specifically, described in conjunction with the example aboveIt is complete as each time
During office divides, the difference percentage of single sample.
S032 recycles the numerical value that the basis is randomly selected from each global several numerical value divided, calculates each time entirely
Office is in dividing the step of single packet size corresponding difference value, until obtain each time it is global divide in the grouping of the second preset value
The corresponding difference value of size.
Specifically, it is calculated according to the numerical value randomly selected from each global several numerical value divided described in the ring
Each time global divide in single packet size corresponding difference value the step of, until obtaining the second preset value in each global division
The corresponding difference value of a packet size is that cycle executes step S0311, S0312, S0313 the second preset value time finally so that each
In secondary global division, obtaining the second preset value packet size isDifference value.Wherein, in conjunction with the example above,
In the present embodiment, it is assumed that the second preset value is N, and after completing step S032 herein, then each global correspondence that divides obtains N number of point
Organizing size isDifference value.Wherein, in the present embodiment, by step S03, above each method and step with each time
The follow-up calculating step carried out based on several groups that the overall situation divides or generated numerical value do not interfere independently between each other.Into
One step, the global division of the first preset value time for carrying out the flow that AB tests system is managed in step S01, can be independent step
Suddenly, as follows：
The flow that AB tests system is managed is subjected to primary global division, obtains several groups；
It calculates several and organizes corresponding index value, obtain several numerical value of respective numbers；
According to the numerical value randomly selected from several numerical value, the corresponding difference value of the second preset value packet size is obtained；
Above three step is recycled, until obtaining corresponding second preset value grouping under the global division of the first preset value
The corresponding difference value of size；It is mutually parallel between each global division.
S04 merges the corresponding difference value of the second preset value packet size described in each global division；
Further, since the flow managed AB tests system has carried out M parallel global division, it is at this time
Obtaining M*N packet size isDifference value.
Specifically, in conjunction with Fig. 4, merge described in step S04 each time it is global divide described in the second preset value packet size
Corresponding difference value, includes the following steps：
S041 integrates the corresponding difference value of the second preset value packet size described in each global division, and it is pre- to obtain third
If the corresponding difference value of value packet size；
S042 arranges the corresponding difference value ascending order of the third preset value packet size, obtains ordered series of numbers；
Wherein, the third preset value is first the second preset values of preset value *, i.e. M*N.Due to each time it is global divide under,
A user group is not interfere with each other independently of each other, and corresponding index value is not interfere with each other, and in this step, by each overall situation
The corresponding difference value of second preset value packet size described in division merges, for by each time it is global divide in corresponding to it is each
The corresponding difference value of packet size is used as same item data resource, i.e. by step S041 first by each difference
Value is integrated, and forms ordered series of numbers by the way that all differences value is carried out ascending sort in step S042 so that former each independence is non-interfering
Difference value forms the difference value that is mutually related, is divided to confidence with mitigating the single overall situation under actual conditions by simply sorting
The influence of interval computation result.
S05 obtains confidence interval according to the difference value after merging.
Further, confidence interval is obtained, is included the following steps according to the difference value after merging described in step S05：
Confidence level is multiplied by the third preset value, the numerical value obtained after rounding is as index value；
The difference value that will be found in the ordered series of numbers according to the index value, as the corresponding difference of the confidence level
Value；
According to the corresponding difference value of the confidence level, confidence interval is obtained.
Wherein, the confidence level is a fixed value, in the present embodiment, preferably 90%, and accordingly, the index
Value is 90%*M*N (rounding).And the corresponding difference value of the confidence level be in the ordered series of numbers, under sorting successively, rope
Draw the difference value of corresponding position.
Embodiment two
In conjunction with Fig. 5, in order to further test the AB described in above-described embodiment the computational methods of system index confidence interval
It is illustrated, modularization explanation is carried out to it, provide the AB computing devices of test system index confidence interval comprising with lower die
Block：
Division module 51, the flow for AB to be tested to system administration carry out the first preset value time overall situation and divide, obtain each
Secondary lower corresponding several groups being divided into of global division, each overall situation are mutually parallel between dividing；
Specifically, the division module 51 includes that the flow for being managed AB tests system carries out the first preset value
The overall situation divides, wherein whole users are divided into different groups by each global divide, and each user is pertaining only to a group.
First computing module 52 is organized corresponding index values for calculating each corresponding several of global division, is obtained each
Secondary global several numerical value for dividing respective numbers；
Second computing module 53, for according to the numerical value randomly selected from each global several numerical value divided, obtaining
The corresponding difference value of second preset value packet size in each global division；
Specifically, in conjunction with Fig. 6, second computing module 53 includes with lower unit：
Difference value computing unit 531, for according to the numerical value randomly selected from several numerical value of each global division,
Obtain the corresponding difference value of single packet size in each global division；
Further, in conjunction with Fig. 7, the difference value computing unit 531 includes following subelement：
First summation subelement 5311, for default to randomly selecting the 4th from each global several numerical value divided
Value numerical value summation, obtain each time it is global divide in first sample with；
Second summation subelement 5312, for randomly selecting the from several remaining numerical value of each global division
Four preset value numerical value are summed, obtain each global the second sample divided with；
Difference value computation subunit 5313, for subtracting the value of (the second sample and divided by first sample and) using 1 as each time
The corresponding difference value of single packet size during the overall situation divides.
Cycling element 532, the number randomly selected from each global several numerical value divided for recycling the basis
Value, calculate each time it is global divide in single packet size corresponding difference value the step of, until obtaining in each overall situation division the
The corresponding difference value of two preset value packet sizes.
Wherein, the 4th preset value is less than described each time global several numerical value divided.
Merging module 54, for merging the corresponding difference of the second preset value packet size described in each global division
Value；
Specifically, in conjunction with Fig. 8, the merging module 54 includes with lower unit：
Integral unit 541, for integrating the corresponding difference of the second preset value packet size described in each global division
Value obtains the corresponding difference value of third preset value packet size；
Sequencing unit 542 obtains number for arranging the corresponding difference value ascending order of the third preset value packet size
Row；
Wherein, the third preset value is first the second preset values of preset value *.
Third computing module 55, for according to the difference value after merging, obtaining confidence interval.
Further, in conjunction with Fig. 9, the third computing module 55 includes following subelement：
Indexing units 551, for confidence level to be multiplied by the third preset value, the numerical value obtained after rounding is as index
Value；
Searching unit 552, the difference value for will be found in the ordered series of numbers according to the index value are set as described
The horizontal corresponding difference value of letter；
Computing unit 553, for according to the corresponding difference value of the confidence level, obtaining confidence interval.
Embodiment three
The embodiment of the present invention provides a kind of server, including：
One or more processors；
Memory；
And one or more computer programs, one or more of computer programs are stored in the memory；
Realize that the AB as described in embodiment one is surveyed when one or more of computer programs are executed by one or more of processors
The computational methods of test system index confidence interval.
The server conventionally comprises processor and with the computer program product of form of memory or computer-readable
Medium.Memory can be such as flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk or ROM
Etc electronic memory.Memory has the storage of the program code for executing any method and step in the above method empty
Between.For example, the memory space for program code may include each of the various steps being respectively used in realization above method
A program code.These program codes can read or be written to this from one or more computer program product
Or in multiple computer program products.These computer program products include such as hard disk, compact-disc (CD), storage card or
The program code carrier of floppy disk etc.Such computer program product is usually the portable or fixed storage list
Member.The storage unit can have memory paragraph or memory space of memory similar arrangement etc..Program code can for example with
Appropriate form is compressed.In general, storage unit includes the program code for executing steps of a method in accordance with the invention, you can
With the code read by such as processor, these codes cause the server to execute describe above when being run by server
Method in each step.
In embodiments of the present invention, the processor included by the server is with the following functions：
Flow progress the first preset value time overall situation that AB is tested to system administration divides, and obtains each global lower correspondence of division
Several groups being divided into are mutually parallel between described each time global division；
It calculates each corresponding several of global division and organizes corresponding index values, obtain each overall situation and divide respective numbers
Several numerical value；
According to the numerical value randomly selected from each global several numerical value divided, obtain each time it is global divide in second in advance
If the corresponding difference value of value packet size；
Merge the corresponding difference value of the second preset value packet size described in each global division；
According to the difference value after merging, confidence interval is obtained.
Namely processor has the computational methods for executing AB tests system index confidence interval described in the above embodiments one
Function, details are not described herein.
Example IV
The present invention provides a kind of computer readable storage medium, and computer is stored on the computer readable storage medium
Program realizes the computational methods of the AB test system index confidence intervals described in embodiment one when the program is executed by processor.
Wherein, the storage medium include but not limited to any kind of disk (including floppy disk, hard disk, CD, CD-ROM,
And magneto-optic disk), ROM (Read-Only Memory, read-only memory), (Random AcceSS Memory, store RAM immediately
Device), EPROM (EraSable Programmable Read-Only Memory, Erarable Programmable Read only Memory),
(Electrically EraSable Programmable Read-Only Memory, electric erazable programmable is read-only to be deposited EEPROM
Reservoir), flash memory, magnetic card or light card.It is, storage medium includes by equipment (for example, computer) can read
Form storage or transmission information any medium.Can be read-only memory, disk or CD etc..
It should be understood that although each step in the flow chart of attached drawing is shown successively according to the instruction of arrow,
These steps are not that the inevitable sequence indicated according to arrow executes successively.Unless expressly stating otherwise herein, these steps
Execution there is no stringent sequences to limit, can execute in the other order.Moreover, at least one in the flow chart of attached drawing
Part steps may include that either these sub-steps of multiple stages or stage are not necessarily in synchronization to multiple sub-steps
Completion is executed, but can be executed at different times, execution sequence is also not necessarily to be carried out successively, but can be with other
Either the sub-step of other steps or at least part in stage execute step in turn or alternately.
The serial number of the embodiments of the present invention is for illustration only, can not represent the quality of embodiment.The above is only this
The some embodiments of invention, it is noted that for those skilled in the art, do not departing from original of the invention
Under the premise of reason, several improvements and modifications can also be made, these improvements and modifications also should be regarded as protection scope of the present invention.
Claims (14)
1. a kind of computational methods of AB tests system index confidence interval, which is characterized in that include the following steps：
The global division of flow progress the first preset value time that AB is tested to system administration obtains each global lower correspondence of division and is divided into
Several groups, described each time global divide between be mutually parallel；
It calculates each corresponding several of global division and organizes corresponding index values, obtain each overall situation and divide the several of respective numbers
A numerical value；
According to the numerical value randomly selected from each global several numerical value divided, the second preset value in each global division is obtained
The corresponding difference value of a packet size；
Merge the corresponding difference value of the second preset value packet size described in each global division；
According to the difference value after merging, confidence interval is obtained.
2. according to the method described in claim 1, it is characterized in that, described merge the second preset value described in each global division
The corresponding difference value of a packet size, includes the following steps：
The corresponding difference value of the second preset value packet size described in each global division is integrated, third preset value point is obtained
The corresponding difference value of group size；
By the corresponding difference value ascending order arrangement of the third preset value packet size, ordered series of numbers is obtained；
Wherein, the third preset value is first the second preset values of preset value *.
3. according to the method described in claim 2, it is characterized in that, the difference value according to after merging, obtains confidence interval,
Include the following steps：
Confidence level is multiplied by the third preset value, the numerical value obtained after rounding is as index value；
The difference value that will be found in the ordered series of numbers according to the index value, as the corresponding difference value of the confidence level；
According to the corresponding difference value of the confidence level, confidence interval is obtained.
4. according to the method described in claim 1, it is characterized in that, the basis from each global several numerical value divided with
The numerical value that machine extracts obtains the corresponding difference value of the second preset value packet size in each global division, includes the following steps：
According to the numerical value randomly selected from each global several numerical value divided, calculates and be individually grouped in each global division
The corresponding difference value of size；
The numerical value that the basis is randomly selected from each global several numerical value divided is recycled, is calculated in each global division
The step of single packet size corresponding difference value, until obtain each time it is global divide in the second preset value packet size correspondence
Difference value.
5. according to the method described in claim 4, it is characterized in that, the basis is from each global several numerical value divided
The numerical value randomly selected calculates the corresponding difference value of single packet size in each global division, includes the following steps：
To randomly selecting the summation of the 4th preset value numerical value from each global several numerical value divided, obtains each overall situation and draw
Point in first sample and；
To randomly selecting the summation of the 4th preset value numerical value from each global several remaining numerical value divided, obtain each time
The overall situation divide the second sample and；
1 is subtracted the value of (the second sample with divided by first sample with) as each time it is global divide in the corresponding difference of single packet size
Different value.
6. according to the method described in claim 1, it is characterized in that, the flow that AB is tested to system administration carries out first in advance
If in the global division of value time, each time whole users are divided into different groups by global divide, and each user is pertaining only to one
A group.
7. a kind of computing device of AB tests system index confidence interval, which is characterized in that comprise the following modules：
Division module, the flow for AB to be tested to system administration carry out the first preset value time overall situation and divide, and obtain each overall situation
Lower corresponding several groups being divided into are divided, are mutually parallel between described each time global division；
First computing module organizes corresponding index values for calculating each corresponding several of global division, obtains each overall situation
Divide several numerical value of respective numbers；
Second computing module, for according to the numerical value randomly selected from each global several numerical value divided, obtaining each time entirely
The corresponding difference value of second preset value packet size during office divides；
Merging module, for merging the corresponding difference value of the second preset value packet size described in each global division；
Third computing module, for according to the difference value after merging, obtaining confidence interval.
8. device according to claim 7, which is characterized in that the merging module includes with lower unit：
Integral unit is obtained for integrating the corresponding difference value of the second preset value packet size described in each global division
The corresponding difference value of third preset value packet size；
Sequencing unit obtains ordered series of numbers for arranging the corresponding difference value ascending order of the third preset value packet size；
Wherein, the third preset value is first the second preset values of preset value *.
9. device according to claim 8, which is characterized in that the third computing module includes with lower unit：
Indexing units, for confidence level to be multiplied by the third preset value, the numerical value obtained after rounding is as index value；
Searching unit, the difference value for will be found in the ordered series of numbers according to the index value, as the confidence level
Corresponding difference value；
Computing unit, for according to the corresponding difference value of the confidence level, obtaining confidence interval.
10. device according to claim 7, which is characterized in that second computing module includes with lower unit：
Difference value computing unit, for according to the numerical value randomly selected from each global several numerical value divided, obtaining each
The corresponding difference value of single packet size in secondary global division；
Cycling element, the numerical value randomly selected from each global several numerical value divided for recycling the basis, calculates
Each time global divide in single packet size corresponding difference value the step of, until obtaining the second preset value in each global division
The corresponding difference value of a packet size.
11. device according to claim 10, which is characterized in that the difference value computing unit includes following subelement：
First summation subelement, for randomly selecting the 4th preset value numerical value from each global several numerical value divided
Summation, obtain each time it is global divide in first sample with；
Second summation subelement, for randomly selecting the 4th preset value from each global several remaining numerical value divided
The summation of a numerical value, obtain each global the second sample divided with；
Difference value computation subunit, for subtracting the value of (the second sample and divided by first sample and) using 1 as each global division
In the single corresponding difference value of packet size.
12. device according to claim 7, which is characterized in that the division module includes for AB to be tested system institute
The flow of management carries out the global division of the first preset value time, wherein whole users are divided into different groups by each global divide
In, and each user is pertaining only to a group.
13. a kind of server, including memory and processor, the memory is for storing computer program, the computer
The calculating side of the AB test system index confidence intervals described in any one of claim 1-6 is realized when program is executed by processor
Method.
14. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program realizes that the AB described in any one of claim 1-6 tests system index confidence interval when the program is executed by processor
Computational methods.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810266136.5A CN108415845B (en) | 2018-03-28 | 2018-03-28 | Calculation method, device and the server of AB test macro index confidence interval |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810266136.5A CN108415845B (en) | 2018-03-28 | 2018-03-28 | Calculation method, device and the server of AB test macro index confidence interval |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108415845A true CN108415845A (en) | 2018-08-17 |
| CN108415845B CN108415845B (en) | 2019-05-31 |
Family
ID=63133577
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810266136.5A Active CN108415845B (en) | 2018-03-28 | 2018-03-28 | Calculation method, device and the server of AB test macro index confidence interval |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108415845B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109445811A (en) * | 2018-09-07 | 2019-03-08 | 平安科技（深圳）有限公司 | Gray scale dissemination method, device, computer equipment and computer storage medium |
| CN109688217A (en) * | 2018-12-24 | 2019-04-26 | 北京达佳互联信息技术有限公司 | A kind of information push method, device and electronic equipment |
| CN109710511A (en) * | 2018-12-04 | 2019-05-03 | 北京达佳互联信息技术有限公司 | AB test method, device, server and storage medium |
| CN110162552A (en) * | 2019-05-09 | 2019-08-23 | 山东科技大学 | Time series feature extracting method and system based on confidence interval |
| CN110443648A (en) * | 2019-08-01 | 2019-11-12 | 北京字节跳动网络技术有限公司 | Information distribution method, device, electronic equipment and storage medium |
| CN110443647A (en) * | 2019-08-01 | 2019-11-12 | 北京字节跳动网络技术有限公司 | Information distribution method and equipment |
| CN111294253A (en) * | 2020-01-15 | 2020-06-16 | 腾讯科技（深圳）有限公司 | Test data processing method and device, computer equipment and storage medium |
| CN112162918A (en) * | 2020-09-07 | 2021-01-01 | 北京达佳互联信息技术有限公司 | Application program testing method and device and electronic equipment |
| CN112463577A (en) * | 2019-09-09 | 2021-03-09 | 北京达佳互联信息技术有限公司 | Sample data processing method and device and electronic equipment |
| CN112699035A (en) * | 2020-12-29 | 2021-04-23 | 中国航空工业集团公司西安飞机设计研究所 | Multi-partition airborne application software association index testing method and device |
| CN112711524A (en) * | 2019-10-25 | 2021-04-27 | 腾讯科技（深圳）有限公司 | Data testing method and device based on AB testing and computer storage medium |
| CN112905476A (en) * | 2021-03-12 | 2021-06-04 | 网易（杭州）网络有限公司 | Test execution method and device, electronic equipment and storage medium |
| CN113297277A (en) * | 2021-06-18 | 2021-08-24 | 北京有竹居网络技术有限公司 | Test statistic determination method, device, readable medium and electronic equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102014031A (en) * | 2010-12-31 | 2011-04-13 | 湖南神州祥网科技有限公司 | Method and system for network flow anomaly detection |
| CN102868628A (en) * | 2011-07-06 | 2013-01-09 | 阿里巴巴集团控股有限公司 | Flow segmentation method, device and system |
| US20160103758A1 (en) * | 2014-10-08 | 2016-04-14 | Yahoo! Inc. | Online product testing using bucket tests |
| CN105913145A (en) * | 2016-04-08 | 2016-08-31 | 北京吆喝科技有限公司 | Data driving-based AB test method |
| CN106294364A (en) * | 2015-05-15 | 2017-01-04 | 阿里巴巴集团控股有限公司 | Realize the method and apparatus that web crawlers captures webpage |
| CN106959925A (en) * | 2017-04-25 | 2017-07-18 | 北京云测信息技术有限公司 | A kind of version method of testing and device |
| US9804954B2 (en) * | 2016-01-07 | 2017-10-31 | International Business Machines Corporation | Automatic cognitive adaptation of development assets according to requirement changes |
| CN107451020A (en) * | 2017-06-28 | 2017-12-08 | 北京五八信息技术有限公司 | A kind of AB test systems and method of testing |
-
2018
- 2018-03-28 CN CN201810266136.5A patent/CN108415845B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102014031A (en) * | 2010-12-31 | 2011-04-13 | 湖南神州祥网科技有限公司 | Method and system for network flow anomaly detection |
| CN102868628A (en) * | 2011-07-06 | 2013-01-09 | 阿里巴巴集团控股有限公司 | Flow segmentation method, device and system |
| US20160103758A1 (en) * | 2014-10-08 | 2016-04-14 | Yahoo! Inc. | Online product testing using bucket tests |
| CN106294364A (en) * | 2015-05-15 | 2017-01-04 | 阿里巴巴集团控股有限公司 | Realize the method and apparatus that web crawlers captures webpage |
| US9804954B2 (en) * | 2016-01-07 | 2017-10-31 | International Business Machines Corporation | Automatic cognitive adaptation of development assets according to requirement changes |
| CN105913145A (en) * | 2016-04-08 | 2016-08-31 | 北京吆喝科技有限公司 | Data driving-based AB test method |
| CN106959925A (en) * | 2017-04-25 | 2017-07-18 | 北京云测信息技术有限公司 | A kind of version method of testing and device |
| CN107451020A (en) * | 2017-06-28 | 2017-12-08 | 北京五八信息技术有限公司 | A kind of AB test systems and method of testing |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109445811B (en) * | 2018-09-07 | 2024-05-28 | 平安科技（深圳）有限公司 | Gray release method, device, computer equipment and computer storage medium |
| CN109445811A (en) * | 2018-09-07 | 2019-03-08 | 平安科技（深圳）有限公司 | Gray scale dissemination method, device, computer equipment and computer storage medium |
| CN109710511A (en) * | 2018-12-04 | 2019-05-03 | 北京达佳互联信息技术有限公司 | AB test method, device, server and storage medium |
| CN109710511B (en) * | 2018-12-04 | 2022-04-01 | 北京达佳互联信息技术有限公司 | AB test method, device, server and storage medium |
| CN109688217B (en) * | 2018-12-24 | 2021-09-03 | 北京达佳互联信息技术有限公司 | Message pushing method and device and electronic equipment |
| CN109688217A (en) * | 2018-12-24 | 2019-04-26 | 北京达佳互联信息技术有限公司 | A kind of information push method, device and electronic equipment |
| CN110162552A (en) * | 2019-05-09 | 2019-08-23 | 山东科技大学 | Time series feature extracting method and system based on confidence interval |
| CN110443647B (en) * | 2019-08-01 | 2022-09-06 | 北京字节跳动网络技术有限公司 | Information delivery method and device |
| CN110443647A (en) * | 2019-08-01 | 2019-11-12 | 北京字节跳动网络技术有限公司 | Information distribution method and equipment |
| CN110443648A (en) * | 2019-08-01 | 2019-11-12 | 北京字节跳动网络技术有限公司 | Information distribution method, device, electronic equipment and storage medium |
| CN112463577A (en) * | 2019-09-09 | 2021-03-09 | 北京达佳互联信息技术有限公司 | Sample data processing method and device and electronic equipment |
| CN112711524A (en) * | 2019-10-25 | 2021-04-27 | 腾讯科技（深圳）有限公司 | Data testing method and device based on AB testing and computer storage medium |
| CN111294253A (en) * | 2020-01-15 | 2020-06-16 | 腾讯科技（深圳）有限公司 | Test data processing method and device, computer equipment and storage medium |
| CN112162918A (en) * | 2020-09-07 | 2021-01-01 | 北京达佳互联信息技术有限公司 | Application program testing method and device and electronic equipment |
| CN112699035A (en) * | 2020-12-29 | 2021-04-23 | 中国航空工业集团公司西安飞机设计研究所 | Multi-partition airborne application software association index testing method and device |
| CN112699035B (en) * | 2020-12-29 | 2023-06-23 | 中国航空工业集团公司西安飞机设计研究所 | Multi-partition airborne application software association index testing method and device |
| CN112905476A (en) * | 2021-03-12 | 2021-06-04 | 网易（杭州）网络有限公司 | Test execution method and device, electronic equipment and storage medium |
| CN112905476B (en) * | 2021-03-12 | 2023-08-11 | 网易（杭州）网络有限公司 | Test execution method and device, electronic equipment and storage medium |
| CN113297277A (en) * | 2021-06-18 | 2021-08-24 | 北京有竹居网络技术有限公司 | Test statistic determination method, device, readable medium and electronic equipment |
| CN113297277B (en) * | 2021-06-18 | 2024-07-09 | 北京有竹居网络技术有限公司 | Test statistic determining method and device, readable medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108415845B (en) | 2019-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108415845B (en) | Calculation method, device and the server of AB test macro index confidence interval | |
| CN110515739B (en) | Deep learning neural network model load calculation method, device, equipment and medium | |
| JP7343568B2 (en) | Identifying and applying hyperparameters for machine learning | |
| CN112800095B (en) | Data processing method, device, equipment and storage medium | |
| US20160314064A1 (en) | Systems and methods to identify and classify performance bottlenecks in cloud based applications | |
| US10878335B1 (en) | Scalable text analysis using probabilistic data structures | |
| US20120150860A1 (en) | Clustering with Similarity-Adjusted Entropy | |
| CN105824855B (en) | Method and device for screening and classifying data objects and electronic equipment | |
| CN106605222B (en) | Guided data exploration | |
| CN111190696A (en) | Docker container deployment method, system, device and storage medium | |
| CN112070550A (en) | Keyword determination method, device and equipment based on search platform and storage medium | |
| EP2965492A1 (en) | Selection of data storage settings for an application | |
| CN110928636A (en) | Virtual machine live migration method, device and equipment | |
| CN113553341A (en) | Multidimensional data analysis method, multidimensional data analysis device, multidimensional data analysis equipment and computer readable storage medium | |
| US11354297B2 (en) | Detecting positivity violations in multidimensional data | |
| US9727561B1 (en) | Context- and activity-aware content selection | |
| CN110737691B (en) | Method and apparatus for processing access behavior data | |
| US20200334297A1 (en) | Automatic adaption of a search configuration | |
| CN112579422A (en) | Scheme testing method and device, server and storage medium | |
| CN115051863A (en) | Abnormal flow detection method and device, electronic equipment and readable storage medium | |
| CN111967938B (en) | Cloud resource recommendation method and device, computer equipment and readable storage medium | |
| CN117290078A (en) | Method, device, electronic equipment and medium for distributing cloud storage resources | |
| CN111625615B (en) | Method and system for processing text data | |
| CN114281549A (en) | Data processing method and device | |
| CN114461390A (en) | Evaluation method combining multi-dimensional analysis and critical path method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |