CN102780573A - Method and equipment for carrying out experiments based on network flow - Google Patents

Method and equipment for carrying out experiments based on network flow Download PDF

Info

Publication number
CN102780573A
CN102780573A CN2011101214926A CN201110121492A CN102780573A CN 102780573 A CN102780573 A CN 102780573A CN 2011101214926 A CN2011101214926 A CN 2011101214926A CN 201110121492 A CN201110121492 A CN 201110121492A CN 102780573 A CN102780573 A CN 102780573A
Authority
CN
China
Prior art keywords
experimental
experiment
experiments
network flow
flow
Prior art date
Application number
CN2011101214926A
Other languages
Chinese (zh)
Inventor
毛继明
高楠
Original Assignee
百度在线网络技术(北京)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 百度在线网络技术(北京)有限公司 filed Critical 百度在线网络技术(北京)有限公司
Priority to CN2011101214926A priority Critical patent/CN102780573A/en
Publication of CN102780573A publication Critical patent/CN102780573A/en

Links

Abstract

The invention aims at providing a method and equipment for carrying out experiments by utilizing network flow in parallel, wherein the experimental equipment utilizes at least part of network flow in the experimental network flow to carry out multiple experiments, so as to obtain experimental feedback information. The equipment acquires at least part of network flow corresponding to the multiple experiments through filtering the experimental network flow according to relevant information of the multiple experiments to be carried out by utilizing the network flow in parallel; the multiple experiments corresponding to the part of network flow are carried out according to the part of network flow, so that the input quantity of network flow samples obtained by each experiment can be improved; and under the condition of certain experimental time, the accuracy of experimental results is greatly improved, and the time cost for carrying out the experiments is lowered.

Description

The method and apparatus that a kind of flow Network Based experimentizes

Technical field

The present invention relates to the network experiment technical field, relate in particular to the technology that a kind of flow Network Based experimentizes.

Background technology

Along with the continuous development of network experiment technology, the network development attendant is parallel when utilizing network traffics to experimentize, and is also increasingly high to the requirement that obtains experimental result promptly and accurately effectively.Existing parallel the utilization in the technology that network traffics experimentize; Generally the Experimental Network flow that obtains is divided; The part that each experiment will be got this Experimental Network flow is as the sample of experiment, and according to the operation that experimentizes of this part Experimental Network flow.For example shown in Figure 1; The Experimental Network flow is X; Wait to carry out experiment and be E1, E2, E3, Experimental Network flow X is divided into Experimental Network flow x1, x2, the x3 of a plurality of parts through the network flow quantity shunting device, and satisfies X=x1+x2+x3; The sample of the Experimental Network flow of a part in the Experimental Network flow of these a plurality of parts as experiment got in each experiment, and promptly x1 carries out that experiment E1, x2 carry out experiment E2, x3 carries out experiment E3.

Yet; The problem that above-mentioned way produces is because the Experimental Network flow sample that each experiment can access is limited, possibly cause experimental result not accurate enough; Or for obtaining experimental result more accurately; Prolong experimental period to obtain more Experimental Network flow, this makes the network development attendant can't within the short period, obtain the experimental result that it hopes that acquisition is desirable exactly, and then increases the time cost of carrying out experiment.

Therefore, under the situation that the Experimental Network flow is fixed, how to effectively utilize this Experimental Network flow and carry out experiment,, become one of present problem demanding prompt solution to obtain experimental result more accurately.

Summary of the invention

The purpose of this invention is to provide a kind of parallel method and apparatus that utilizes network traffics to experimentize.

According to an aspect of the present invention, a kind of computer implemented parallel method of utilizing network traffics to experimentize is provided, wherein, this method may further comprise the steps:

A is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information.

According to a further aspect in the invention, a kind of parallel equipment that utilizes network traffics to experimentize is provided also, wherein, this equipment comprises:

The experiment final controlling element, be used for parallel utilize the Experimental Network flow at least the subnetwork flow carry out a plurality of experiments, to obtain the experiment feedback information.

Compared with prior art; The present invention is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments; Improve the network traffics sample size that each experiment obtains thus; Under the certain situation of experimental period, make the experimental result accuracy be greatly improved, and reduce the time cost that experiment is carried out.

Description of drawings

Through reading the detailed description of doing with reference to following accompanying drawing that non-limiting example is done, it is more obvious that other features, objects and advantages of the present invention will become:

Fig. 1 illustrates prior art based on the principle schematic of utilizing network traffics to experimentize;

Fig. 2 illustrates the parallel equipment sketch map that utilizes network traffics to experimentize according to one aspect of the invention;

Fig. 3 illustrates the parallel equipment principle sketch map that utilizes network traffics to experimentize according to one aspect of the invention;

Fig. 4 illustrates the parallel equipment sketch map that utilizes network traffics to experimentize in accordance with a preferred embodiment of the present invention;

Fig. 5 illustrates the parallel equipment sketch map that utilizes network traffics to experimentize according to another preferred embodiment of the present invention;

Fig. 6 illustrates the parallel equipment sketch map that utilizes network traffics to experimentize of another preferred embodiment according to the present invention;

Fig. 7 illustrates the parallel equipment sketch map that utilizes network traffics to experimentize of another preferred embodiment according to the present invention;

Fig. 8 illustrates the parallel equipment sketch map that utilizes network traffics to experimentize in accordance with a preferred embodiment of the present invention;

Fig. 9 illustrates the parallel equipment sketch map that utilizes network traffics to experimentize according to another preferred embodiment of the present invention;

Figure 10 illustrates the parallel equipment principle sketch map that utilizes network traffics to experimentize according to another preferred embodiment of the present invention;

Figure 11 illustrates the parallel method flow diagram that utilizes network traffics to experimentize according to another preferred embodiment of the present invention;

Figure 12 illustrates the parallel method principle flow chart that utilizes network traffics to experimentize according to another preferred embodiment of the present invention;

Figure 13 illustrates in accordance with a preferred embodiment of the present invention parallel and utilizes the network traffics method flow diagram that experimentizes;

Figure 14 illustrates the parallel method flow diagram that utilizes network traffics to experimentize according to another preferred embodiment of the present invention;

Figure 15 illustrates the parallel method flow diagram that utilizes network traffics to experimentize of another preferred embodiment according to the present invention;

Figure 16 illustrates the parallel method flow diagram that utilizes network traffics to experimentize of another preferred embodiment according to the present invention;

Figure 17 illustrates the parallel method flow diagram that utilizes network traffics to experimentize in accordance with a preferred embodiment of the present invention;

Figure 18 illustrates the parallel method flow diagram that utilizes network traffics to experimentize according to another preferred embodiment of the present invention.

Same or analogous Reference numeral is represented same or analogous parts in the accompanying drawing.

Embodiment

Below in conjunction with accompanying drawing the present invention is described in further detail.

Fig. 2 is the equipment sketch map according to one aspect of the invention, and the parallel equipment that utilizes network traffics to experimentize is shown.Wherein, experimental facilities 1 comprises experiment final controlling element 11.Particularly, experiment final controlling element 11 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information.At this; Experimental facilities 1 includes but not limited to network host, single network server, a plurality of webserver collection or based on the set of computers of cloud computing; At this, cloud computing is a kind of of Distributed Calculation, a super virtual machine of being made up of the loosely-coupled computer collection of a group.Those skilled in the art will be understood that above-mentioned experimental facilities is merely for example, and other experimental facilities existing or that possibly occur from now on also should be included in the protection range of the present invention, and be contained in this with way of reference as applicable to the present invention.

More specifically, experiment final controlling element 11 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information.Particularly, as shown in Figure 3, there are a plurality of parallel experiment E1, E2, E3 that utilize network traffics to be carried out; For Experimental Network flow X, experiment final controlling element 11 generals a part of network traffics wherein are used for carrying out some experiments of these a plurality of experiments, like E1; To obtain to test accordingly feedback information, utilize another part network traffics to carry out at least two experiments in these a plurality of experiments and walk abreast, like E1, E2; To obtain to test accordingly feedback information; Perhaps test final controlling element 11 parallel these whole Experimental Network flows that utilize and carry out at least two experiments in these a plurality of experiments, like E1, E3 and E1, E2, E3, to obtain to test accordingly feedback information; Perhaps test final controlling element 11 parallel these overall network flows that utilize again and carry out the whole experiments in these a plurality of experiments, to obtain to test accordingly feedback information.At this, " the parallel utilization " means the same subnetwork flow that utilizes in the network traffics and carries out a plurality of experiments, and the executive mode of these a plurality of experiments is not limited to concurrent execution or order is carried out.Wherein, said Experimental Network flow includes but not limited to the access request of user's webpage searching request, User Page access request, user subscription information, user's historical behavior record, other Website servers etc.

For example, as shown in Figure 4, E1 is the experiment that obtains Search Results according to user's webpage searching request; The experiment of E2 for Search Results is sorted by advertising rates; The experiment of E3 for Search Results is represented by from left to right mode, experiment final controlling element 11 extracts a user search web-page requests I1 from the Experimental Network flow, and carries out Webpage search according to the search sequence among the I1; Experiment E1 performance element 111a obtains Search Results in this process; Then, experiment E2 performance element 112a sorts by advertising rates to the Search Results that experiment E1 performance element 111a obtains, then; Experiment E3 performance element 113a is provided with Search Results after the ordering that experiment E2 performance element 112a obtains for representing by from left to right mode, thereby obtains result of page searching as feedback information.

Again for example; As shown in Figure 5, E1 is the experiment that obtains Search Results according to user's webpage searching request, the experiment of E2 for Search Results is sorted by advertising rates; The experiment of E3 for Search Results is represented by from left to right mode; Experiment final controlling element 11 extracts a user search web-page requests I2 from the Experimental Network flow, and carries out Webpage search according to the search sequence among the I2, and experiment E1 performance element 111b obtains Search Results in this process; Then, experiment E2 performance element 112b sorts by advertising rates to the Search Results that experiment E1 performance element 111b obtains; When carrying out experiment E2, experiment E3 performance element 113b is concurrent to be provided with Search Results that experiment E1 performance element 111b obtains for representing by from left to right mode, thereby obtains a result of page searching as feedback information.

Again for example; As shown in Figure 6, the experiment of E3 for Search Results being represented by from left to right mode, the experiment that E4 represents for the mode that the page is shown with yellow background according to user's webpage searching request; E5 is the pressure test experiment; The maximum pressure load that the Test Network site server can bear extracts a user search web-page requests I3 from the Experimental Network flow, and carries out Webpage search according to the search sequence among the I3; Experiment E3 performance element 113c represents Search Results by from left to right mode in this process, thereby obtains a result of page searching as feedback information; When carrying out experiment E3, experiment E4 performance element 114c is concurrent to be represented result of page searching with the mode that yellow background shows, thereby obtains another result of page searching as feedback information; When carrying out experiment E3, E4, experiment E5 performance element 115c is concurrent to carry out the pressure test experiment, thereby the maximum search pressure load that the acquisition Website server can bear is as another feedback information.

Again for example, as shown in Figure 7, E2 is according to user's webpage searching request; The experiment that Search Results is sorted by advertising rates; The experiment of E3 for Search Results is represented by from left to right mode, E6 is according to user subscription information, pushes this subscription information update content; Experiment final controlling element 11 extracts a user search web-page requests I4 from the Experimental Network flow; Carry out Webpage search according to the search sequence among the I4, experiment E2 performance element 112d sorts by advertising rates to this Search Results in this process, then; Experiment E3 performance element 113d is provided with Search Results after the ordering that experiment E2 performance element 112d obtains for representing by from left to right mode, thereby obtains result of page searching as feedback information; Then, from the Experimental Network flow, extract a user subscription information I5, the corresponding experiment of I5 E6 performance element 116d pushes the update content of this user subscription information, and with this user subscription information update content as feedback information.

The mode that those skilled in the art will be understood that a plurality of experiments of above-mentioned execution or obtain feedback information is merely for example; Other a plurality of experiments of execution existing or that possibly occur from now on and the mode of obtaining feedback information are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

Preferably, experiment final controlling element 11 is to work continuously.Particularly, experiment final controlling element 11 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information.At this; It will be understood by those skilled in the art that " continuing " is meant that experiment final controlling element 11 requires to carry out a plurality of experiments and obtain the experiment feedback information according to mode of operation that set or adjustment in real time, stop the parallel subnetwork flow that utilizes in the Experimental Network flow at least in a long time until experiment final controlling element 11 and carry out a plurality of experiments.

(with reference to Fig. 8) in a preferred embodiment, experimental facilities 1 also comprise experiment relevant information deriving means 12 ' and network traffics deriving means 13 ', and experiment deriving means 12 ' obtains the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out; Network traffics deriving means 13 ' comes the filtration experiment network traffics according to the relevant information of said a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments.Followingly with reference to Fig. 8 the preferred embodiment is described in detail, wherein, experiment final controlling element 11 ' is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information; Its detailed process for for simplicity, is contained in this with way of reference with aforementioned identical with reference to the performed process of experiment final controlling element 11 among the described embodiment of Fig. 2, does not give unnecessary details and do not do.

Particularly; Experiment relevant information deriving means 12 ' is through such as receive the preset parallel relevant information of utilizing a plurality of experiments of network traffics of sending from network equipments such as the webservers according to the communication protocol of making an appointment; Or, from third party device, obtain the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out through the application programming interfaces (API) that third party device provides.For example; Suppose that experimental facilities 1 is the webserver; Experiment relevant information deriving means 12 ' like communication protocols such as http, https, receives the preset parallel relevant information of waiting to carry out a plurality of experiments of utilizing network traffics that this webserver sends according to the communication protocol of making an appointment.And for example; The application programming interfaces (API) that experiment relevant information deriving means 12 ' provides according to third party device; Send to third party device and to obtain the parallel request-related information of utilizing preset a plurality of experiments of network traffics, and receive the relevant information of preset a plurality of experiments that this third party device returns based on this request.At this, the relevant information of experiment includes but not limited to participate in carrying out the experiment of user's request, like a step in the execution user Request Processing, or carries out complete user's Request Processing; Carry out the experiment that content pushes,, carry out user's subscribed content and push like subscription information according to the user; And the mutual experiment of no user, as be used to other pressure test experiment of maximum service level that the system that obtains can provide.

Subsequently, network traffics deriving means 13 ' comes the filtration experiment network traffics according to the relevant information of a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments.Particularly; Network traffics deriving means 13 ' filtration experiment network traffics, as travel through current Experimental Network flow, extract wherein network traffics; These network traffics are analyzed; Extract this network traffics type, this network traffics type and the parallel relevant information of a plurality of experiments of network traffics of utilizing to be carried out that experiment relevant information deriving means 12 ' obtains are carried out matching ratio one by one, if this network traffics type satisfies the relevant information of these a plurality of experiments; Then network traffics deriving means 13 ' obtains this and corresponding these network traffics of these a plurality of experiments; Circulation is carried out and should be operated, and accomplishes until network traffics deriving means 13 ' and filters whole Experimental Network flows, and obtain and the corresponding subnetwork at least of these a plurality of experiments flow.Wherein, the Experimental Network flow includes but not limited to the subnetwork flow that is used for testing of random extraction from the whole network flow or the artificial network flow that is used to test that extracts from the artificial network flow.For example; Experiment relevant information deriving means 12 ' obtains parallel two experiment E2 of network traffics, the relevant information of E4 utilized to be carried out; Wherein, E2 is according to the user search web-page requests, with the experiment that Search Results sorts by advertising rates, and the experiment that E4 represents for the mode that the page is shown with yellow background; Network traffics deriving means 13 ' utilizes the Experimental Network flow that is used to test that from the whole network flow, extracts at random; Extract wherein network traffics I6; I6 is analyzed; Extracting its network traffics type is the User Page access request, and the network traffics type that these two experiments that comprise in the relevant information with this network traffics type and experiment E2, E4 can be handled is carried out matching ratio, comprises the network traffics type of I6 in the network traffics type that the result can handle for experiment E4; But the network traffics type that does not comprise I6 in the network traffics type that experiment E2 can handle, then these network traffics of network traffics deriving means 13 ' filtering I6; Next bar network traffics I7 of network traffics deriving means 13 ' circulation extraction; I7 is analyzed; Extracting its network traffics type is user's webpage searching request; The network traffics type that two experiments that comprise in the relevant information with this network traffics type and experiment E2, E4 can be handled is carried out matching ratio, includes the network traffics type of I7 in the network traffics type that the result can handle for experiment E2, E4, and then network traffics deriving means 13 ' obtains these network traffics I7.Those skilled in the art will be understood that above-mentioned obtain the relevant information of experiment or obtain at least the mode of subnetwork flow be merely for example; Other existing or relevant informations of obtaining experiment that possibly occur from now on and the mode of obtaining subnetwork flow at least are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

(with reference to Fig. 9) experimental facilities 1 also comprises experimental sequence control device 14 in a further advantageous embodiment " (with reference to Figure 10), this experimental sequence control device 14 " according to the relevant information of said a plurality of experiments, confirm the execution sequence of said a plurality of experiments; Wherein, said experiment final controlling element 11 " also be used for according to said execution sequence, the parallel subnetwork flow that utilizes in the Experimental Network flow is at least carried out a plurality of experiments, to obtain the experiment feedback information.Followingly this another preferred embodiment is described in detail wherein, experiment relevant information deriving means 12 with reference to Fig. 9 " obtain the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out; Network traffics deriving means 13 " come the filtration experiment network traffics according to the relevant information of said a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments; Its detailed process for for simplicity, is contained in this with way of reference with aforementioned identical with reference to the performed process of experiment relevant information deriving means 12 ' among the described embodiment of Fig. 8, network traffics deriving means 13 ', does not give unnecessary details and do not do.

Particularly; Experimental sequence control device 14 " according to experiment relevant information deriving means 12 " obtain the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out; Extract a plurality of experiment relation of interdependences in the relevant information of a plurality of experiments; The output of for example testing E1 is the input of experiment E2, confirms the execution sequence of these a plurality of experiments; Wherein, experiment final controlling element 11 " also be used for according to this execution sequence, the parallel subnetwork flow that utilizes in the Experimental Network flow is at least carried out a plurality of experiments, to obtain the experiment feedback information.For example, experiment relevant information deriving means 12 " obtain the parallel a plurality of experiments that utilize network traffics to be carried out and be respectively E1 and obtain the experiment of Search Results, E2 sorts Search Results by advertising rates experiment, E3 represent Search Results by from left to right mode experiment according to user's webpage searching request; Wherein, The relation of interdependence of three experiments is the input that E1 is output as E2 and E3; Then the experimental sequence control device 14 " confirm that according to this dependence the order that three experiments are carried out obtains the experiment output result of E1 for the execution E1 of elder generation, and will export experiment concurrent execution E2 of input and E3 as E2 and E3; Then; Experiment final controlling element 11 " from the Experimental Network flow, extract a user search web-page requests I2, and carry out Webpage search according to the search sequence among the I2, and in this process, carry out experiment E1 to obtain Search Results; then, carry out experiment E2 Search Results is sorted by advertising rates; When carrying out experiment E2, concurrent execution E3 is provided with Search Results for representing by from left to right mode, thereby obtains a result of page searching as feedback information.Those skilled in the art will be understood that the mode of the execution sequence of above-mentioned definite a plurality of experiments is merely for example; The mode of the execution sequence of other definite a plurality of experiments existing or that possibly occur from now on is as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

(with reference to Fig. 2) in a further advantageous embodiment, experiment final controlling element 11 is also parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, and the output of each experiment is merged to obtain the experiment feedback information.Particularly, experiment final controlling element 11 is also parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, thereby obtains the experimental result output of each experiment, and the output of each experiment is merged to obtain the experiment feedback information.For example, the experiment of E2 for Search Results is sorted by advertising rates, the experiment of E3 for Search Results is represented by from left to right mode, the experiment that E4 represents for the mode that the page is shown with yellow background; Experiment final controlling element 11 utilizes user's webpage searching request I8 in the Experimental Network flow; And carry out Webpage search according to the search sequence among the I8; Thereby in this process, carry out the experimental result output that experiment E2 obtains testing E2, thereby and carry out the experimental result output that E3 obtains testing E3, thereby and carry out the experimental result output that E4 obtains testing E4; Three experimental result outputs are merged, and then the result of page searching after obtaining to merge is as feedback information.Those skilled in the art will be understood that the mode of above-mentioned merging acquisition feedback information is merely for example; The mode that other merging existing or that possibly occur from now on obtain feedback informations is as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

(with reference to Fig. 2) in a further advantageous embodiment, experiment final controlling element 11 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, and obtains to test feedback information respectively based on the output of each experiment.Particularly, experiment final controlling element 11 is also parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, thereby obtains the experimental result output of each experiment, and the output of each experiment is generated feedback result respectively.For example, E5 is the pressure test experiment, and the maximum pressure load that the Test Network site server can bear obtains the maximum service rank that this Website server provides, and E6 is according to user subscription information, pushes this subscription information update content; Experiment final controlling element 11 utilizes a user subscription information I9 in the Experimental Network flow; Carrying out experiment E6 pushes the user subscription information update content; With this user subscription information update content as feedback information; Carry out experiment E5 simultaneously and carry out pressure test, the maximum that can bear with the acquisition server pushes pressure load as feedback information.Those skilled in the art will be understood that the above-mentioned mode that obtains feedback information respectively is merely for example; Other existing or modes that obtain feedback information respectively that possibly occur from now on are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

In another preferred embodiment (with reference to Fig. 2), experimental facilities 1 also comprises the gathering-device (not shown), and this gathering-device is collected experimental result information according to said feedback information.Followingly with reference to Fig. 2 this another preferred embodiment is described in detail, wherein, experiment final controlling element 11 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information; Its detailed process for for simplicity, is contained in this with way of reference with aforementioned identical with reference to the performed process of experiment final controlling element 11 among the described embodiment of Fig. 2, does not give unnecessary details and do not do.

Particularly, gathering-device is collected experimental result information according to the experiment feedback information that experiment final controlling element 11 obtains.For example feedback information is a displayed page as a result, and gathering-device is collected experimental result information according to this displayed page as a result.At this, experimental result information includes but not limited to the content revealing amount, represents price etc. thousand times.For example, gathering-device is according to all result of page searching that experiment final controlling element 11 obtains, and what obtain in these result of page searching each advertisement represents number of times, i.e. each advertisement content amount of representing; And represent price and content revealing amount thereof according to the advertisement single of each advertisement that represents in all result of page searching; Through formula " k=content revealing amount ", represent price thousand times that calculate to obtain each advertisement.Those skilled in the art will be understood that the mode of above-mentioned collection experimental result information is merely for example; The mode of other collection experimental result information existing or that possibly occur from now on is as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

Preferably, gathering-device also comprises provides unit (not shown) and collection unit (not shown), and this provides the unit to the user said feedback information to be provided; This collection unit to the response of said feedback information, is collected said experimental result information according to said feedback information and said user.Particularly, the feedback information such as result of page searching according to experiment final controlling element 11 obtains provides the unit for example this feedback information to be sent to this user's subscriber equipment, and the user browses this feedback information through its subscriber equipment; Perhaps the URL of this feedback information is sent to this user's subscriber equipment, clicks this URL through this subscriber equipment, thereby get into respective page to obtain this feedback information by this user.For example, all result of page searching as feedback information that experiment final controlling element 11 obtains provide the unit that this result of page searching is sent to user's subscriber equipment, and this user browses this result of page searching through this subscriber equipment.

The collection unit according to the feedback information that provides the unit to provide to the user with should (etc.) user to this (etc.) response of feedback information; Collect experimental result information; For example according to the result of page searching that provides the unit to provide to the user; Browse and the clicking operation record through what subscriber equipment carried out with these users, experimental result information is collected in the collection unit.At this, experimental result information includes but not limited to the content revealing amount, represents price for thousand times, average clicking rate and average price etc.For example, the collection unit is according to all result of page searching that experiment final controlling element 11 obtains, and what obtain in these result of page searching each advertisement represents number of times, i.e. each advertisement content amount of representing; The collection unit represents price and content revealing amount thereof according to the advertisement single of each advertisement that represents in all result of page searching; Through formula " k=content revealing amount ", represent price thousand times that calculate to obtain each advertisement; The collection unit according to the user through subscriber equipment browsing and the clicking operation record to result of page searching; Obtain the number of clicks of user to each advertisement in all result of page searching; Calculate the user to the number of clicks of each advertisement of being represented in all result of page searching and the ratio that represents number of times of each advertisement in all result of page searching, i.e. user's clicking rate; The collection unit according to the user through subscriber equipment browsing and the clicking operation record to result of page searching; Obtain the user to the number of clicks of each advertisement in all result of page searching and the single price of each advertisement; Through formula " n=ad click number of times "; Example; Other existing or modes that feedback information are provided and collect experimental result information that possibly occur from now on are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

More preferably, experimental facilities 1 also comprises the experiement control device (not shown), when this experiement control device is restrained in said experimental result information, finishes said experiment.Particularly, experiment final controlling element 11 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain feedback information; Gathering-device is collected experimental result information according to the feedback information that the experiment final controlling element obtains; Experiment final controlling element 11 is constantly carried out based on new Experimental Network flow circulation with gathering-device, and experiement control device detects the each experimental result information that obtains of this circulation, and when the experimental result information convergence of collecting, finishes experiment.At this, as for user's clicking rate x1, x2 etc., there is a certain function f (x) in convergence expression experimental result information data, when f (x1), f (x2) etc. converges on f (x0), and the convergence of experimental result information.For example, user's clicking rate that gathering-device is collected is added up, the acquisition clicking rate is x1, x2, x3 etc.; And these user's clicking rates satisfy function f (x), for any real number b>0, have c>0; And when 0<| x1-x0|<c, 0<| x2-x0|<c has function | f (x1)-f (x2) | and<b exists; Then user's clicking rate converges on f (x0), and experiement control device finishes should experiment.Those skilled in the art will be understood that the mode of above-mentioned end experiment is merely for example, and the mode of other end experiments existing or that possibly occur from now on also should be included in the protection range of the present invention, and be contained in this with way of reference as applicable to the present invention.

The Experimental Network flow of different time sections is different, therefore has fluctuation according to the Experimental Network flow of the every period experimental result that is obtained that experimentizes.Carry out this experiment to obtain different experimental results according to the different experiments network traffics, be tending towards convergence up to this experimental result, the experimental result information accuracy reliability that is obtained in view of the above improves greatly.

In another preferred embodiment (with reference to Fig. 8), experimental facilities 1 also comprises flow deriving means (not shown), and the experimental relevant information of this flow deriving means is obtained and the corresponding Experimental Network flow of said a plurality of experiments.Following with reference to Fig. 8 to this again a preferred embodiment be described in detail, wherein, experiment relevant information deriving means 12 ' obtains the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out; Network traffics deriving means 13 ' comes the filtration experiment network traffics according to the relevant information of said a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments; Experiment final controlling element 11 ' is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information; Its detailed process is with aforementioned identical with reference to the performed process of experiment relevant information deriving means 12 ' among the described embodiment of Fig. 8, network traffics deriving means 13 ' and experiment final controlling element 11 '; For for simplicity; Be contained in this with way of reference, do not give unnecessary details and do not do.

Particularly; The flow deriving means is according to the relevant information of the experiment of experiment relevant information deriving means 12 ' acquisition; As test content or experiment parameter setting; Obtain with should (etc.) experiment the corresponding Experimental Network flow of relevant information; Wherein, experiment content includes but not limited to the request of process user Webpage search, the propelling movement of process user subscription information update content, the mutual pressure test experiment of no user etc., and the experiment parameter setting includes but not limited to as testing included geographic range, experimental subjects etc.At this, the obtain manner of Experimental Network flow includes but not limited to from whole network flow or artificial network flow, extract flow randomly as the Experimental Network flow.For example; Preset experiment content is that User Page is opened request in the experimental relevant information; Preset experiment geographic range is the North China; The flow deriving means is according to the IP address of every user request in the whole network flow, extract targetedly open request from North China's User Page network traffics as the Experimental Network flow.At this, those skilled in the art will be understood that in the relevant information of above-mentioned experiment each can be used to obtain the Experimental Network flow separately, also can multinomial combination be used to obtain the Experimental Network flow.Those skilled in the art will be understood that the above-mentioned mode of obtaining the Experimental Network flow is merely for example; Other existing or modes of obtaining the Experimental Network flow that possibly occur from now on are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

Preferably, the flow deriving means is adjusted the ratio of Experimental Network flow in the whole network flow also according to experimental situation; Wherein, said experimental situation include but not limited to following at least each:

Real network flow in the-said whole network flow;

-be used to carry out the time span of said a plurality of experiments.

Particularly, under predetermined Experimental Network flow, the flow deriving means according to the size of the real network flow of experimental situation such as whole network flow, carry out the duration of a plurality of experiments etc., improve or reduce the ratio of extracting the whole network flow with as the Experimental Network flow.For example; Suppose when the real network flow of whole network flow is 150M; Extract in this whole network flow 50% flow as the Experimental Network flow, then when the real network flow of whole network flow is reduced to 100M, under the constant situation of experimental period length; Satisfy the required Experimental Network flow of this experiment for reaching, the flow deriving means improves the withdrawal ratio to 75% of whole network flow.Again for example; Suppose that the real network flow when the whole network flow is 250M; The flow that extracts 40% in this whole network flow is as the Experimental Network flow, then under the situation of the real network flow unchanged of whole network flow, and the experimental period contraction in length; Satisfy the required Experimental Network flow of this experiment for reaching, the flow deriving means improves the withdrawal ratio to 80% of whole network flow.Those skilled in the art will be understood that the above-mentioned mode of obtaining resize ratio is merely for example; The mode of other resize ratios existing or that possibly occur from now on is as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

Preferably, said Experimental Network flow include but not limited to following at least each:

-user network flow;

-artificial network flow.

Particularly, if the Experimental Network flow is the user network flow, then the flow deriving means extracts the subnetwork flow as the Experimental Network flow from the whole network flow of real user; If the Experimental Network flow is the artificial network flow, then the flow deriving means is through carry out the modeling and simulation of discharge model at dedicated emulated platform such as Opnet, Matlab etc., to obtain the artificial network flow.For example, the flow deriving means extracts flow as the Experimental Network flow from the user network flow randomly.For another example; The flow deriving means utilizes dedicated emulated software Matlab; Carry out the modeling and simulation of user network flow; Obtain the artificial network flow identical with the user network flow, this artificial network flow reaches predetermined data on flows and each item performance index, and from this artificial network flow, extracts the Experimental Network flow.Those skilled in the art will be understood that above-mentioned Experimental Network flow is merely for example, and other Experimental Network flows existing or that possibly occur from now on also should be included in the protection range of the present invention, and be contained in this with way of reference as applicable to the present invention.

Figure 11 is the method flow diagram according to one aspect of the invention, and the parallel process of utilizing network traffics to experimentize is shown.Particularly, in step S1, experimental facilities 1 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information.At this; Experimental facilities 1 includes but not limited to network host, single network server, a plurality of webserver collection or based on the set of computers of cloud computing; At this, cloud computing is a kind of of Distributed Calculation, a super virtual machine of being made up of the loosely-coupled computer collection of a group.Those skilled in the art will be understood that above-mentioned experimental facilities is merely for example, and other experimental facilities existing or that possibly occur from now on also should be included in the protection range of the present invention, and be contained in this with way of reference as applicable to the present invention.

More specifically, in step S1, experimental facilities 1 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information.Particularly, shown in figure 12, there are a plurality of parallel experiment E1, E2, E3 that utilize network traffics to be carried out; For Experimental Network flow X, in step S1, experimental facilities 1 is used for a part of network traffics wherein to carry out some experiments of these a plurality of experiments; Like E1,, utilize another part network traffics to carry out at least two experiments in these a plurality of experiments and walk abreast to obtain to test accordingly feedback information; Like E1, E2, to obtain to test accordingly feedback information, perhaps in step S1; Experimental facilities 1 parallel these whole Experimental Network flows that utilize are carried out at least two experiments in these a plurality of experiments; Like E1, E3 and E1, E2, E3, to obtain to test accordingly feedback information, perhaps in step S1; Experimental facilities 1 parallel this overall network flow that utilizes is carried out the whole experiments in these a plurality of experiments, to obtain to test accordingly feedback information.At this, " the parallel utilization " means the same subnetwork flow that utilizes in the network traffics and carries out a plurality of experiments, and the executive mode of these a plurality of experiments is not limited to concurrent execution or order is carried out.Wherein, said Experimental Network flow includes but not limited to the access request of user's webpage searching request, User Page access request, user subscription information, user's historical behavior record, other Website servers etc.

For example, shown in figure 13, E1 is the experiment that obtains Search Results according to user's webpage searching request; The experiment of E2 for Search Results is sorted by advertising rates, the experiment of E3 for Search Results is represented by from left to right mode, experimental facilities 1 extracts a user search web-page requests I1 from the Experimental Network flow; And carry out Webpage search according to the search sequence among the I1, in step S11a, experimental facilities 1 is carried out experiment E1 to obtain Search Results; Then, in step S12a, experimental facilities 1 is carried out experiment E2 the Search Results that it obtains in step S11a is sorted by advertising rates; Then; In step S13a, experimental facilities 1 is carried out E3 and Search Results after its ordering that in step S12a, obtains is set for representing by from left to right mode, thereby obtains result of page searching as feedback information.

Again for example, shown in figure 14, E1 is the experiment that obtains Search Results according to user's webpage searching request; The experiment of E2 for Search Results is sorted by advertising rates, the experiment of E3 for Search Results is represented by from left to right mode, experimental facilities 1 extracts a user search web-page requests I2 from the Experimental Network flow; And carry out Webpage search according to the search sequence among the I2; In step S11b, experimental facilities 1 is carried out experiment E1 to obtain Search Results, then; In step S12b, experimental facilities 1 sorts by advertising rates to the Search Results that it obtains in step S11b; When carrying out experiment E2, experimental facilities 1 is concurrent to be carried out E3 and its Search Results that in step S11b, obtains is set for representing by from left to right mode in step S13b, thereby obtains a result of page searching as feedback information.

Again for example, shown in figure 15, the experiment of E3 for Search Results being represented by from left to right mode according to user's webpage searching request; The experiment that E4 represents for the mode that the page is shown with yellow background; E5 is the pressure test experiment, the maximum pressure load that the Test Network site server can bear, and experimental facilities 1 extracts a user search web-page requests I3 from the Experimental Network flow; And carry out Webpage search according to the search sequence among the I3; In step S13c, experimental facilities 1 is carried out experiment E3 Search Results is represented by from left to right mode, thereby obtains a result of page searching as feedback information; When carrying out experiment E3, experimental facilities 1 concurrent in step S 14c concurrent execution E4 result of page searching is represented with the mode that yellow background shows, thereby obtain another result of page searching as feedback information; When carrying out experiment E3, E4, experimental facilities 1 concurrent in step S15c concurrent execution experiment E5 carry out the pressure test experiment, thereby obtain maximum search pressure load that Website server can bear as another feedback information.

Again for example, shown in figure 16, E2 is according to user's webpage searching request; With the experiment that Search Results sorts by advertising rates, the experiment of E3 for Search Results is represented by from left to right mode, E6 is according to user subscription information; Push this subscription information update content, experimental facilities 1 extracts a user search web-page requests I4 from the Experimental Network flow, carry out Webpage search according to the search sequence among the I4; In step S12d; Experimental facilities 1 is carried out experiment E2 this Search Results is sorted by advertising rates, then, and in step S13d; Experimental facilities 1 is carried out E3 and Search Results after its ordering that in step S12d, obtains is set for representing by from left to right mode, thereby obtains result of page searching as feedback information; Then, from the Experimental Network flow, extract a user subscription information I5, in step S16d, experimental facilities 1 is carried out the corresponding experiment of I5 E6 pushing the update content of this user subscription information, and with this user subscription information update content as feedback information.

The mode that those skilled in the art will be understood that a plurality of experiments of above-mentioned execution or obtain feedback information is merely for example; Other a plurality of experiments of execution existing or that possibly occur from now on and the mode of obtaining feedback information are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

Preferably, experimental facilities 1 is to work continuously in step S1.Particularly, in step S1, experimental facilities 1 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information.At this; It will be understood by those skilled in the art that " continuing " is meant that experimental facilities 1 requires to carry out a plurality of experiments and obtain the experiment feedback information according to mode of operation that set or adjustment in real time in step S1, stop the parallel subnetwork flow that utilizes in the Experimental Network flow at least in a long time until experimental facilities 1 and carry out a plurality of experiments.

(with reference to Figure 17) in a preferred embodiment, this process also comprises step S2 ' and step S3 ', in step S2 ', experimental facilities 1 obtains the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out; In step S3 ', experimental facilities 1 comes the filtration experiment network traffics according to the relevant information of said a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments.Followingly with reference to Figure 17 the preferred embodiment is described in detail, wherein, in step S1 ', experimental facilities 1 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information; Its detailed process for for simplicity, is contained in this with way of reference with aforementioned identical with reference to the performed process in step S1 of experimental facilities 1 among the described embodiment of Figure 11, does not give unnecessary details and do not do.

Particularly; In step S2 '; Experimental facilities 1 is through such as receive the preset parallel relevant information of utilizing a plurality of experiments of network traffics of sending from network equipments such as the webservers according to the communication protocol of making an appointment; Or, from third party device, obtain the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out through the application programming interfaces (API) that third party device provides.For example; Suppose that experimental facilities 1 is the webserver, in step S2 ', experimental facilities 1 is according to the communication protocol of making an appointment; Like communication protocols such as http, https, receive the preset parallel relevant information of waiting to carry out a plurality of experiments of utilizing network traffics that this webserver sends.And for example; In step S2 '; The application programming interfaces that experimental facilities 1 provides according to third party device (API); Send to third party device and to obtain the parallel request-related information of utilizing preset a plurality of experiments of network traffics, and receive the relevant information of preset a plurality of experiments that this third party device returns based on this request.At this, the relevant information of experiment includes but not limited to participate in carrying out the experiment of user's request, like a step in the execution user Request Processing, or carries out complete user's Request Processing; Carry out the experiment that content pushes,, carry out user's subscribed content and push like subscription information according to the user; And the mutual experiment of no user, as be used to other pressure test experiment of maximum service level that the system that obtains can provide.

Subsequently, in step S3 ', experimental facilities 1 comes the filtration experiment network traffics according to the relevant information of a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments.Particularly, in step S3 ', experimental facilities 1 filtration experiment network traffics; As travel through current Experimental Network flow, and extract wherein network traffics, these network traffics are analyzed; Extract this network traffics type; The parallel relevant information of a plurality of experiments of network traffics of utilizing to be carried out that this network traffics type and experimental facilities 1 obtain is carried out matching ratio one by one, if this network traffics type satisfies the relevant information of these a plurality of experiments, then in step S3 ' in step S2 '; Experimental facilities 1 obtains this and corresponding these network traffics of these a plurality of experiments; Circulation is carried out and should be operated, and accomplishes in step S3 ' until experimental facilities 1 and filters whole Experimental Network flows, and obtain and the corresponding subnetwork at least of these a plurality of experiments flow.Wherein, the Experimental Network flow includes but not limited to the subnetwork flow that is used for testing of random extraction from the whole network flow or the artificial network flow that is used to test that extracts from the artificial network flow.For example; In step S2 '; Experimental facilities 1 obtains parallel two experiment E2 of network traffics, the relevant information of E4 utilized to be carried out, and wherein, E2 is according to the user search web-page requests; With the experiment that Search Results sorts by advertising rates, the experiment that E4 represents for the mode that the page is shown with yellow background; In step S3 '; Experimental facilities 1 utilizes the Experimental Network flow that is used to test that from the whole network flow, extracts at random, extracts wherein network traffics I6, and I6 is analyzed; Extracting its network traffics type is the User Page access request; The network traffics type that these two experiments that comprise in the relevant information with this network traffics type and experiment E2, E4 can be handled is carried out matching ratio, comprises the network traffics type of I6 in the network traffics type that the result can handle for experiment E4, but does not comprise the network traffics type of I6 in the network traffics type that experiment E2 can handle; Then in step S3 ', these network traffics of experimental facilities 1 filtering I6; Experimental facilities 1 is next bar network traffics I7 of circulation extraction in step S3 '; I7 is analyzed; Extracting its network traffics type is user's webpage searching request; The network traffics type that two experiments that comprise in the relevant information with this network traffics type and experiment E2, E4 can be handled is carried out matching ratio, includes the network traffics type of I7 in the network traffics type that the result can handle for experiment E2, E4, and then experimental facilities 1 ' obtains these network traffics I7.Those skilled in the art will be understood that above-mentioned obtain the relevant information of experiment or obtain at least the mode of subnetwork flow be merely for example; Other existing or relevant informations of obtaining experiment that possibly occur from now on and the mode of obtaining subnetwork flow at least are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

(with reference to Figure 18) in a further advantageous embodiment, this process also comprises step S4 ", at step S4 " in, experimental facilities 1 is confirmed the execution sequence of said a plurality of experiments according to the relevant information of said a plurality of experiments; Wherein, at step S1 " in, experimental facilities 1 also is used for according to said execution sequence, and the parallel subnetwork flow that utilizes in the Experimental Network flow is at least carried out a plurality of experiments, to obtain the experiment feedback information.Followingly this another preferred embodiment is described in detail with reference to Figure 18, wherein, at step S2 " in, experimental facilities 1 obtains the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out; At step S3 " in, experimental facilities 1 comes the filtration experiment network traffics according to the relevant information of said a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments; Its detailed process for for simplicity, is contained in this with way of reference with aforementioned identical with reference to the performed process in step S2 ' and step S3 ' of experimental facilities 1 among the described embodiment of Figure 17, does not give unnecessary details and do not do.

Particularly; At step S4 " in; experimental facilities 1 according to it at step S2 " in obtain the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out; Extract a plurality of experiment relation of interdependences in the relevant information of a plurality of experiments, the output of for example testing E1 is the input of experiment E2, confirms the execution sequence of these a plurality of experiments; Wherein, at step S1 " in, experimental facilities 1 also is used for according to this execution sequence, and the parallel subnetwork flow that utilizes in the Experimental Network flow is at least carried out a plurality of experiments, to obtain the experiment feedback information.For example; At step S2 " in, experimental facilities 1 obtains the parallel a plurality of experiments that utilize network traffics to be carried out and is respectively E1 and obtains the experiment of Search Results, E2 sorts Search Results by advertising rates experiment, E3 represent Search Results by from left to right mode experiment according to user's webpage searching request; Wherein, The relation of interdependence of three experiments is the input that E1 is output as E2 and E3; Then at step S4 " in; experimental facilities 1 confirms that according to this dependence the order that three experiments are carried out obtains the experiment output result of E1 for the execution E1 of elder generation, and will export experiment concurrent execution E2 of input and the E3 as E2 and E3; At step S1 " in; experimental facilities 1 extracts a user search web-page requests I2 from the Experimental Network flow, and carries out Webpage search according to the search sequence among the I2, in this process, carries out experiment E1 to obtain Search Results; then, carries out experiment E2 Search Results is sorted by advertising rates; When carrying out experiment E2, concurrent execution E3 is provided with Search Results for representing by from left to right mode, thereby obtains a result of page searching as feedback information.Those skilled in the art will be understood that the mode of the execution sequence of above-mentioned definite a plurality of experiments is merely for example; The mode of the execution sequence of other definite a plurality of experiments existing or that possibly occur from now on is as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

(with reference to Figure 11) in a further advantageous embodiment, in step S1, experimental facilities 1 is also parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, and the output of each experiment is merged to obtain the experiment feedback information.Particularly, in step S1, experimental facilities 1 is also parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, thereby obtains the experimental result output of each experiment, and the output of each experiment is merged to obtain the experiment feedback information.For example, the experiment of E2 for Search Results is sorted by advertising rates, the experiment of E3 for Search Results is represented by from left to right mode, the experiment that E4 represents for the mode that the page is shown with yellow background; In step S1; Experimental facilities 1 utilizes user's webpage searching request I8 in the Experimental Network flow, and carries out Webpage search according to the search sequence among the I8, thereby in this process, carries out the experimental result output that experiment E2 obtains testing E2; And thereby execution E3 obtains testing the experimental result output of E3; Thereby and carry out the experimental result output that E4 obtains testing E4, three experimental results outputs are merged, and then the result of page searching after obtaining to merge is as feedback information.Those skilled in the art will be understood that the mode of above-mentioned merging acquisition feedback information is merely for example; The mode that other merging existing or that possibly occur from now on obtain feedback informations is as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

(with reference to Figure 11) in a further advantageous embodiment, in step S1, experimental facilities 1 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, and obtains to test feedback information respectively based on the output of each experiment.Particularly, in step S1, experimental facilities 1 is also parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, thereby obtains the experimental result output of each experiment, and the output of each experiment is generated feedback result respectively.For example, E5 is the pressure test experiment, and the maximum pressure load that the Test Network site server can bear obtains the maximum service rank that this Website server provides, and E6 is according to user subscription information, pushes this subscription information update content; In step S1; Experimental facilities 1 utilizes a user subscription information I9 in the Experimental Network flow; Carrying out experiment E6 pushes the user subscription information update content; This user subscription information update content as feedback information, is carried out experiment E5 simultaneously and carried out pressure test, and the maximum that can bear with the acquisition server pushes pressure load as feedback information.Those skilled in the art will be understood that the above-mentioned mode that obtains feedback information respectively is merely for example; Other existing or modes that obtain feedback information respectively that possibly occur from now on are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

In another preferred embodiment (with reference to Figure 11), this process also comprises step S5 (not shown), and in step S5, experimental facilities 1 is collected experimental result information according to said feedback information.Followingly with reference to Figure 11 this another preferred embodiment is described in detail, wherein, in step S1, experimental facilities 1 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information; Its detailed process for for simplicity, is contained in this with way of reference with aforementioned identical with reference to the performed process in step S1 of experimental facilities 1 among the described embodiment of Figure 11, does not give unnecessary details and do not do.

Particularly, in step S5, experimental facilities 1 is collected experimental result information according to its experiment feedback information that in step S1, obtains.For example feedback information is a displayed page as a result, and in step S5, experimental facilities 1 is collected experimental result information according to this displayed page as a result.At this, experimental result information includes but not limited to the content revealing amount, represents price etc. thousand times.For example, in step S5, experimental facilities 1 is according to its all result of page searching that in step S1, obtain, and what obtain in these result of page searching each advertisement represents number of times, i.e. each advertisement content amount of representing; And represent price and content revealing amount thereof according to the advertisement single of each advertisement that represents in all result of page searching; Through formula " k=content revealing amount ", represent price thousand times that calculate to obtain each advertisement.Those skilled in the art will be understood that the mode of above-mentioned collection experimental result information is merely for example; The mode of other collection experimental result information existing or that possibly occur from now on is as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

Preferably, this process also comprises step S51 (not shown) and S52 (not shown), and in step S51, experimental facilities 1 provides said feedback information to the user; In step S52, experimental facilities 1 to the response of said feedback information, is collected said experimental result information according to said feedback information and said user.Particularly, according to the feedback information such as result of page searching that experimental facilities 1 obtains in step S1, in step S51, experimental facilities 1 for example is sent to this feedback information this user's subscriber equipment, and the user browses this feedback information through its subscriber equipment; Perhaps the URL of this feedback information is sent to this user's subscriber equipment, clicks this URL through this subscriber equipment, thereby get into respective page to obtain this feedback information by this user.For example; In step S1, all result of page searching as feedback information that experimental facilities 1 obtains are in step S51; Experimental facilities 1 is sent to user's subscriber equipment with this result of page searching, and this user browses this result of page searching through this subscriber equipment.

In step S52; Experimental facilities 1 according to its feedback information that in step S51, provides to the user with should (etc.) user to this (etc.) response of feedback information; Collect experimental result information, for example browse and the clicking operation record through what subscriber equipment carried out according to result of page searching and these users that experimental facilities 1 provides to the user in step S51; In step S52, experimental facilities 1 is collected experimental result information.At this, experimental result information includes but not limited to the content revealing amount, represents price for thousand times, average clicking rate and average price etc.For example, in step S52, experimental facilities 1 is according to its all result of page searching that in step S1, obtain, and what obtain in these result of page searching each advertisement represents number of times, i.e. each advertisement content amount of representing; In step S52; Experimental facilities 1 represents price and content revealing amount thereof according to the advertisement single of each advertisement that represents in all result of page searching; Through formula " k=content revealing amount ", represent price thousand times that calculate to obtain each advertisement; In step S52; Experimental facilities 1 according to the user through subscriber equipment browsing and the clicking operation record to result of page searching; Obtain the number of clicks of user to each advertisement in all result of page searching; Calculate the user to the number of clicks of each advertisement of being represented in all result of page searching and the ratio that represents number of times of each advertisement in all result of page searching, i.e. user's clicking rate; In step S52; Experimental facilities 1 according to the user through subscriber equipment browsing and the clicking operation record to result of page searching; Obtain the user to the number of clicks of each advertisement in all result of page searching and the single price of each advertisement; Through formula " n=ad click number of times ", calculate the average price that obtains each advertisement.Those skilled in the art will be understood that the above-mentioned mode of feedback information or collection experimental result information that provides is merely for example; Other existing or modes that feedback information are provided and collect experimental result information that possibly occur from now on are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

More preferably, this process also comprises step S6 (not shown), in step S6, when experimental facilities 1 is restrained in said experimental result information, finishes said experiment.Particularly, in step S1, experimental facilities 1 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain feedback information; In step S5, experimental facilities 1 is collected experimental result information according to its feedback information that in step S1, obtains; In step S1 and step S5, experimental facilities 1 is constantly carried out based on new Experimental Network flow circulation, and in step S6, experimental facilities 1 detects the each experimental result information that obtains of this circulation, and when the experimental result information convergence of collecting, finishes experiment.At this, as for user's clicking rate x1, x2 etc., there is a certain function f (x) in convergence expression experimental result information data, when f (x1), f (x2) etc. converges on f (x0), and the convergence of experimental result information.For example, user's clicking rate that experimental facilities 1 is collected in step S5 is added up, the acquisition clicking rate is x1, x2, x3 etc.; And these user's clicking rates satisfy function f (x), for any real number b>0, have c>0; And when 0<| x1-x0|<c, 0<| x2-x0|<c has function | f (x1)-f (x2) | and<b exists; Then user's clicking rate converges on f (x0), and in step S6, experimental facilities 1 finishes should experiment.Those skilled in the art will be understood that the mode of above-mentioned end experiment is merely for example, and the mode of other end experiments existing or that possibly occur from now on also should be included in the protection range of the present invention, and be contained in this with way of reference as applicable to the present invention.

The Experimental Network flow of different time sections is different, therefore has fluctuation according to the Experimental Network flow of the every period experimental result that is obtained that experimentizes.Carry out this experiment to obtain different experimental results according to the different experiments network traffics, be tending towards convergence up to this experimental result, the experimental result information accuracy reliability that is obtained in view of the above improves greatly.

In another preferred embodiment (with reference to Figure 17), this process also comprises step S7 ', and in step S7 ', experimental facilities 1 experimental relevant information is obtained and the corresponding Experimental Network flow of said a plurality of experiments.Following with reference to Figure 17 to this again a preferred embodiment be described in detail, wherein, in step S2 ', experimental facilities 1 obtains the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out; In step S3 ', experimental facilities 1 comes the filtration experiment network traffics according to the relevant information of said a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments; In step S1 ', experimental facilities 1 is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information; Its detailed process for for simplicity, is contained in this with way of reference with aforementioned identical with reference to the performed process in step S2 ', step S3 ' neutralization procedure S1 ' of experimental facilities 1 among the described embodiment of Figure 17, does not give unnecessary details and do not do.

Particularly; In step S7 '; Experimental facilities 1 is according to the relevant information of its experiment that in step S2 ', obtains; As test content or experiment parameter setting, obtain with should (etc.) experiment the corresponding Experimental Network flow of relevant information, wherein; Experiment content includes but not limited to the request of process user Webpage search, the propelling movement of process user subscription information update content, the mutual pressure test experiment of no user etc., and the experiment parameter setting includes but not limited to as testing included geographic range, experimental subjects etc.At this, the obtain manner of Experimental Network flow includes but not limited to from whole network flow or artificial network flow, extract flow randomly as the Experimental Network flow.For example; Preset experiment content is that User Page is opened request in the experimental relevant information; Preset experiment geographic range is the North China; In step S7 ', experimental facilities 1 is according to the IP address of every user request in the whole network flow, extract targetedly open request from North China's User Page network traffics as the Experimental Network flow.At this, those skilled in the art will be understood that in the relevant information of above-mentioned experiment each can be used to obtain the Experimental Network flow separately, also can multinomial combination be used to obtain the Experimental Network flow.Those skilled in the art will be understood that the above-mentioned mode of obtaining the Experimental Network flow is merely for example; Other existing or modes of obtaining the Experimental Network flow that possibly occur from now on are as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

Preferably, in step S7 ', experimental facilities 1 is adjusted the ratio of Experimental Network flow in the whole network flow also according to experimental situation; Wherein, said experimental situation include but not limited to following at least each:

Real network flow in the-said whole network flow;

-be used to carry out the time span of said a plurality of experiments.

Particularly; Under predetermined Experimental Network flow; In step S7 ', experimental facilities 1 according to the size of the real network flow of experimental situation such as whole network flow, carry out the duration of a plurality of experiments etc., improve or reduce the ratio of extracting the whole network flow with as the Experimental Network flow.For example; Suppose when the real network flow of whole network flow is 150M, extract in this whole network flow 50% flow as the Experimental Network flow, then when the real network flow of whole network flow is reduced to 100M; Under the constant situation of experimental period length; Satisfy the required Experimental Network flow of this experiment for reaching, in step S7 ', experimental facilities 1 improves the withdrawal ratio to 75% of whole network flow.Again for example; Suppose that the real network flow when the whole network flow is 250M, the flow that extracts 40% in this whole network flow is as the Experimental Network flow, then under the situation of the real network flow unchanged of whole network flow; The experimental period contraction in length; Satisfy the required Experimental Network flow of this experiment for reaching, in step S7 ', experimental facilities 1 improves the withdrawal ratio to 80% of whole network flow.Those skilled in the art will be understood that the above-mentioned mode of obtaining resize ratio is merely for example; The mode of other resize ratios existing or that possibly occur from now on is as applicable to the present invention; Also should be included in the protection range of the present invention, and be contained in this with way of reference.

Preferably, said Experimental Network flow include but not limited to following at least each:

-user network flow;

-artificial network flow.

Particularly, if the Experimental Network flow is the user network flow, then in step S7 ', experimental facilities 1 extracts the subnetwork flow as the Experimental Network flow from the whole network flow of real user; If the Experimental Network flow is the artificial network flow, then in step S7 ', experimental facilities 1 is through carry out the modeling and simulation of discharge model at dedicated emulated platform such as Opnet, Matlab etc., to obtain the artificial network flow.For example, in step S7 ', experimental facilities 1 extracts flow as the Experimental Network flow from the user network flow randomly.For another example; In step S7 '; Experimental facilities 1 utilizes dedicated emulated software Matlab, carries out the modeling and simulation of user network flow, obtains the artificial network flow identical with the user network flow; This artificial network flow reaches predetermined data on flows and each item performance index, and from this artificial network flow, extracts the Experimental Network flow.Those skilled in the art will be understood that above-mentioned Experimental Network flow is merely for example, and other Experimental Network flows existing or that possibly occur from now on also should be included in the protection range of the present invention, and be contained in this with way of reference as applicable to the present invention.

To those skilled in the art, obviously the invention is not restricted to the details of above-mentioned example embodiment, and under the situation that does not deviate from spirit of the present invention or essential characteristic, can realize the present invention with other concrete form.Therefore; No matter from which point; All should regard embodiment as exemplary; And be nonrestrictive, scope of the present invention is limited accompanying claims rather than above-mentioned explanation, therefore is intended to the implication of the equivalents that drops on claim and all changes in the scope are included in the present invention.Should any Reference numeral in the claim be regarded as limit related claim.In addition, obviously other unit or step do not got rid of in " comprising " speech, and odd number is not got rid of plural number.A plurality of unit of stating in the device claim or device also can be realized through software or hardware by a unit or device.The first, the second word such as grade is used for representing title, and does not represent any specific order.

Claims (22)

1. method that experimentizes by computer implemented flow Network Based, wherein, this method may further comprise the steps:
A is parallel to utilize in the Experimental Network flow at least the subnetwork flow to carry out a plurality of experiments, to obtain the experiment feedback information.
2. method according to claim 1, wherein, this method also comprises:
-obtain the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out;
-come the filtration experiment network traffics according to the relevant information of said a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments.
3. method according to claim 2, wherein, this method also comprises:
-according to the relevant information of said a plurality of experiments, confirm the execution sequence of said a plurality of experiments;
Wherein, said steps A also comprises:
-according to said execution sequence, the parallel subnetwork flow that utilizes in the Experimental Network flow is at least carried out a plurality of experiments, to obtain the experiment feedback information.
4. according to each described method in the claim 1 to 3, wherein, said steps A also comprises:
-parallel subnetwork the flow that utilizes in the Experimental Network flow is at least carried out a plurality of experiments, and the output of each experiment is merged to obtain the experiment feedback information.
5. according to each described method in the claim 1 to 3, wherein, said steps A also comprises:
-parallel subnetwork the flow that utilizes in the Experimental Network flow is at least carried out a plurality of experiments, and is obtained to test feedback information respectively based on the output of each experiment.
6. according to each described method in the claim 1 to 5, wherein, this method also comprises:
I collects experimental result information according to said feedback information.
7. method according to claim 6, wherein, said step I also comprises:
-to the user said feedback information is provided;
-according to said feedback information and said user to the response of said feedback information, collect said experimental result information.
8. according to each described method in the claim 6 to 7, wherein, this method also comprises:
-repeat said steps A and i, up to said experimental result information convergence.
9. according to each described method in the claim 2 to 8, wherein, this method also comprises:
X obtains and the corresponding Experimental Network flow of said a plurality of experiments according to the relevant information of said a plurality of experiments.
10. method according to claim 9, wherein, said step x also comprises:
-according to experimental situation, adjust the ratio of said Experimental Network flow in the whole network flow;
Wherein, said experimental situation comprise following at least each:
Real network flow in the-said whole network flow;
-be used to carry out the time span of said a plurality of experiments.
11. method according to claim 10, wherein, said Experimental Network flow comprise following at least each:
-user network flow;
-artificial network flow.
12. the equipment that flow Network Based experimentizes, wherein, this equipment comprises:
The experiment final controlling element, be used for parallel utilize the Experimental Network flow at least the subnetwork flow carry out a plurality of experiments, to obtain the experiment feedback information.
13. equipment according to claim 12, wherein, this equipment also comprises:
Experiment relevant information deriving means is used to obtain the parallel relevant information of utilizing a plurality of experiments of network traffics to be carried out;
The network traffics deriving means is used for coming the filtration experiment network traffics according to the relevant information of said a plurality of experiments, to obtain and the corresponding said flow of subnetwork at least of said a plurality of experiments.
14. equipment according to claim 13, wherein, this equipment also comprises:
The experimental sequence control device is used for the relevant information according to said a plurality of experiments, confirms the execution sequence of said a plurality of experiments;
Wherein, said experiment final controlling element also is used for according to said execution sequence, and the parallel subnetwork flow that utilizes in the Experimental Network flow is at least carried out a plurality of experiments, to obtain the experiment feedback information.
15. according to each described equipment in the claim 12 to 14; Wherein, The experiment final controlling element also be used for parallel utilize the Experimental Network flow at least the subnetwork flow carry out a plurality of experiments, the output of each experiment is merged to obtain the experiment feedback information.
16. according to each described equipment in the claim 12 to 14, wherein, the experiment final controlling element also be used for parallel utilize the Experimental Network flow at least the subnetwork flow carry out a plurality of experiments, and obtain to test feedback information respectively based on the output of each experiment.
17. according to each described equipment in the claim 12 to 16, wherein, this equipment also comprises:
Gathering-device is used for according to said feedback information, collects experimental result information.
18. equipment according to claim 17, wherein, said gathering-device also comprises:
The unit is provided, is used for said feedback information being provided to the user;
The collection unit is used for according to said feedback information and said user said experimental result information is collected in the response of said feedback information.
19. according to each described equipment in the claim 17 to 18, wherein, this equipment also comprises:
Experiement control device is used for when said experimental result information restrains, finishing said experiment.
20. according to each described equipment in the claim 13 to 19, wherein, this equipment also comprises:
The flow deriving means is used for experimental relevant information, obtains and the corresponding Experimental Network flow of said a plurality of experiments.
21. equipment according to claim 20, wherein, said flow deriving means also is used for according to experimental situation, the ratio of adjustment Experimental Network flow in the whole network flow;
Wherein, said experimental situation comprise following at least each:
Real network flow in the-said whole network flow;
-be used to carry out the time span of said a plurality of experiments.
22. equipment according to claim 21, wherein, said Experimental Network flow comprise following at least each:
-user network flow;
-artificial network flow.
CN2011101214926A 2011-05-11 2011-05-11 Method and equipment for carrying out experiments based on network flow CN102780573A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN2011101214926A CN102780573A (en) 2011-05-11 2011-05-11 Method and equipment for carrying out experiments based on network flow

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN2011101214926A CN102780573A (en) 2011-05-11 2011-05-11 Method and equipment for carrying out experiments based on network flow

Publications (1)

Publication Number Publication Date
CN102780573A true CN102780573A (en) 2012-11-14

Family

ID=47125339

Family Applications (1)

Application Number Title Priority Date Filing Date
CN2011101214926A CN102780573A (en) 2011-05-11 2011-05-11 Method and equipment for carrying out experiments based on network flow

Country Status (1)

Country Link
CN (1) CN102780573A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105243006A (en) * 2015-09-30 2016-01-13 百度在线网络技术(北京)有限公司 Flow layer setting method and apparatus based on flow experiment and flow experiment implementing method and apparatus

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101136839A (en) * 2006-08-31 2008-03-05 中兴通讯股份有限公司 Method and equipment for discovering and controlling terminal-to-terminal equity network user flux
US20080313701A1 (en) * 2007-06-14 2008-12-18 Choi Young Han System and method for managing network by value-based estimation
CN101661474A (en) * 2008-08-26 2010-03-03 华为技术有限公司 Search method and system
CN101685521A (en) * 2008-09-23 2010-03-31 北京搜狗科技发展有限公司 Method for showing advertisements in webpage and system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101136839A (en) * 2006-08-31 2008-03-05 中兴通讯股份有限公司 Method and equipment for discovering and controlling terminal-to-terminal equity network user flux
US20080313701A1 (en) * 2007-06-14 2008-12-18 Choi Young Han System and method for managing network by value-based estimation
CN101661474A (en) * 2008-08-26 2010-03-03 华为技术有限公司 Search method and system
CN101685521A (en) * 2008-09-23 2010-03-31 北京搜狗科技发展有限公司 Method for showing advertisements in webpage and system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
王海波: "《中国优秀硕士学位论文全文数据库信息科技辑》", 30 August 2009 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105243006A (en) * 2015-09-30 2016-01-13 百度在线网络技术(北京)有限公司 Flow layer setting method and apparatus based on flow experiment and flow experiment implementing method and apparatus

Similar Documents

Publication Publication Date Title
JP6211605B2 (en) Ranking search results based on click-through rate
CN103761309B (en) Operation data processing method and system
US20170032251A1 (en) Method and system for applying probabilistic topic models to content in a tax environment to improve user satisfaction with a question and answer customer support system
CN105027121B (en) The five application page of the machine application is indexed
CN101911042B (en) The relevance ranking of the browser history of user
CN102411583B (en) Method and device for matching texts
CN103473230B (en) Service area determines that method, logistics service provider recommend method and related device
JP5449628B2 (en) Determining category information using multistage
CN102394798B (en) Multi-feature based prediction method of propagation behavior of microblog information and system thereof
CN104750789B (en) The recommendation method and device of label
CN102087577B (en) Location independent execution of user interface operations
CN102999586B (en) A kind of method and apparatus of recommendation of websites
CN103020846B (en) A kind of method for pushing and system of mobile application
CN101957844B (en) On-line application system and implementation method thereof
CN105117474A (en) Method and device for loading recommendation information in webpage reading mode
US9135647B2 (en) Methods and systems for flexible and scalable databases
CN102420813B (en) Method and device for providing target information according to terminal attributes of user equipment
CN104090886A (en) Method and device for constructing real-time portrayal of user
CN104145264A (en) Systems and methods for sentiment detection, measurement, and normalization over social networks
CN103218431B (en) A kind ofly can identify the system that info web gathers automatically
CN103095743A (en) Handling method and system of grey release
CN103886090A (en) Content recommendation method and device based on user favorites
CN104572840A (en) Method and equipment used for providing push message
CN102939601A (en) Initiating font subsets
US10410128B2 (en) Method, device, and server for friend recommendation

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20121114