CN102710748A - Data acquisition method, system and equipment - Google Patents
Data acquisition method, system and equipment Download PDFInfo
- Publication number
- CN102710748A CN102710748A CN2012101333949A CN201210133394A CN102710748A CN 102710748 A CN102710748 A CN 102710748A CN 2012101333949 A CN2012101333949 A CN 2012101333949A CN 201210133394 A CN201210133394 A CN 201210133394A CN 102710748 A CN102710748 A CN 102710748A
- Authority
- CN
- China
- Prior art keywords
- data
- resource
- web page
- client
- page address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a data acquisition method, a data acquisition system and data acquisition equipment, and belongs to the field of computer networks. The method comprises the following steps of: receiving a data acquisition request from a client, wherein the data acquisition request comprises a webpage address; establishing at least two data channels with a webpage server, and capturing data resources corresponding to the webpage address in parallel; and sending the captured data resources to the client. According to a scheme of establishing a plurality of data channels by using a proxy server, then capturing data resources of a webpage and actively sending the data resources to the client, the problems of waste of flow and relatively long response time when the client acquires network data are solved; and an effect that the client can quickly acquire all the data resources of the webpage so as to display the data resources to a user when the data acquisition request is sent by one time is realized.
Description
Technical field
The present invention relates to computer network field, particularly a kind of data capture method, system and equipment.
Background technology
Along with the fast development of mobile Internet, use the mobile terminal viewing network data such as smart mobile phone, panel computer to become people's necessary part in daily life.
Be example to use browser on the smart mobile phone to surf the web; A kind of data capture method that is used to obtain web data that has existed in the prior art is: first; Smart mobile phone and server are set up a TCP (Transmission Control Protocol; Transmission control protocol) channel utilizes this TCP channel to initiate the data acquisition request of a web page address to server then; The second, server returns the primary resource of this web page address to smart mobile phone, and primary resource can simply be interpreted as the related data of the main contents that are used to show this webpage; The 3rd, smart mobile phone carries out buffer memory with this primary resource, and resolves this primary resource to judge whether webpage also has child resource, and child resource can be Javascript script, picture or music or the like; The 4th; If smart mobile phone judges that this webpage also has child resource; Then smart mobile phone need rebulid a TCP channel; Because a same time of TCP channel can only be a data acquisition request service, smart mobile phone utilizes newly-built TCP channel to initiate the data acquisition request of child resource to server then; The 5th, server returns the primary resource of this child resource to smart mobile phone; ,,; The 6th, when webpage comprises a plurality of child resource, repeat the acquisition process of above-mentioned child resource, all got access to smart mobile phone this locality up to the primary resource of this webpage and all child resources, smart mobile phone shows this webpage then.
There is following problem in prior art: the first, and smart mobile phone whenever obtains a child resource and all will rebulid the TCP channel and send one-time request, thereby cause wasting very much flow in the process of obtaining a web data; The second, when a webpage comprises a plurality of child resource, need repeatedly data acquisition; Response time is longer; Add mobile network itself and postpone just than higher, make a webpage successfully need the very long time from beginning to get access to final demonstration, user experience is relatively poor.
Summary of the invention
In order to solve client when obtaining network data, waste flow and response time be the problem of length, and the embodiment of the invention provides a kind of data capture method, system and equipment.Said technical scheme is following:
According to an aspect of the present invention, the embodiment of the invention provides a kind of data capture method, is used for acting server, and said method comprises:
Reception is from the data acquisition request of client, and said data acquisition request comprises web page address;
Set up at least two data channels pairing data resource of the said web page address of extracting that walk abreast with web page server;
The said data resource that grabs is pushed to said client.
According to a further aspect in the invention, the embodiment of the invention also provides a kind of data capture method, is used for client, and said method comprises:
Reception is from user's web page access request, and said web page access request comprises web page address;
Send data acquisition request according to said web page address to acting server;
The corresponding data resource of said web page address that reception is pushed by said acting server.
In accordance with a further aspect of the present invention, the embodiment of the invention also provides a kind of acting server, and said acting server comprises:
The request receiver module is used to receive the data acquisition request from client, and said data acquisition request comprises web page address;
The resource grabbing module is used for setting up at least two data channels pairing data resource of the said web page address of extracting that walk abreast with web page server;
The resource supplying module is used for the data resource that grabs is pushed to said client.
According to another aspect of the invention, the embodiment of the invention also provides a kind of client, and said client comprises:
The web-page requests receiver module is used to receive the web page access request from the user, and said web page access request comprises web page address;
Obtain request sending module, be used for sending data acquisition request to acting server according to said web page address;
The data resource receiver module is used to receive all the corresponding data resources of said web page address that pushed by said acting server.
Also on the one hand the embodiment of the invention also provides a kind of data-acquisition system according to of the present invention, and it comprises the client that provides in the acting server that provides in the embodiment of the invention and the embodiment of the invention.
The beneficial effect that the technical scheme that the embodiment of the invention provides is brought is:
Set up the data resource that many data channel grasp webpage through utilizing acting server; Active push is given the scheme of client then; Solved client when obtaining network data; The problem that waste flow and response time are long has reached client and only need initiate data acquisition request one time, and all data resources that just can obtain webpage fast are to be shown to user's effect.
Description of drawings
In order to be illustrated more clearly in the technical scheme in the embodiment of the invention; The accompanying drawing of required use is done to introduce simply in will describing embodiment below; Obviously, the accompanying drawing in describing below only is some embodiments of the present invention, for those of ordinary skills; Under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the structural representation of the related enforcement environment of the embodiment of the invention;
Fig. 2 is the method flow diagram of the data capture method that provides of the embodiment of the invention one;
Fig. 3 is the method flow diagram of the data capture method that provides of the embodiment of the invention two;
Fig. 4 is the method flow diagram of the data capture method that provides of the embodiment of the invention three;
Fig. 5 is the block diagram of the acting server that provides of the embodiment of the invention four;
Fig. 6 is another block diagram of the acting server that provides of the embodiment of the invention four;
Fig. 7 is a block diagram again of the acting server that provides of the embodiment of the invention four;
Fig. 8 is the another block diagram of the acting server that provides of the embodiment of the invention four;
Fig. 9 is the block diagram of the client that provides of the embodiment of the invention five;
Figure 10 is the another block diagram of the client that provides of the embodiment of the invention five;
Figure 11 is the block diagram of the data-acquisition system that provides of the embodiment of the invention six.
Embodiment
For making the object of the invention, technical scheme and advantage clearer, will combine accompanying drawing that embodiment of the present invention is done to describe in detail further below.
Please at first with reference to figure 1, it shows the structural representation of the related enforcement environment of the embodiment of the invention.This enforcement environment not only comprises the client 110 and web page server 120 that the user uses, and also includes the acting server 130 that is used for bridge joint client 110 and web page server 120.
Wherein, the client 110 that the user uses can be the portable terminal such as mobile phone, panel computer and ultra portable PC, and the user can use this portable terminal to visit network.
Please refer to Fig. 2, it shows the method flow diagram of the data capture method that the embodiment of the invention one provides.This data capture method goes in the acting server shown in Figure 1, also is that present embodiment is mainly described with acting server one side, and this data capture method can comprise:
Acting server can receive the data acquisition request from client; This data acquisition request can be HTTP (Hypertext Transfer Protocol; HTTP) request can comprise web page address in this data acquisition request, and web page address can use URL (Universal Resource Locator; URL) represent, such as: a web page address is http://www.xxx.com/index.html.
Acting server can be set up some data channels with web page server and walk abreast and grasp the pairing data resource of web page address, and wherein, data resource generally includes primary resource and child resource.That is to say; When a pairing data resource of web page address comprises a primary resource and a plurality of child resource; Acting server can be set up some data channels simultaneously with web page server; Every data channel is used for obtaining a primary resource or a child resource, and acting server can get access to pairing all data resources of web page address concurrently through these some data channels then.
Primary resource can be HTML (Hypertext Markup Language, HTML) the described content of speech like sound, is mainly used in the body matter or the framework that show a webpage.Child resource can be a pictures, music file or a JS script (also being the Javascript script), and its major part all is to be come content or the data directly described by the HTML speech like sound.Generally, a webpage is made up of a primary resource and experimental process resource.
Acting server can be pushed to client with the data resource that grabs after pairing all data resources of this web page address are all grabbed local cache.At this moment, client just can obtain the pairing data resource of this web page address.
In sum; The data capture method that the embodiment of the invention one provides walks abreast and grasps the data resource of webpage through utilizing acting server to set up many data channel; Active push is given the scheme of client then, has solved client when obtaining network data, the problem that waste flow and response time are long; Reached client and only need initiate data acquisition request one time, all data resources that just can obtain webpage fast are to be shown to user's effect.
Please refer to Fig. 3, it shows the method flow diagram of the data capture method that the embodiment of the invention two provides.This data capture method goes in the acting server shown in Figure 1, also is that present embodiment is mainly described with acting server one side, and this data capture method can comprise:
When the browser in the client has just begun to start, just can send a lasting passage request of setting up and give acting server, acting server can receive from the lasting passage of client and set up request.
Step 302 is set up lasting passage with client, so that receive at least one data acquisition request from client, and keeps lasting passage through heartbeat signal and client;
Acting server can be set up lasting passage with client, and keep this lasting passage through heartbeat signal and client after the lasting passage foundation request that receives from client.Lasting passage can be meant that the TCP that keeps a long period connects, and it is multiplexed that this TCP connects support, also promptly connects a plurality of HTTP requests of parallel transmission simultaneously at the TCP of a full duplex.After this TCP connection was created, the HTTP that acting server can utilize this TCP to connect and monitor client asked.The implementation procedure of this step can be through accomplishing at TCP layer and the new agreement of HTTP layer realization.
Acting server can receive the data acquisition request from client; This data acquisition request can be the HTTP request; Can comprise web page address in this data acquisition request; Web page address can use URL to represent, such as: a web page address is http://www.xxx.com/index.html.
Acting server can be set up some data channels with web page server and walk abreast and grasp the pairing data resource of web page address, and wherein, data resource comprises primary resource and child resource.That is to say; When a pairing data resource of web page address comprises a primary resource and a plurality of child resource; Acting server can be set up some data channels simultaneously with web page server; Every data channel is used for obtaining a primary resource or a child resource, and acting server can get access to pairing all data resources of web page address concurrently through these some data channels then.This data channel can be meant the TCP passage of setting up between acting server and the web page server; Such as; Acting server can at first be set up a TCP passage with web page server and obtain primary resource; Determining pairing all data resources of web page address when also comprising a plurality of child resource then, setting up many TCP passages respectively and walk abreast and obtain the pairing child resource of web page address.Because client has the restriction that under same domain name, can only set up 6 TCP passages usually, and acting server can be optimized with the network bandwidth based on stronger disposal ability, makes and the concurrent TCP passage of web page server increases to 20.So acting server can get access to pairing all data resources of web page address with very fast speed.
Primary resource can be HTML (Hypertext Markup Language, HTML) the described content of speech like sound, is mainly used in the body matter or the framework that show a webpage.Child resource can be a pictures, music file or a JS script (also being the Javascript script), and its major part all is to be come content or the data directly described by the HTML speech like sound.Generally, a webpage is made up of a primary resource and experimental process resource.
Step 305 judges whether the primary resource or the child resource that get access to comprise script data, if then get into step 306; If, then directly do not get into step 307;
When acting on behalf of server through a data passages, when getting access to primary resource or the child resource of pairing all data resources of web page address, acting server can judge whether comprise script data in this primary resource or the child resource.This script data can be meant the Javascript script.
Step 306 if primary resource that gets access to or child resource comprise script data, is then carried out script data in advance;
Comprise script data if acting server determines in primary resource or the child resource, then can carry out this script data in advance by instead of client.Client alleviates the workload of client significantly with regard to not carrying out executable operations to this part script data like this, can improve the loading velocity of webpage further.
Step 307 is cached to local cache with pairing primary resource of the web page address that gets access to or child resource;
When acting on behalf of server through a data passages, get access to after the primary resource or child resource of pairing all data resources of web page address, can this primary resource or child resource be cached in the local cache.Local cache can be realized by the memory of fair speeds such as internal memory and Flash, also can be realized than the memory of low velocity by hard disk or distributed memory system etc.
More preferably, when this primary resource or child resource are cached to local cache, can specifically comprise:
To carry out buffer memory corresponding to all data resources of a web page address, and do the unique identification that obtains after the hashed according to web page address and store.Simultaneously one visit capacity is set for each unique identification.
Step 308 monitors in the local cache whether be cached to pairing primary resource of web page address or child resource;
Acting server can monitor whether be cached to pairing primary resource of web page address or child resource in the local cache.In other words, in step 307, be cached to a primary resource or child resource, acting server can detect this variation in real time at every turn.Such as; A message queue is set; When being cached to a primary resource or child resource, send a message to this message queue, acting server can learn from this message queue at once and be cached to pairing primary resource of web page address or child resource the local cache at every turn.
Step 309 if be cached to pairing primary resource of web page address or child resource in the local cache, then pushes primary resource or child resource to client.
When act on behalf of server listen to be cached to pairing primary resource of web page address or child resource in the local cache after, then this primary resource of active push or child resource are to client.When pairing all data resources of web page address comprise a primary resource and a plurality of child resource; Step 305 to step 309 will repeat repeatedly; Also be after acting server whenever gets access to a primary resource or child resource; Just carry out step 305 to step 309 one time, so that this primary resource or child resource are pushed to client, till pairing all data resources of this web page address all are pushed to client.
Obviously, in said process, client only need be sent the HTTP request that once comprises web page address to acting server in step 303, just receive getting final product with pairing all data resources of this web page address of acting server propelling movement then always.Utilize stronger disposal ability of acting server and higher high speed bandwidth, can get access to web data and demonstration very fast.
What also need remark additionally is that in step 307, acting server can carry out buffer memory with the data resource that a web page address gets access to.Can come buffer memory as the storage index this moment with the unique identification that this web page address carries out obtaining after the hashed, and corresponding simultaneously and each unique identification is provided with a record: pushed number of times.Moreover, local cache also can be combined to realize by high-speed memory such as internal memory or flash memory and the slow memory such as hard disk or distributed memory system simultaneously.
In such cases; In step 303, after acting on behalf of the data acquisition request that server receives different clients, at first according to the web page address within this data acquisition request; In local cache, search earlier the buffer memory that whether has pairing all data resources of this web page address; If any, leap in the step 309, the pairing data resource of this web page address is pushed to client.Such as, a lot of clients may compare the concentrated area and visit the pairing news pages of same web page address in a period of time.When acting server is pushed to client with the data resource in the local cache at every turn; Can be pushed number of times and added 1 this web page address is pairing; Whether acting server can also push number of times above reservation threshold according to the quilt of some web page addresses then; Select the pairing data resource of this web page address is cached in the Cache, still store in the low speed buffer.
In sum; The data capture method that the embodiment of the invention two provides walks abreast and grasps the data resource of webpage through utilizing acting server to set up many data channel; Active push is given the scheme of client then, has solved client when obtaining network data, the problem that waste flow and response time are long; Reached client and only need initiate data acquisition request one time, all data resources that just can obtain webpage fast are to be shown to user's effect.Simultaneously, the method that present embodiment provides is also through increasing lasting passage mechanism, the preparatory execution mechanism of script data and adopting the mechanism of Cache cache web pages data, reached the effect that acting server can be pushed to web data client quickly.
Please refer to Fig. 4, it shows the method flow diagram of the data capture method that the embodiment of the invention three provides.This data capture method goes in the client shown in Figure 1, also is that present embodiment is mainly described with client one side, and this data capture method can comprise:
Step 401 is sent lasting passage to acting server and is set up request;
When the browser in the client has just begun to start, just can send a lasting passage request of setting up and give acting server, acting server can receive from the lasting passage of client and set up request.
Step 402 is set up lasting passage with acting server, so that utilize lasting passage to send at least one data acquisition request to acting server, and keeps lasting passage through heartbeat signal and acting server;
Acting server can feed back this lasting passage foundation request of whether accepting after the lasting passage foundation request that receives from client.If accept, then client can be set up lasting passage with acting server, and keeps this lasting passage through heartbeat signal and acting server.Lasting passage can be meant that the TCP that keeps a long period connects, and it is multiplexed that this TCP connects support, also promptly connects a plurality of HTTP requests of parallel transmission simultaneously at the TCP of a full duplex.After this TCP connection was created, the HTTP that acting server can utilize this TCP to connect and monitor client asked.The implementation procedure of this step can be through accomplishing at TCP layer and the new agreement of HTTP layer realization.
Step 403 receives the web page access request from the user, and the web page access request comprises web page address;
When the user uses the browser on the client, can initiate the web page access request to client through this browser, include web page address in this web page access request, client can receive the web page access request from the user.
Step 404 is sent data acquisition request according to web page address to acting server;
Client receives after the web page access request from the user, can utilize the lasting passage of setting up in advance in the step 402 to send data acquisition request to acting server, and this data acquisition request comprises web page address.
Because lasting passage need be kept through heartbeat signal, the situation of lasting channel failure may appear, and for this reason, this step can also specifically comprise:
The first, judge whether to exist lasting passage;
Client has at first judged whether and has had between the acting server lasting passage after the web page access request that receives from the user.
The second, if there has been lasting passage, then continue to judge whether lasting passage is normal;
If client has been judged and has been had lasting passage between the acting server, then continued to judge whether lasting passage is normal.
The 3rd, if lasting passage is normal, then utilize lasting passage to send data acquisition request to acting server, this data acquisition request comprises web page address.
If client continues to judge that lasting passage is normal, just utilize lasting passage to send data acquisition request to acting server, this data acquisition request comprises web page address.
Step 405 receives the corresponding data resource of web page address that is pushed by acting server.
Client awaits receives acting server and pushes the corresponding data resource of web page address of coming.If the local cache in the acting server has had the corresponding data resource of this web page address; Then acting server can all be pushed to client terminal local by disposable all data resources that this web page address is corresponding; Client all data resources that this web page address is corresponding store in the buffer memory of client terminal local, are shown to the user then.If the local cache in the acting server does not also have the corresponding data resource of this web page address; When then acting server whenever gets access to primary resource or the child resource in all corresponding data resources of this web page address; Can this primary resource or child resource be pushed to client at once; Client can receive all corresponding data resources of this web page address in batches, and is cached to successively in the buffer memory of client terminal local, is shown to the user then.
In sum; The data capture method that the embodiment of the invention three provides walks abreast and grasps the data resource of webpage through utilizing acting server to set up many data channel; Active push is given the scheme of client then, has solved client when obtaining network data, the problem that waste flow and response time are long; Reached client and only need initiate data acquisition request one time, all data resources that just can obtain webpage fast are to be shown to user's effect.
Please refer to Fig. 5, it shows the block diagram of the acting server that the embodiment of the invention four provides.This acting server comprises request receiver module 520, resource grabbing module 540 and resource supplying module 560.
Preferably, acting server can also comprise: signal receiving module 512, passage set up module 514 and passage is kept module 516, and be as shown in Figure 6.Wherein, signal receiving module 512 is used to receive the lasting passage foundation request from client.Passage is set up module 514 and is used for setting up lasting passage with client, so that receive at least one data acquisition request from client.Passage is kept module 516 and is used for keeping lasting passage through heartbeat signal and client.
Wherein, resource grabbing module 540 can specifically comprise: parallel placement unit 542, script judging unit 544 and script executing unit 546, and as shown in Figure 7.Wherein, Parallel placement unit 542 is used for setting up at least two data channels pairing data resource of extracting web page address that walk abreast with web page server; Data resource comprises primary resource and child resource, and every data channel is used to grasp a primary resource or a child resource; Script judging unit 544 is used to judge whether the primary resource or the child resource that get access to comprise script data; If script executing unit 546 is used for script judging unit 544 and determines primary resource or the child resource that parallel placement unit 542 gets access to and comprise script data, then carry out script data in advance.
Preferably, acting server can also comprise: data cache module 550, and as shown in Figure 8.Data cache module 550 is used for pairing primary resource of the web page address that gets access to or child resource are cached to local cache.At this moment, resource supplying module 560 can specifically comprise: resource monitoring unit 562 and resource supplying unit 564.Wherein, resource monitoring unit 562 is used for monitoring local cache and whether is cached to pairing primary resource of web page address or child resource.If resource supplying unit 564 is used for resource judgment unit 562 and determines local cache and be cached to pairing primary resource of web page address or child resource, then push primary resource or child resource to client.
In sum; The acting server that the embodiment of the invention four provides walks abreast and grasps the data resource of webpage through utilizing acting server to set up many data channel; Active push is given the scheme of client then, has solved client when obtaining network data, the problem that waste flow and response time are long; Reached client and only need initiate data acquisition request one time, all data resources that just can obtain webpage fast are to be shown to user's effect.Simultaneously; The acting server that the embodiment of the invention four provides is also through increasing lasting passage mechanism, the preparatory execution mechanism of script data and adopting the mechanism of Cache cache web pages data, reached the effect that acting server can be pushed to web data client quickly.
Please refer to Fig. 9, it shows the block diagram of the client that the embodiment of the invention five provides.This client can comprise web-page requests receiver module 920, obtain request sending module 940 and data resource receiver module 960.
Web-page requests receiver module 920 is used to receive the web page access request from the user, and this web page access request comprises web page address.
Obtaining request sending module 940 is used for sending data acquisition request according to web page address to acting server.
Data resource receiver module 960 is used to receive the corresponding data resource of web page address that is pushed by acting server.
Further, client can also comprise: set up request sending module 912, lasting passage sets up module 914 and lasting passage is kept module 916, and is shown in figure 10.Wherein, setting up request sending module 912 is used for sending lasting passage foundation request to acting server; Lasting passage is set up module 914 and is used for setting up lasting passage with acting server, so that utilize lasting passage to send at least one data acquisition request to acting server; Lasting passage is kept module 916 and is used for keeping lasting passage through heartbeat signal and acting server.
At this moment, obtaining request sending module 940 can also specifically comprise: first passage judging unit 942, second channel judging unit 944 and obtain request transmitting unit 946.Wherein, first passage judging unit 942 is used to judge whether existed lasting passage; If second channel judging unit 944 is used for existing lasting passage, then continue to judge whether lasting passage is normal; To be used for lasting passage normal if obtain request transmitting unit 946, then utilizes lasting passage to send data acquisition request to acting server, and data acquisition request comprises web page address.
In sum; The client that the embodiment of the invention five provides walks abreast and grasps the data resource of webpage through utilizing acting server to set up many data channel; Active push is given the scheme of client then, has solved client when obtaining network data, the problem that waste flow and response time are long; Reached client and only need initiate data acquisition request one time, all data resources that just can obtain webpage fast are to be shown to user's effect.
Please refer to Figure 11, it shows the structural representation of the data-acquisition system that the embodiment of the invention six provides.This data-acquisition system can comprise acting server 11a and client 11b.
Wherein, acting server 11a can be the acting server that provides among the embodiment four.Client 11b can be the acting server that provides among the embodiment five.
Need to prove: the acting server that the foregoing description provides, client and data-acquisition system are when obtaining web data; Only the division with above-mentioned each functional module is illustrated; In the practical application; Can as required above-mentioned functions be distributed by the different functional completion, the internal structure that is about to device is divided into different functional, to accomplish all or part of function of above description.In addition, the acting server that the foregoing description provides, client and data-acquisition system and data capture method embodiment belong to same design, and its concrete implementation procedure sees method embodiment for details, repeats no more here.
The all or part of step that one of ordinary skill in the art will appreciate that realization the foregoing description can be accomplished through hardware; Also can instruct relevant hardware to accomplish through program; Described program can be stored in a kind of computer-readable recording medium; The above-mentioned storage medium of mentioning can be a read-only memory, disk or CD etc.
The above is merely preferred embodiment of the present invention, and is in order to restriction the present invention, not all within spirit of the present invention and principle, any modification of being done, is equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (17)
1. a data capture method is characterized in that, said method comprises:
Reception is from the data acquisition request of client, and said data acquisition request comprises web page address;
Set up at least two data channels pairing data resource of the said web page address of extracting that walk abreast with web page server;
The said data resource that grabs is pushed to said client.
2. data capture method according to claim 1 is characterized in that, said reception is from the data acquisition request of client, and said data acquisition request comprises and also comprising before the web page address:
Reception is set up request from the lasting passage of client;
Set up lasting passage with said client, so that receive at least one data acquisition request from said client;
Keep said lasting passage through heartbeat signal and said client.
3. data capture method according to claim 1 is characterized in that, said walking abreast with at least two data channels of web page server foundation grasped the pairing data resource of said web page address, specifically comprises:
Set up at least two data channels with web page server and walk abreast and grasp the pairing data resource of said web page address, said data resource comprises primary resource and child resource, and every data channel is used to grasp perhaps child resource of a primary resource;
Judge whether the primary resource or the child resource that get access to comprise script data;
If primary resource that gets access to or child resource comprise script data, then carry out said script data in advance.
4. according to claim 1 or 3 described data capture methods, it is characterized in that said and at least two data channels of web page server foundation walk abreast and grasp after the pairing data resource of said web page address, also comprise:
The pairing primary resource of said web page address or the child resource that get access to are cached to local cache.
5. data capture method according to claim 4 is characterized in that, said all data resources that will grab are pushed to said client, specifically comprise:
Monitor and whether be cached to pairing primary resource of said web page address or child resource in the said local cache;
If be cached to pairing primary resource of said web page address or child resource in the said local cache, then push said primary resource or child resource to said client.
6. a data capture method is characterized in that, said method comprises:
Reception is from user's web page access request, and said web page access request comprises web page address;
Send data acquisition request according to said web page address to acting server;
The data resource corresponding that reception is pushed by said acting server with said web page address.
7. data capture method according to claim 6 is characterized in that, before the web page access request of said reception from the user, also comprises:
Send lasting passage to said acting server and set up request;
Set up lasting passage with said acting server, so that utilize said lasting passage to send at least one data acquisition request to said acting server;
Keep said lasting passage through heartbeat signal and said acting server.
8. data capture method according to claim 7 is characterized in that, saidly sends data acquisition request according to said web page address to acting server, specifically comprises:
Judge whether to exist lasting passage;
If there has been lasting passage, then continue to judge whether said lasting passage is normal;
If said lasting passage is normal, then utilize said lasting passage to send data acquisition request to acting server, comprise said web page address in the said data acquisition request.
9. an acting server is characterized in that, said acting server comprises:
The request receiver module is used to receive the data acquisition request from client, and said data acquisition request comprises web page address;
The resource grabbing module is used for setting up at least two data channels pairing data resource of the said web page address of extracting that walk abreast with web page server;
The resource supplying module is used for the said data resource that grabs is pushed to said client.
10. acting server according to claim 9 is characterized in that, said acting server also comprises:
Signal receiving module, passage set up module and passage is kept module;
Said signal receiving module, the lasting passage that is used to receive from client is set up request;
Said passage is set up module, is used for setting up lasting passage with said client, so that receive at least one data acquisition request from said client;
Said passage is kept module, is used for keeping said lasting passage through heartbeat signal and said client.
11. acting server according to claim 9 is characterized in that, said resource grabbing module specifically comprises:
Parallel placement unit, script judging unit and script executing unit;
Said parallel placement unit; Be used for setting up at least two data channels pairing data resource of the said web page address of extracting that walk abreast with web page server; Said data resource comprises primary resource and child resource, and every data channel is used to grasp a primary resource or a child resource;
Said script judging unit is used to judge whether the primary resource or the child resource that get access to comprise script data;
Said script executing unit if primary resource that is used for getting access to or child resource comprise script data, is then carried out said script data in advance.
12., it is characterized in that said acting server also comprises according to claim 9 or 11 described acting servers:
Data cache module;
Said data cache module is used for the pairing primary resource of said web page address or the child resource that get access to are cached to local cache.
13. acting server according to claim 12 is characterized in that, said resource supplying module specifically comprises:
Resource monitoring unit and resource supplying unit;
Said monitoring judging unit is used for monitoring said local cache and whether is cached to pairing primary resource of said web page address or child resource;
Said resource supplying unit is cached to pairing primary resource of said web page address or child resource if be used for said local cache, then pushes said primary resource or child resource to said client.
14. a client is characterized in that, said client comprises:
The web-page requests receiver module is used to receive the web page access request from the user, and said web page access request comprises web page address;
Obtain request sending module, be used for sending data acquisition request to acting server according to said web page address;
The data resource receiver module is used to receive the data resource corresponding with said web page address that is pushed by said acting server.
15. client according to claim 14 is characterized in that, said client also comprises:
Set up request sending module, lasting passage sets up module and lasting passage is kept module;
The said request sending module of setting up is used for sending lasting passage to said acting server and sets up request;
Said lasting passage is set up module, is used for setting up lasting passage with said acting server, so that utilize said lasting passage to send at least one data acquisition request to said acting server;
Said lasting passage is kept module, is used for keeping said lasting passage through heartbeat signal and said acting server.
16., it is characterized in that the said request sending module of obtaining specifically comprises according to claim 14 or 16 described clients:
First passage judging unit, second channel judging unit and obtain request transmitting unit;
Said first passage judging unit is used to judge whether existed lasting passage;
Said second channel judging unit if be used for existing lasting passage, then continues to judge whether said lasting passage is normal;
The said request transmitting unit of obtaining if it is normal to be used for said lasting passage, then utilizes said lasting passage to send data acquisition request to acting server, and said data acquisition request comprises said web page address.
17. a data-acquisition system is characterized in that, it comprises like the arbitrary described acting server of claim 9 to 13 with like the arbitrary described client of claim 14 to 16.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210133394.9A CN102710748B (en) | 2012-05-02 | 2012-05-02 | Data capture method, system and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210133394.9A CN102710748B (en) | 2012-05-02 | 2012-05-02 | Data capture method, system and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102710748A true CN102710748A (en) | 2012-10-03 |
| CN102710748B CN102710748B (en) | 2016-01-27 |
Family
ID=46903294
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210133394.9A Active CN102710748B (en) | 2012-05-02 | 2012-05-02 | Data capture method, system and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102710748B (en) |
Cited By (31)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902803A (en) * | 2012-10-12 | 2013-01-30 | 北京奇虎科技有限公司 | Webpage transcoding system |
| CN103258056A (en) * | 2013-05-31 | 2013-08-21 | 北京奇虎科技有限公司 | Method, server, client and system for handling style sheets |
| CN103281387A (en) * | 2013-05-31 | 2013-09-04 | 北京奇虎科技有限公司 | Method, server, client side and system for processing child resources in web page |
| CN103530390A (en) * | 2013-10-22 | 2014-01-22 | 北京奇虎科技有限公司 | Webpage crawling method and device |
| CN104113564A (en) * | 2013-04-17 | 2014-10-22 | 中国移动通信集团公司 | Hyper text transfer protocol connection multiplexing method, device, system, and terminal |
| CN104239302A (en) * | 2013-06-07 | 2014-12-24 | 腾讯科技（深圳）有限公司 | Page content acquiring method and device, application device, and mobile terminal |
| CN104602034A (en) * | 2014-12-31 | 2015-05-06 | 北京奇艺世纪科技有限公司 | Mobile webpage playing method and system |
| CN104615771A (en) * | 2015-02-13 | 2015-05-13 | 广州华多网络科技有限公司 | Webpage data acquiring method and device |
| CN104798091A (en) * | 2012-11-21 | 2015-07-22 | 奥迪股份公司 | Motor vehicle comprising an operating device for operating an internet portal of a social network service |
| CN104850560A (en) * | 2014-02-18 | 2015-08-19 | 北京京东尚科信息技术有限公司 | Method and system for loading business data in webpage in real time |
| CN104866499A (en) * | 2014-02-24 | 2015-08-26 | 腾讯科技（深圳）有限公司 | Webpage loading method and device |
| CN105610836A (en) * | 2015-12-31 | 2016-05-25 | 浙江省公众信息产业有限公司 | Data transmission method and system |
| CN105893451A (en) * | 2015-12-31 | 2016-08-24 | 乐视移动智能信息技术（北京）有限公司 | Resource acquisition method and device |
| CN106095506A (en) * | 2016-06-14 | 2016-11-09 | 乐视控股（北京）有限公司 | A kind of page loading method and device |
| CN106302572A (en) * | 2015-05-15 | 2017-01-04 | Tcl集团股份有限公司 | A kind of multichannel communication method between server and system thereof |
| CN106293794A (en) * | 2015-06-05 | 2017-01-04 | 阿里巴巴集团控股有限公司 | Load the methods, devices and systems of the page |
| CN106549989A (en) * | 2015-09-17 | 2017-03-29 | 腾讯科技（深圳）有限公司 | A kind of data transmission method and its system, user terminal, application server |
| CN106612261A (en) * | 2015-10-26 | 2017-05-03 | 北京国双科技有限公司 | Website data obtaining method, devices and system |
| CN106776947A (en) * | 2016-12-02 | 2017-05-31 | 乐视控股（北京）有限公司 | Resource acquiring method, device and terminal |
| CN107025234A (en) * | 2016-02-01 | 2017-08-08 | 中国移动通信集团公司 | A kind of information-pushing method and caching server |
| CN107230130A (en) * | 2017-04-28 | 2017-10-03 | 杭州集盒网络技术有限公司 | A kind of commercial circle data display method |
| CN107656934A (en) * | 2016-07-25 | 2018-02-02 | 腾讯科技（深圳）有限公司 | One kind preloads method, apparatus, equipment |
| CN107798061A (en) * | 2017-09-18 | 2018-03-13 | 维沃移动通信有限公司 | A kind of webpage loading method and mobile terminal |
| CN107798008A (en) * | 2016-08-31 | 2018-03-13 | 腾讯科技（深圳）有限公司 | Content push system, method and device |
| CN110134896A (en) * | 2019-05-17 | 2019-08-16 | 山东渤聚通云计算有限公司 | A kind of the monitoring process and intelligent buffer method of proxy server |
| CN110569462A (en) * | 2019-07-31 | 2019-12-13 | 深圳市富途网络科技有限公司 | Network request data management method, data management equipment and readable medium |
| CN110674379A (en) * | 2019-09-26 | 2020-01-10 | 凡普数字技术有限公司 | Method, device and storage medium for acquiring information |
| CN110737447A (en) * | 2018-07-18 | 2020-01-31 | 阿里巴巴集团控股有限公司 | Application updating method and device |
| CN111131019A (en) * | 2019-12-12 | 2020-05-08 | 华为技术有限公司 | Multiplexing method and terminal for multiple HTTP channels |
| CN111191158A (en) * | 2019-12-27 | 2020-05-22 | 北京达佳互联信息技术有限公司 | Webpage resource caching method and device, computer equipment and storage medium |
| CN112968939A (en) * | 2017-01-05 | 2021-06-15 | 创新先进技术有限公司 | Mobile terminal resource loading method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080119177A1 (en) * | 2006-09-15 | 2008-05-22 | Speedus Corp. | Metadata Content Delivery System for Wireless Networks |
| CN101635718A (en) * | 2009-08-26 | 2010-01-27 | 中兴通讯股份有限公司 | Network crawler system and method for acquiring resource as well as network resource gripping device |
| CN101651707A (en) * | 2009-09-22 | 2010-02-17 | 西安交通大学 | Method for automatically acquiring user behavior log of network |
| CN102143187A (en) * | 2011-04-07 | 2011-08-03 | 北京星网锐捷网络技术有限公司 | Method and system for terminal equipment to access network as well as network access proxy device |
| CN102184231A (en) * | 2011-05-12 | 2011-09-14 | 广州市动景计算机科技有限公司 | Method and device for acquiring page resources |
-
2012
- 2012-05-02 CN CN201210133394.9A patent/CN102710748B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080119177A1 (en) * | 2006-09-15 | 2008-05-22 | Speedus Corp. | Metadata Content Delivery System for Wireless Networks |
| CN101635718A (en) * | 2009-08-26 | 2010-01-27 | 中兴通讯股份有限公司 | Network crawler system and method for acquiring resource as well as network resource gripping device |
| CN101651707A (en) * | 2009-09-22 | 2010-02-17 | 西安交通大学 | Method for automatically acquiring user behavior log of network |
| CN102143187A (en) * | 2011-04-07 | 2011-08-03 | 北京星网锐捷网络技术有限公司 | Method and system for terminal equipment to access network as well as network access proxy device |
| CN102184231A (en) * | 2011-05-12 | 2011-09-14 | 广州市动景计算机科技有限公司 | Method and device for acquiring page resources |
Cited By (47)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902803A (en) * | 2012-10-12 | 2013-01-30 | 北京奇虎科技有限公司 | Webpage transcoding system |
| CN102902803B (en) * | 2012-10-12 | 2016-01-13 | 北京奇虎科技有限公司 | Webpage trans-coding system |
| US10158597B2 (en) | 2012-11-21 | 2018-12-18 | Audi Ag | Motor vehicle comprising an operating device for operating an internet portal of a social network service |
| CN104798091B (en) * | 2012-11-21 | 2018-02-02 | 奥迪股份公司 | The motor vehicle of operation device with the portal website for operating public network service |
| CN104798091A (en) * | 2012-11-21 | 2015-07-22 | 奥迪股份公司 | Motor vehicle comprising an operating device for operating an internet portal of a social network service |
| CN104113564A (en) * | 2013-04-17 | 2014-10-22 | 中国移动通信集团公司 | Hyper text transfer protocol connection multiplexing method, device, system, and terminal |
| CN103258056A (en) * | 2013-05-31 | 2013-08-21 | 北京奇虎科技有限公司 | Method, server, client and system for handling style sheets |
| CN103281387A (en) * | 2013-05-31 | 2013-09-04 | 北京奇虎科技有限公司 | Method, server, client side and system for processing child resources in web page |
| CN103281387B (en) * | 2013-05-31 | 2017-04-12 | 北京奇虎科技有限公司 | Method, server, client side and system for processing child resources in web page |
| CN103258056B (en) * | 2013-05-31 | 2016-06-29 | 北京奇虎科技有限公司 | Process the method for style design table, server, client and system |
| CN104239302A (en) * | 2013-06-07 | 2014-12-24 | 腾讯科技（深圳）有限公司 | Page content acquiring method and device, application device, and mobile terminal |
| CN104239302B (en) * | 2013-06-07 | 2017-10-03 | 腾讯科技（深圳）有限公司 | Content of pages acquisition methods, device and application apparatus and mobile terminal |
| CN103530390A (en) * | 2013-10-22 | 2014-01-22 | 北京奇虎科技有限公司 | Webpage crawling method and device |
| CN103530390B (en) * | 2013-10-22 | 2018-09-04 | 北京奇虎科技有限公司 | The method and apparatus of webpage capture |
| CN104850560A (en) * | 2014-02-18 | 2015-08-19 | 北京京东尚科信息技术有限公司 | Method and system for loading business data in webpage in real time |
| CN104866499A (en) * | 2014-02-24 | 2015-08-26 | 腾讯科技（深圳）有限公司 | Webpage loading method and device |
| CN104602034B (en) * | 2014-12-31 | 2019-05-31 | 北京奇艺世纪科技有限公司 | A kind of playback method and system of mobile webpage |
| CN104602034A (en) * | 2014-12-31 | 2015-05-06 | 北京奇艺世纪科技有限公司 | Mobile webpage playing method and system |
| CN104615771B (en) * | 2015-02-13 | 2018-12-21 | 广州华多网络科技有限公司 | A kind of method and device obtaining web data |
| CN104615771A (en) * | 2015-02-13 | 2015-05-13 | 广州华多网络科技有限公司 | Webpage data acquiring method and device |
| CN106302572A (en) * | 2015-05-15 | 2017-01-04 | Tcl集团股份有限公司 | A kind of multichannel communication method between server and system thereof |
| CN106302572B (en) * | 2015-05-15 | 2020-09-22 | Tcl科技集团股份有限公司 | Multi-channel communication method and system between servers |
| CN106293794A (en) * | 2015-06-05 | 2017-01-04 | 阿里巴巴集团控股有限公司 | Load the methods, devices and systems of the page |
| CN106549989B (en) * | 2015-09-17 | 2020-02-18 | 腾讯科技（深圳）有限公司 | Data transmission method and system, user terminal and application server |
| CN106549989A (en) * | 2015-09-17 | 2017-03-29 | 腾讯科技（深圳）有限公司 | A kind of data transmission method and its system, user terminal, application server |
| CN106612261A (en) * | 2015-10-26 | 2017-05-03 | 北京国双科技有限公司 | Website data obtaining method, devices and system |
| CN105893451A (en) * | 2015-12-31 | 2016-08-24 | 乐视移动智能信息技术（北京）有限公司 | Resource acquisition method and device |
| CN105610836A (en) * | 2015-12-31 | 2016-05-25 | 浙江省公众信息产业有限公司 | Data transmission method and system |
| CN107025234A (en) * | 2016-02-01 | 2017-08-08 | 中国移动通信集团公司 | A kind of information-pushing method and caching server |
| CN107025234B (en) * | 2016-02-01 | 2020-11-06 | 中国移动通信集团公司 | Information pushing method and cache server |
| CN106095506A (en) * | 2016-06-14 | 2016-11-09 | 乐视控股（北京）有限公司 | A kind of page loading method and device |
| CN107656934A (en) * | 2016-07-25 | 2018-02-02 | 腾讯科技（深圳）有限公司 | One kind preloads method, apparatus, equipment |
| CN107798008A (en) * | 2016-08-31 | 2018-03-13 | 腾讯科技（深圳）有限公司 | Content push system, method and device |
| CN107798008B (en) * | 2016-08-31 | 2020-06-26 | 腾讯科技（深圳）有限公司 | Content pushing system, method and device |
| CN106776947A (en) * | 2016-12-02 | 2017-05-31 | 乐视控股（北京）有限公司 | Resource acquiring method, device and terminal |
| CN112968939A (en) * | 2017-01-05 | 2021-06-15 | 创新先进技术有限公司 | Mobile terminal resource loading method and device |
| CN112968939B (en) * | 2017-01-05 | 2023-12-05 | 创新先进技术有限公司 | Mobile terminal resource loading method and device |
| CN107230130A (en) * | 2017-04-28 | 2017-10-03 | 杭州集盒网络技术有限公司 | A kind of commercial circle data display method |
| CN107798061A (en) * | 2017-09-18 | 2018-03-13 | 维沃移动通信有限公司 | A kind of webpage loading method and mobile terminal |
| CN110737447A (en) * | 2018-07-18 | 2020-01-31 | 阿里巴巴集团控股有限公司 | Application updating method and device |
| CN110737447B (en) * | 2018-07-18 | 2023-11-14 | 阿里巴巴集团控股有限公司 | Application updating method and device |
| CN110134896A (en) * | 2019-05-17 | 2019-08-16 | 山东渤聚通云计算有限公司 | A kind of the monitoring process and intelligent buffer method of proxy server |
| CN110569462A (en) * | 2019-07-31 | 2019-12-13 | 深圳市富途网络科技有限公司 | Network request data management method, data management equipment and readable medium |
| CN110674379A (en) * | 2019-09-26 | 2020-01-10 | 凡普数字技术有限公司 | Method, device and storage medium for acquiring information |
| CN111131019A (en) * | 2019-12-12 | 2020-05-08 | 华为技术有限公司 | Multiplexing method and terminal for multiple HTTP channels |
| CN111131019B (en) * | 2019-12-12 | 2021-06-22 | 华为技术有限公司 | Multiplexing method and terminal for multiple HTTP channels |
| CN111191158A (en) * | 2019-12-27 | 2020-05-22 | 北京达佳互联信息技术有限公司 | Webpage resource caching method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102710748B (en) | 2016-01-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102710748A (en) | Data acquisition method, system and equipment | |
| US10411956B2 (en) | Enabling planned upgrade/downgrade of network devices without impacting network sessions | |
| CN102594877B (en) | In conjunction with the method, the system that are redirected download request and the service of agency service accelerating network | |
| US10015204B2 (en) | Method, terminal, and server for restoring transmission of session content | |
| US20110145687A1 (en) | Method and System For enabling Access To Data Files Unsupported by A Computing Device | |
| EP2888862B1 (en) | Information transmission method, apparatus, and system, terminal, and server | |
| WO2004104867A2 (en) | System for doing service location management taking into account the node and network characteristics | |
| US20140280768A1 (en) | Migration of network connection under mobility | |
| WO2015106692A1 (en) | Webpage pushing method, client, server, and system | |
| CN103401946B (en) | HTTP uploads accelerated method and system | |
| CN104519139B (en) | Caching method, caching edges server, caching Core server and caching system System | |
| CN102394880A (en) | Method and device for processing jump response in content delivery network | |
| US20120197971A1 (en) | Enhanced web-browsing system and method | |
| CN103905496A (en) | Picture downloading method and device | |
| EP1625725B1 (en) | Method for adapting service location placement based on recent data received from service nodes and actions of the service location manger | |
| CN106330607A (en) | Method and system for estimating TCP connection application quality based on one-way monitoring log | |
| JP2009110041A (en) | Web screen sharing system, its terminal for sharing, and its sharing program | |
| CN105554116A (en) | Remote interconnection method and system for UPnP (Universal Plug and Play) equipment | |
| CN104468642B (en) | Browser rs cache resource prediction method and system, reverse proxy device and browser | |
| CN113542335A (en) | Information sharing method and device, electronic equipment and storage medium | |
| Fleming et al. | Improving the performance of the world wide web over wireless networks | |
| CN108471375A (en) | A kind of message treatment method, device and terminal | |
| RU120259U1 (en) | SYSTEM OF MULTIMEDIA INTERACTIVE INTERACTION | |
| Soe et al. | AgentUDM: A mobile agent based support infrastructure for ubiquitous data mining | |
| CN102143196A (en) | Client communication method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |