CN101997915B - Deep packet detection device, webpage data processing method, and webpage data acquisition method and system - Google Patents
Deep packet detection device, webpage data processing method, and webpage data acquisition method and system Download PDFInfo
- Publication number
- CN101997915B CN101997915B CN201010532086.4A CN201010532086A CN101997915B CN 101997915 B CN101997915 B CN 101997915B CN 201010532086 A CN201010532086 A CN 201010532086A CN 101997915 B CN101997915 B CN 101997915B
- Authority
- CN
- China
- Prior art keywords
- protocol message
- http protocol
- data
- webpage
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a webpage data processing method, a webpage data acquisition method, a deep packet detection device and a webpage data acquisition system. The webpage data acquisition method comprises the following steps: selectively capturing hyper text transport protocol (HTTP) messages of a flow webpage server according to a webpage address information base; analyzing the content of the captured HTTP messages; extracting the content of tag fields in the HTTP messages; and selectively acquiring the data in the captured HTTP messages according to the content of the tag fields. According to the invention, the deep packet detection technique with the webpage data acquisition technique are combined, the acquisition and analysis efficiency of the webpage data is improved, the cost for acquiring and analyzing mass data is reduced, and simultaneously the webpage data more accurately can be acquired because the tag fields are adopted.
Description
Technical field
The present invention relates to Internet technical field, especially, relate to a kind of deep packet detection device, web data processing method, webpage data acquiring method and collecting webpage data system.
Background technology
Along with the fast development of WEB technology and WEB application, also more and more extensive to the application of the centralized monitor in various WEB applications website, the particularly platform such as electronic channel, ecommerce, user data collection and statistical analysis.But, because the user data of the platforms such as the huge electronic channel of customer volume and ecommerce is magnanimity, therefore, in real work, need mass data is optionally gathered.
Yet, existing webpage is not considered data collection problems at the beginning of design, and the problem such as existing webpage ubiquity page address and image data are mixed and disorderly, accuracy is not high, therefore, based on existing webpage, be difficult to carry out efficient and data acquisition exactly.
Summary of the invention
The technical problem that the present invention will solve is to provide a kind of deep packet detection device, web data processing method, webpage data acquiring method and collecting webpage data system, can be efficiently and exactly the data of webpage are gathered.
According to an aspect of the present invention, propose a kind of web data processing method, comprised the data acquisition scope according to the http protocol message of each webpage of data acquisition Location of requirement; Add label field in the http protocol message of each webpage, the data acquisition scope of the http protocol message of the content representation webpage of label field.
The web data processing method embodiment according to the present invention, label field is arranged in the header fields of http protocol message of each webpage.
Another embodiment of web data processing method according to the present invention, the data acquisition scope of the http protocol message of webpage comprises total data, the partial data in extraction http protocol message extracted in the http protocol message and does not extract any data in the http protocol message.
According to a further aspect in the invention, also proposed a kind of webpage data acquiring method, comprised according to the web page address information bank and optionally capture the http protocol message that flows to web page server; The content of the http protocol message that parsing grabs; Extract the content of the label field in the http protocol message; According to the content of label field, the data in the http protocol message grabbed are carried out to the selectivity collection.
Webpage data acquiring method embodiment according to the present invention forms the http protocol message that flows to web page server by following step: according to the data acquisition scope of the http protocol message of each webpage of data acquisition Location of requirement; Add label field in the http protocol message of each webpage, form the http protocol message that flows to web page server, wherein, the data acquisition scope of the http protocol message of the content representation webpage of label field.
Another embodiment of webpage data acquiring method according to the present invention, label field is arranged in the header fields of http protocol message of each webpage.
The another embodiment of webpage data acquiring method according to the present invention, the data acquisition scope of the http protocol message of webpage comprises total data, the partial data in extraction http protocol message extracted in the http protocol message and does not extract any data in the http protocol message.
According to another aspect of the invention, also propose a kind of deep packet detection device, comprised address screening module, for optionally capture the http protocol message that flows to web page server according to the web page address information bank; The packet parsing module, be connected with address screening module, the content of the http protocol message grabbed for parsing; The label substance extraction module, be connected with the packet parsing module, for the content of the label field of extracting the http protocol message, wherein, the data acquisition scope of the http protocol message of the content representation webpage of label field; Data acquisition module, be connected with the label substance extraction module, for the content according to label field, the data of the http protocol message that grabs carried out to the selectivity collection.
Deep packet detection device embodiment according to the present invention, label field is arranged in the header fields of the http protocol message that flows to web page server.
Another embodiment of deep packet detection device according to the present invention, the data acquisition scope of the http protocol message of webpage comprises total data, the partial data in extraction http protocol message extracted in the http protocol message and does not extract any data in the http protocol message.
In accordance with a further aspect of the present invention, a kind of collecting webpage data system has also been proposed, comprise deep packet detection device and web data processing unit in above-described embodiment, wherein, the web data processing unit comprises the acquisition range determination module, for the data acquisition scope of the http protocol message according to each webpage of data acquisition Location of requirement; Data processing module, be connected with the acquisition range determination module, for the http protocol message at each webpage, add label field, form the http protocol message that flows to web page server, wherein, the data acquisition scope of the http protocol message of the content representation webpage of label field.
Deep packet detection device provided by the invention, web data processing method, webpage data acquiring method and collecting webpage data system, can be by deep-packet detection (Deep Packet Inspection, DPI) technology combines with the collecting webpage data technology, promote the collecting efficiency to web data, reduced the cost that mass data is gathered and analyzes.Simultaneously, owing to adopting label field, thus can determine more accurately the data acquisition scope of webpage, thus the accuracy of data acquisition improved.
The accompanying drawing explanation
Accompanying drawing described herein is used to provide a further understanding of the present invention, forms the application's a part.In the accompanying drawings:
Fig. 1 is the schematic flow sheet of an embodiment of web data processing method of the present invention.
Fig. 2 is the schematic flow sheet of an embodiment of webpage data acquiring method of the present invention.
Fig. 3 is the schematic flow sheet of the another embodiment of webpage data acquiring method of the present invention.
Fig. 4 is the structural representation of an embodiment of deep packet detection device of the present invention.
Fig. 5 is the structural representation of an embodiment of collecting webpage data system of the present invention.
Embodiment
With reference to the accompanying drawings the present invention is described more fully, exemplary embodiment of the present invention wherein is described.Exemplary embodiment of the present invention and explanation thereof are used for explaining the present invention, but do not form inappropriate limitation of the present invention.
Below to the description only actually of at least one exemplary embodiment, be illustrative, never as any restriction to the present invention and application or use.
The present invention combines DPI technology and WEB collecting webpage data technology, on the basis of having analyzed DPI selective data acquisition principle, in order to promote the efficiency of collection analysis, proposed to be convenient to web data processing method, webpage data acquiring method, deep packet detection device and the collecting webpage data system that DPI gathers.
When carrying out the collection of DPI selectivity, at first need to set up a storehouse, store page address to be collected, each request is first carried out address lookup according to this storehouse after server, if the address of webpage and Zhong page address, storehouse are complementary, extract the content of the page.
Fig. 1 is the schematic flow sheet of an embodiment of web data processing method of the present invention.
As shown in Figure 1, this embodiment can comprise the following steps:
S102, carry out the scope of data acquisition to the http protocol message of each webpage according to the data acquisition Location of requirement;
S104, add label field in the http protocol message of each webpage, the content representation of this label field is carried out the scope of data acquisition to the http protocol message of webpage, wherein, this label field can be arranged in any position of http protocol message, preferably, label field can be arranged in the header fields of http protocol message of each webpage.
In addition, the scope that the http protocol message of webpage is carried out to data acquisition can comprise that the total data extracted in the http protocol message (, comprise all data of heading to message trailer), extract the partial data (for example, IP address, user name, page address, access time, login classification and page parameter etc.) in the http protocol message and do not extract any data in the http protocol message.
This embodiment has considered the DPI technology when carrying out the web data processing, proposed to be convenient to the web data processing method that DPI gathers on the basis of having analyzed DPI selective data acquisition principle, this embodiment can significantly promote the collecting efficiency of web data, and improves the accuracy of data acquisition.
In another embodiment of web data processing method of the present invention, at first need standard (is for example carried out in page address, http: // 202.23.24.153/news/sports, represent the sports content in news), then, the WEB website and webpage are divided into to different levels, corresponding different data acquisition scopes, add label field again in the http protocol message of webpage, the content of this label field is corresponding to different data acquisition scopes.According to the RFC protocol specification, the header fields of http protocol message can need to embed the custom field content according to concrete application, therefore, can when the electronic channel webpage is realized, embed self-defining HTTP header fields information (, label field), the data acquisition demand for different, embed different self-defined informations to webpage, thereby realize the level classification to webpage, can get ready for data acquisition further.
Fig. 2 is the schematic flow sheet of an embodiment of webpage data acquiring method of the present invention.
As shown in Figure 2, this embodiment can comprise the following steps:
S202, optionally capture the http protocol message that flows to web page server according to the web page address information bank, wherein, the page address that can store webpage to be captured in this web page address information bank, meet the requirement of web page address information bank in the address of the page that flows to web page server (for example, this page address is stored in the web page address information bank) time, just crawled and carry out follow-up packet parsing and data extraction;
S204, the content of the http protocol message that parsing grabs;
S206, the content of the label field in extraction http protocol message;
S208, carry out the selectivity collection according to the content of label field to the data in the http protocol message grabbed.
Wherein, can form the http protocol message that flows to web page server by following step: according to the data acquisition Location of requirement, the http protocol message of each webpage is carried out the scope of data acquisition; Add label field in the http protocol message of each webpage, form the http protocol message that flows to web page server, wherein, the content representation of label field is carried out the scope of data acquisition to the http protocol message of webpage.
In an example, the scope that the http protocol message of webpage is carried out to data acquisition can comprise that the total data extracted in the http protocol message (, comprise all data of heading to message trailer), extract the partial data (for example, IP address, user name, page address, access time, login classification and page parameter etc.) in the http protocol message and do not extract any data in the http protocol message.
Alternatively, label field can be arranged in any position of the http protocol message of each webpage, preferably, label field can be arranged in the header fields of http protocol message of each webpage.
This embodiment, when carrying out data acquisition, at first screens webpage to be collected according to the web page address information bank, has reduced to a great extent the interference of mass data.Further, this embodiment also resolves the HTTP header fields content of captured webpage, extract self-defining header fields label substance, according to the content of label, take different data acquisitions to extract flow process, for example, can extract the http protocol message full content, extract the partial content of http protocol message or do not extract any content, thereby realize being with optionally data acquisition, reduce the pressure of mass data for technology and cost, improved efficiency and the accuracy of data acquisition simultaneously.
In another embodiment of webpage data acquiring method of the present invention, the http protocol message according to RFC protocol specification resolution flow to the WEB Website server, extract specifying information according to the content of the label field parsed in the relevant position of data content to be collected.Particularly, the DPI device, when processing http protocol, is resolved corresponding self-defined header fields content (that is, the content of label field), calls different data acquisition flows according to the definition of self-defined header fields content, to realize the extraction of web data.The self-defined content that http protocol head field embeds can be divided into label and two parts of content, self-defining header fields can arrange to start with " X-", for example, " X-type:0 " can mean to extract all the elements of http protocol message, and " X-type:1 " can mean only to extract the URL address.According to the level needs of data acquisition content, can define one or more self-defined header tag, give respectively different contents, different data are extracted in representative.
Fig. 3 is the schematic flow sheet of the another embodiment of webpage data acquiring method of the present invention.
As shown in Figure 3, this embodiment can comprise the following steps:
S302, build the DPI acquisition system, gathers website with target and carry out data image;
S304, set up the web page address information bank, wherein stored the address of webpage to be captured;
S306, set up selectivity and resolve the content depth information bank, wherein store the data acquisition that different customized labels are corresponding and resolved subprogram, for example, the part data acquisition that all bus data acquisition parsing subprogram that the full content of extraction http protocol message is used, the partial content that extracts the http protocol message are used is resolved subprogram etc.;
S308, carry out the selectivity crawl according to the web page address information bank to the page that flows to web page server;
S310, the data that storage captures;
S312, the content of the http protocol message of the page that parsing grabs, carry out the selectivity collection according to the content of the label field in the http protocol message to the data in the http protocol message grabbed;
S314, the data after the classification storing and resolving.
Fig. 4 is the structural representation of an embodiment of deep packet detection device of the present invention.
As shown in Figure 4, the deep packet detection device 10 of this embodiment can comprise:
Address screening module 11, for optionally capturing the http protocol message that flows to web page server according to the web page address information bank;
Packet parsing module 12, be connected with address screening module, the content of the http protocol message grabbed for parsing;
Label substance extraction module 13, with the packet parsing module, be connected, for extracting the content of label field of http protocol message, wherein, the data acquisition scope of the http protocol message of the content representation webpage of label field, alternatively, the data acquisition scope of the http protocol message of webpage can comprise total data, the partial data in extraction http protocol message extracted in the http protocol message and not extract any data in the http protocol message;
Data acquisition module 14, be connected with the label substance extraction module, for the content according to label field, the data of the http protocol message that grabs carried out to the selectivity collection.
Alternatively, label field can be arranged in the header fields of the http protocol message that flows to web page server.
This embodiment, when carrying out data acquisition, at first screens webpage to be collected according to web page address, has reduced to a great extent the processing to magnanimity.In addition, this embodiment also resolves the HTTP header fields content of captured webpage, extract self-defining header fields label substance, according to the content of label, take different data acquisitions to extract flow process, can extract the http protocol message full content, extract the partial content of http protocol message or do not extract any content etc., thereby realize being with optionally data acquisition, reduce the pressure of mass data for technology and cost, improved efficiency and the accuracy of data acquisition simultaneously.
Fig. 5 is the structural representation of an embodiment of collecting webpage data system of the present invention.
As shown in Figure 5, the collecting webpage data system of this embodiment can comprise deep packet detection device 10 and the web data processing unit 21 in previous embodiment, and wherein, web data processing unit 21 comprises:
Acquisition range determination module 211, for the data acquisition scope of the http protocol message according to each webpage of data acquisition Location of requirement;
Data processing module 212, be connected with the acquisition range determination module, for the http protocol message at each webpage, add label field, form the http protocol message that flows to web page server, wherein, the data acquisition scope of the http protocol message of the content representation webpage of label field.
Although by example, specific embodiments more of the present invention are had been described in detail, it should be appreciated by those skilled in the art, above example is only in order to describe, rather than in order to limit the scope of the invention.It should be appreciated by those skilled in the art, can without departing from the scope and spirit of the present invention, above embodiment be modified.Scope of the present invention is limited by claims.
Claims (9)
1. a web data processing method, is characterized in that, comprising:
Data acquisition scope according to the http protocol message of each webpage of data acquisition Location of requirement;
According to the level needs of data acquisition content, define one or more self-defined header tag, give respectively different contents, different data are extracted in representative;
Add self-defined header tag field in the http protocol message of each webpage, described self-defined header tag field comprises self-defined header tag and label substance, and described label substance means the data acquisition scope of the http protocol message of webpage.
2. method according to claim 1, is characterized in that, described self-defined header tag field is arranged in the header fields of http protocol message of each webpage.
3. method according to claim 1, it is characterized in that, the data acquisition scope of the http protocol message of described webpage comprises total data, the partial data in extraction http protocol message extracted in the http protocol message and does not extract any data in the http protocol message.
4. a webpage data acquiring method, is characterized in that, comprising:
Optionally capture the http protocol message that flows to web page server according to the web page address information bank;
The content of the http protocol message that parsing grabs;
Extract the content of the self-defined header tag field in described http protocol message;
According to the content of described self-defined header tag field, the data in the described http protocol message grabbed are carried out to the selectivity collection;
Wherein, form the described http protocol message that flows to web page server by following step:
Data acquisition scope according to the http protocol message of each webpage of data acquisition Location of requirement;
According to the level needs of data acquisition content, define one or more self-defined header tag, give respectively different contents, different data are extracted in representative;
Add self-defined header tag field in the http protocol message of each webpage, form the described http protocol message that flows to web page server, wherein, described self-defined header tag field comprises self-defined header tag and label substance, and described label substance means the data acquisition scope of the http protocol message of webpage.
5. method according to claim 4, is characterized in that, described self-defined header tag field is arranged in the header fields of http protocol message of each webpage.
6. method according to claim 4, it is characterized in that, the data acquisition scope of the http protocol message of described webpage comprises total data, the partial data in extraction http protocol message extracted in the http protocol message and does not extract any data in the http protocol message.
7. a collecting webpage data system, is characterized in that, comprises web data processing unit and deep packet detection device,
Wherein, described web data processing unit comprises:
The acquisition range determination module, for the data acquisition scope of the http protocol message according to each webpage of data acquisition Location of requirement;
Data processing module, with described acquisition range determination module, be connected, for the level needs according to the data acquisition content, define one or more self-defined header tag, give respectively different contents, different data are extracted in representative, add self-defined header tag field in the http protocol message of each webpage, formation flows to the http protocol message of web page server, wherein, described self-defined header tag field comprises self-defined header tag and label substance, and described label substance means the data acquisition scope of the http protocol message of webpage;
Described deep packet detection device comprises:
Address screening module, for optionally capturing the http protocol message that flows to web page server according to the web page address information bank;
The packet parsing module, be connected with described address screening module, the content of the http protocol message grabbed for parsing;
The label substance extraction module, be connected with described packet parsing module, for the content of the self-defined header tag field of extracting described http protocol message;
Data acquisition module, be connected with described label substance extraction module, for the content according to described self-defined header tag field, the data of the described http protocol message grabbed carried out to the selectivity collection.
8. system according to claim 7, is characterized in that, described self-defined header tag field is arranged in the header fields of the described http protocol message that flows to web page server.
9. system according to claim 7, it is characterized in that, the data acquisition scope of the http protocol message of described webpage comprises total data, the partial data in extraction http protocol message extracted in the http protocol message and does not extract any data in the http protocol message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010532086.4A CN101997915B (en) | 2010-10-29 | 2010-10-29 | Deep packet detection device, webpage data processing method, and webpage data acquisition method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010532086.4A CN101997915B (en) | 2010-10-29 | 2010-10-29 | Deep packet detection device, webpage data processing method, and webpage data acquisition method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101997915A CN101997915A (en) | 2011-03-30 |
| CN101997915B true CN101997915B (en) | 2014-01-08 |
Family
ID=43787485
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201010532086.4A Active CN101997915B (en) | 2010-10-29 | 2010-10-29 | Deep packet detection device, webpage data processing method, and webpage data acquisition method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101997915B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103888307B (en) * | 2012-12-20 | 2017-11-17 | 中国电信股份有限公司 | For optimizing method, user side board and the broad access network gate of deep-packet detection |
| CN103685298A (en) * | 2013-12-23 | 2014-03-26 | 上海交通大学无锡研究院 | Deep packet inspection based SSL (Secure Sockets Layer) man-in-the-middle attack discovering method |
| CN104486157A (en) * | 2014-12-16 | 2015-04-01 | 国家电网公司 | Information system performance detecting method based on deep packet analysis |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020091755A1 (en) * | 2000-06-30 | 2002-07-11 | Attila Narin | Supplemental request header for applications or devices using web browsers |
| CN1402156A (en) * | 2001-08-22 | 2003-03-12 | 威瑟科技股份有限公司 | Web site information extracting system and method |
| CN101094135A (en) * | 2006-06-23 | 2007-12-26 | 腾讯科技（深圳）有限公司 | Method and system for extracting information of content in Internet |
| CN101399749A (en) * | 2007-09-27 | 2009-04-01 | 华为技术有限公司 | Method, system and device for packet filtering |
| CN101667182A (en) * | 2008-09-05 | 2010-03-10 | 华为技术有限公司 | Method, system and device for performing secondary operation on web pages |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007101478A1 (en) * | 2006-03-09 | 2007-09-13 | Tecs Research And Development Limited | A method of monitoring online banner activity |
| CN101556609B (en) * | 2009-05-19 | 2010-12-08 | 杭州信杨通信技术有限公司 | Customer behavior analysis and service system based on web contents |
-
2010
- 2010-10-29 CN CN201010532086.4A patent/CN101997915B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020091755A1 (en) * | 2000-06-30 | 2002-07-11 | Attila Narin | Supplemental request header for applications or devices using web browsers |
| CN1402156A (en) * | 2001-08-22 | 2003-03-12 | 威瑟科技股份有限公司 | Web site information extracting system and method |
| CN101094135A (en) * | 2006-06-23 | 2007-12-26 | 腾讯科技（深圳）有限公司 | Method and system for extracting information of content in Internet |
| CN101399749A (en) * | 2007-09-27 | 2009-04-01 | 华为技术有限公司 | Method, system and device for packet filtering |
| CN101667182A (en) * | 2008-09-05 | 2010-03-10 | 华为技术有限公司 | Method, system and device for performing secondary operation on web pages |
Non-Patent Citations (1)
| Title |
|---|
| 中华人民共和国工业和信息化部.深度包检测设备技术要求.《中华人民共和国通信行业标准》.2009, * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101997915A (en) | 2011-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103810425B (en) | The detection method of malice network address and device | |
| CN101673266B (en) | Method for searching audio and video contents | |
| CN104063401B (en) | The method and apparatus that a kind of webpage pattern address merges | |
| CN102882703A (en) | Hyper text transfer protocol (HTTP)-analysis-based uniform resource locator (URL) automatically classifying and grading system and method | |
| CN103297270A (en) | Application type recognition method and network equipment | |
| CN102012894B (en) | Method and system for displaying documents by terminals | |
| CN103744985A (en) | Webpage adaption method and webpage adaption system | |
| CN107018001B (en) | Application fault positioning method and device | |
| CN106415593B (en) | QR code identification processing system | |
| CN103455600B (en) | A kind of video URL grasping means, device and server apparatus | |
| CN106469185A (en) | A kind of method carrying out data collection in website statistics | |
| CN101997915B (en) | Deep packet detection device, webpage data processing method, and webpage data acquisition method and system | |
| CN105374241A (en) | Children self-service reading system | |
| CN111740923A (en) | Method and device for generating application identification rule, electronic equipment and storage medium | |
| CN102541948A (en) | Method and device for extracting document structure | |
| CN101441629A (en) | Automatic acquiring method of non-structured web page information | |
| CN102868773A (en) | Method, device and system for detecting domain name system (DNS) black hole hijack | |
| CN104408185A (en) | Link click rate display method and device of webpage thermodynamic diagram | |
| CN103428249B (en) | A kind of Collecting and dealing method of HTTP request bag, system and server | |
| CN105162822A (en) | Website log data processing method and device | |
| CN102624878B (en) | Method and system for identifying P2P (peer-to-peer) protocol on basis of DNS (domain name server) protocol | |
| CN101937466B (en) | Webpage mailbox identification classifying method and system | |
| CN103152387B (en) | A kind of apparatus and method obtaining HTTP user behavior track | |
| CN113038153A (en) | Financial live broadcast violation detection method, device and equipment and readable storage medium | |
| CN103761257A (en) | Webpage handling method and system based on mobile browser |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |