WO2013026362A1

WO2013026362A1 - Method and system for monitoring network traffic

Info

Publication number: WO2013026362A1
Application number: PCT/CN2012/080039
Authority: WO
Inventors: 陈旭; 宋璇; 尹咸阳; 张仁卓
Original assignee: 华为技术有限公司
Priority date: 2011-08-22
Filing date: 2012-08-13
Publication date: 2013-02-28
Also published as: CN102957571A; CN102957571B

Abstract

The embodiments of the present invention relate to a method and system for monitoring network traffic. The method of the embodiments of the present invention includes: counting the number of times a uniform resource locator (URL) is requested within a pre-determined time period to determine a hotspot URL; actively capturing the resources corresponding to the hotspot URL; performing protocol regrouping on the actively captured resources corresponding to the hotspot URL; and analyzing the contents of the protocol regrouped data. Also provided is a system for monitoring network traffic. The embodiments of the present invention can effectively reduce the overhead of protocol regrouping and background content analysis, improve the efficiency of the entire system, and lower the system costs; additionally, for distributed P2P resources, intelligent regrouping can be performed, and P2P monitoring is also supported.

Description

METHOD AND SYSTEM FOR NETWORK TRAFFIC MONITORING [0001] This application claims priority to Chinese Patent Application, filed on Aug. 22, 2011, to the Chinese Patent Office, Application No. 201110241618.3, entitled "Method and System for Network Traffic Monitoring" The entire contents of which are incorporated herein by reference.

TECHNICAL FIELD The present invention relates to the field of the Internet and, more particularly, to monitoring data traffic in the Internet. BACKGROUND With the rapid development of the Internet, the Internet has become the main channel for information dissemination. However, the traditional Internet lacks supervision, and the information of malicious/yellow/personal attacks is rampant. Even cases of terrorist organizations using the Internet to train terrorists and organize terrorist attacks have emerged. In order to cope with this bad situation, the use of technical means to supervise the Internet has become the consensus of governments and operators. The traffic monitoring system came into being in this context. The traffic monitoring system collects traffic information, restores the original information, and performs intelligent analysis based on the characteristics in the original information, so as to timely discover the vulnerabilities in the network and strive to prevent the network attacks from happening.

The traditional traffic monitoring method is generally divided into three steps: traffic diversion, protocol reorganization, and background content analysis. These three steps are performed by the corresponding three functional units, as shown in Fig. 1, which are the drainage classification unit 102, the protocol reassembly unit 104, and the background content analysis unit 106, respectively. The traffic classification unit 102 distributes the received data packets to different protocol reorganization units 104 according to the protocol type, and the protocol reorganization unit 104 restores the application layer information (for example, from the Simple Mail Transfer Protocol (SMTP) message. Restore email (Email) information, restore Hypertext Markup Language (HTML) from the HyperText Transfer Protocol (HTTP) message, and then restore the restored application layer information along with the time label, chain The road information and the like are sent to the background content analysis unit 106 for analysis. The background content analysis unit 106 is composed of a server cluster, and performs hot spot statistics, Internet information analysis, and the like on the restored application layer information, thereby taking certain measures to suppress network attacks.

However, in the above prior art, the traffic classification unit passively drains, and the large-flow data is directly imported into the protocol reorganization unit and the background content analysis unit. As traffic increases, the processing costs of the server clusters in the protocol reorganization unit and the back-end content analysis unit increase dramatically.

Second, the background content analysis function is completed after the protocol is reorganized, thus repeating a large amount of the same content. Reorganization, resulting in a large demand for the performance of the protocol reorganization unit.

In addition, the above-mentioned conventional traffic monitoring method cannot obtain partial peer-to-peer (Peer-To-Peer, P2P) file fragmentation, and cannot implement smart link other P2P file fragmentation, thereby failing to process the pair. Network traffic monitoring. SUMMARY OF THE INVENTION In view of this, an aspect of the present invention provides a method for network traffic monitoring, the method comprising: requesting a number of times of a Uniform / Universal Resource Locator (URL) within a predetermined time Performing statistics to determine a hotspot URL; actively crawling resources corresponding to the hotspot URL; performing protocol reorganization on resources corresponding to the hotspot URL actively crawled; and performing content analysis on data reorganized by the protocol.

An aspect of the present invention provides a system for network traffic monitoring, the system comprising: a traffic classification unit for classifying traffic packets; and a hotspot statistics unit for using a uniform resource locator URL within a predetermined time The requested number of times is counted to determine a hotspot URL; the active crawling unit is configured to actively capture the resource corresponding to the hotspot URL; and the protocol reorganization unit is configured to perform protocol reorganization on the resource corresponding to the hot spot URL actively crawled And a background content analysis unit for performing content analysis on the data after the protocol reorganization.

The technical solution of the embodiment of the present invention counts the requested number of times of the uniform resource locator URL to determine the hotspot URL, and then actively captures the resource corresponding to the hotspot URL for protocol reorganization and content analysis. Therefore, the burden of protocol reorganization and background content analysis can be reduced. In addition, the technical solution of the embodiment of the present invention can actively capture P2P file fragments distributed throughout the distributed P2P resources to support monitoring of P2P traffic. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic diagram of a conventional network traffic monitoring system in the prior art.

FIG. 2 is a schematic diagram of an embodiment of a network traffic monitoring system according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of another embodiment of a network traffic monitoring system according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of an embodiment of a network traffic monitoring method according to an embodiment of the present invention.

The above summary of the invention, as well as the following detailed description of certain embodiments of the invention Certain embodiments are shown in the drawings for purposes of illustrating the invention. However, it should be understood that the invention is not to be construed as limited The detailed description set forth below with reference to the drawings is intended to illustrate various embodiments of the invention, The detailed description includes specific details in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without the specific details. In some instances, well-known structures and components are shown in the form of a block diagram in order to avoid a description of the invention.

2 depicts a network traffic monitoring system in accordance with an embodiment of the present invention. The system includes: a drainage classification unit 202, a hotspot statistics unit 204, an active crawling unit 206, a protocol reorganization unit 208, and a background content analysis unit 210. The traffic classification unit 202 is configured to perform traffic classification on the data packet. The hotspot statistics unit 204 is configured to perform statistics on the requested number of times of the uniform resource locator URL to determine the hotspot URL within a predetermined time. The active crawling unit 206, For retrieving the resource corresponding to the hotspot URL, the protocol reorganization unit 208 is configured to perform protocol reorganization on the resource corresponding to the hot spot URL that is actively captured; the background content analysis unit 210 is configured to Data for content analysis. In the network traffic monitoring system, the hotspot resource is obtained first, and then the protocol reorganization and background content analysis are performed, so that the same content is processed only once, thereby reducing the burden of the protocol reorganization unit and the background content analysis unit, and providing the entire system. s efficiency. In addition, for distributed P2P resources, P2P file fragments distributed throughout the system can be actively captured to support monitoring of P2P traffic.

FIG. 3 depicts a network traffic monitoring system in accordance with another embodiment of the present invention. The network traffic monitoring system includes:

The traffic classification unit 302 is configured to perform traffic classification on the data packet.

The hotspot statistics unit 304 is configured to calculate the requested number of times of the uniform resource locator URL within a predetermined time to determine the hotspot URL;

The active crawling unit 310 is configured to actively capture resources corresponding to the hotspot URL;

The protocol reorganization unit 312 is configured to perform protocol reorganization on resources corresponding to the hot spot URL that is actively captured; and

The background content analysis unit 314 is configured to perform content analysis on the data after the protocol reorganization.

The hotspot statistics unit 304 further includes a hierarchical statistics unit 306 and a determining unit 308. The ranking statistics unit 306 is configured to establish a resource table to hierarchically count the requested times of the URL to determine whether each level of the URL is a hotspot URL. The resource table will store the number of times each level of URL is requested within a predetermined time and a predetermined threshold. The determining unit 308 is configured to determine the URL when the requested number of times of a certain URL exceeds a predetermined threshold within a predetermined time Is a hot URL. The resource corresponding to the hotspot URL may be a webpage or a P2P file fragmentation.

Figure 4 shows a flow chart of a method for network traffic monitoring. This method can reduce the burden of the protocol reorganization unit and the background content analysis unit, improve the efficiency of the whole system and reduce the cost. Secondly, for the distributed P2P resources, it can actively capture the P2P file fragments distributed throughout, and support the P2P. Traffic monitoring.

The method for monitoring network traffic shown in Figure 4 includes:

402: The traffic classification unit performs traffic classification on the data packet;

According to this embodiment, the data packet is subjected to traffic classification according to the protocol type to which the captured data packet belongs. If the protocol type to which the packet belongs is HTTP, only the request header is sent to the hotspot statistics unit. In the process of establishing a request by HTTP, the request header in the HTTP request message includes a request line, and the request line includes a request method, and the request method may be GET or POST. GET is generally used to get/query resource information, while POST is generally used to update resource information. When the client wants to read a document from the server, the GET request method is used. The GET request method requires the server to return the resource of the URL location to the client in the data part of the response message. The GET request method is used here. The URL of the request link is also included in the GET request line.

404: The hotspot statistics unit performs statistics on the requested times of the uniform resource locator URL to determine a hotspot URL;

Optionally, when the request method is GET, the requested number of Uniform Resource Locators (URLs) in the HTTP request header is counted within a predetermined time. The scheduled time can usually be set to 10 days. The number of times the URL is requested is sorted from high to low within 10 days, and the URLs that are sorted backwards are periodically cleared. When the number of times a certain URL is requested exceeds a predetermined threshold within a predetermined time, the URL is determined to be a hotspot URL, and the active crawling unit is triggered to perform an active crawling action.

406: The active crawling unit actively captures resources corresponding to the hotspot URL.

After the hotspot URL is determined, the active crawling unit actively captures the resource corresponding to the hotspot URL. The resource may be a webpage corresponding to the hotspot URL and other webpages to which the hotspot URL is linked; the resource corresponding to the hotspot URL may also be a file fragment distributed on different nodes in a peer-to-peer network (P2P).

408: The protocol reorganization unit performs protocol reorganization on resources corresponding to the hot spot URL that is actively captured.

410: The background content analysis unit performs content analysis on the data reorganized by the protocol.

For ease of understanding, the following describes two specific application scenarios.

Internet public opinion monitoring

Internet sensation refers to the hot spot that the public is most concerned about in real life. These are high The issues of concern are mainly spread through forums, blogs, microblogs and other means. Due to the rapid spread of the network, after some hot issues occur, it will be out of control in a short period of time. By monitoring the network public opinion, we can respond to public emergencies in the network and fully grasp the social conditions and public opinion.

In this application scenario, the hotspot statistics unit determines the hotspot URL by counting the requested number of URLs in the HTTP/GET request within a predetermined time, and then actively crawling the webpage corresponding to the hotspot URL and other links thereof. The webpage can achieve the purpose of public opinion monitoring.

In some embodiments, the hotspot statistics unit makes a record every time a HTTP/GET message is received within a predetermined time. The URL can be hierarchically counted in the form of a resource table. The depth of the statistics is determined according to the requirements of the monitoring. It will be understood by those skilled in the art that each divisor (/) in the URL is divided into one level. For example, for a URL of www.xxx.com/sport/football/fifa2012/index.html, you can set the stats depth to 3. The first level is 3⁄4 www.xxx.com; the second level is www.xxx.com/sport; the third level is www.xxx.com/sport/football. The statistics and predetermined thresholds are stored in the resource table.

It should be noted that the setting of the threshold is usually referred to the empirical value. If the experience value is set too low, it will cause a large amount of content to be cached locally. If the setting is too high, it will lead to the omission of some hotspot information. The experience value can be reasonably set according to the definition of the monitoring hotspot and the storage capacity of the system. The setting of the predetermined threshold can be related to the system used by the customer. For example, in China's national dry network, the threshold can be set to tens of thousands; in the provincial and municipal export networks, it can be set to several thousand. Table 1 below shows a schematic resource table for statistics on hotspot URLs: Table 1

Wherein, within a predetermined time, the number of requests 10000 of www.xxx.com exceeds the threshold 8000, and the URL is determined to be a hotspot URL. In some embodiments, the resource table may be stored on the data file in a hash table manner, and the index of the resource table is stored in the memory. The hash value is found according to the URL, and the index is found by the hash value, and the data file is directly located according to the index pointer. After the hotspot statistics unit determines the hotspot URL, the active crawling unit actively captures the webpage corresponding to the hotspot URL and other webpages to which the hotspot URL is linked. If the A webpage is a hot webpage, the A webpage contains a link to the B webpage, and the B webpage contains a link to the C webpage. In the case of a depth of 3, the A, B, and C pages are actively captured locally. The specific excavation depth in the actual application is set manually. Under normal circumstances, the excavation depth is 5 levels to complete the monitoring.

For example, if www.xxx.com is determined to be a hotspot URL, the active crawl unit sends an HTTP/GET request to www.xxx.com, which usually returns directly to Index.html. Analyze links on Index.html for breadth or depth crawling. Usually, the Index webpage represents a homepage, and the content of the webpages at all levels is crawled step by step from the homepage. Deep crawling uses recursive crawling of all encountered hyperlinks until the recursion reaches the required crawl level. Breadth crawling is to retrieve all the hyperlinks of a web page, send HTTP requests to fetch all the content, and then drill down to the required crawl level.

The captured resources are reorganized by the protocol for analysis in the background, and data such as independent IP (Internet Protocol, IP protocol, IP) address traffic, website page traffic, independent user traffic, new user traffic, etc. can be learned, thereby achieving sensational monitor.

P2P, Peer-To-Peer, is synonymous with peer-to-peer networks. A P2P network can be simply defined as a direct exchange to enable resource sharing between different systems. In a P2P network environment, computers connected via the Internet are treated as equal participants, their status is equal to each other, and each node participating in the communication is referred to as a Peer. In P2P mode, the boundary between the server and the client is canceled. Since data storage, processing, and network bandwidth are all operated in a completely decentralized, asynchronous manner, the various loads can be perfectly balanced. The P2P application mode is characterized by the more people downloading, the wider the bandwidth provided, the more seeds will be available, and the download speed will be faster and faster.

In a P2P application, the P2P node downloads the required seed file through the browser to the website, and then obtains and connects to the address of the Tracker server. After the connection is successful, the Track server returns other nodes (neighbor nodes) that are downloading the same resource file. Information. After the requesting node obtains the information, it sends a message to these neighboring nodes to establish a connection and download the resources, thereby realizing sharing resources and services between the peer nodes in the network. The seed file is the "index" of the downloaded file, and the index information and the Hash verification code of each block of the downloaded file are written into the seed file. The Tracker server is the server that collects the downloaders and provides this information to other downloaders, allowing the downloaders to connect to each other to transfer data.

It can be seen that the downloader needs to download the file content, first need to get the corresponding seed file, and then parse the seed. The file gets the address of the Tracker server and connects to the Tracker server. The downloader obtains the IP address of other downloaders (neighbor nodes) from the response message of the Tracker server, and connects other downloaders to complete the sharing of data and resources. In this process, the file to be downloaded is divided into several file fragments, which are respectively stored in different nodes, and the Tracker server can know the IP addresses of different nodes stored in each file fragment.

The communication between the node and the Tracker server is based on the HTTP protocol. That is, the node connected to the Tracker server needs to first send an HTTP/GET request to the Tracker server. The URL contained in the request is the address of the Tracker server recorded in the seed file.

In some embodiments, the hotspot statistics unit counts the number of times the P2P node sends the requested number of URLs in the HTTP/GET request to the Tracker server within a predetermined time. The URL is determined to be a hotspot URL when the number of requests for a URL within a predetermined time exceeds a predetermined threshold. The active crawling module requests the tracker corresponding to the hotspot URL to download the IP address of the node stored in each file fragment of the file, and then obtains different file fragments from different nodes, and reassembles the fragments into original content. For background content analysis unit for analysis. It can be understood that the active crawling unit here is similar to a P2P node.

A person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by a program to instruct related hardware, and the program may be stored in a computer readable storage medium, and the above mentioned storage medium may be It is a read-only memory, a disk or a disc.

Various exemplary logic blocks, units, circuits, components and/or components set forth in connection with the embodiments disclosed herein may be implemented by a general purpose processor, a digital signal processing (DSP), or an application specific integrated circuit (Application Specific Integrated) Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic component, discrete gate or transistor logic, discrete hardware component, or any combination thereof designed to perform the functions described herein. Implement or execute. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing components, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration.

The embodiment of the present invention counts the requested number of times of the Uniform Resource Locator URL to determine the hotspot URL within a predetermined time, and then actively crawls the resource corresponding to the hotspot URL for protocol reorganization and content analysis. Therefore, the burden of the protocol reorganization unit and the background content analysis unit can be reduced. In addition, the technical solution of the embodiment of the present invention can actively capture P2P file fragments distributed throughout the distributed P2P resources to support monitoring of P2P traffic. The method and system for network traffic monitoring provided by the present invention are described in detail above. For those skilled in the art, according to the idea of the embodiment of the present invention, there are changes in the specific implementation manner and application scope. Therefore, the content of the specification should not be construed as limiting the invention.

Claims

Rights request

A method for network traffic monitoring, the method comprising:

Divide classification of data packets;

Counting the requested number of Uniform Resource Locator URLs within a predetermined time to determine a hotspot URL;

Actively capturing resources corresponding to the hotspot URL;

Reorganizing the resources corresponding to the hot spot URL actively crawled; and

Content analysis of the data reorganized by the agreement.

2. The method of claim 1 wherein

Counting the requested number of URLs within a predetermined time to determine the hotspot URL includes: rating the requested number of times for the URL to determine whether each level of the URL is a hotspot URL.

3. The method according to claim 1 or 2, wherein

The resources corresponding to the hotspot URL include: a webpage or a peer-to-peer network P2P file fragmentation.

The method according to any one of claims 1 to 3, wherein the number of requested times of the URL is counted within a predetermined time to determine the hotspot URL comprises: when a certain URL is requested in a predetermined time When the predetermined threshold is exceeded, the URL is determined as the hotspot URL.

5. A system for network traffic monitoring, wherein the system comprises:

a traffic classification unit, configured to perform traffic classification on the data packet;

a hotspot statistic unit, configured to perform statistics on the requested times of the uniform resource locator URL within a predetermined time to determine a hotspot URL;

An active crawling unit, configured to actively capture resources corresponding to the hotspot URL;

The protocol reorganization unit is configured to perform protocol reorganization on the resources corresponding to the hot spot URL that is actively captured; and the background content analysis unit is configured to perform content analysis on the data after the protocol reorganization.

The system according to claim 5, wherein the hotspot statistics unit further comprises a hierarchical statistics unit, wherein the hierarchical statistical unit is configured to hierarchically count the number of requests for the URL to determine whether each level of the URL is a hotspot URL. .

7. A system according to claim 5 or claim 6 wherein:

The system according to any one of claims 5-7, wherein the hotspot statistics unit further comprises a determining unit, wherein the determining unit is configured to: when a certain URL is requested for a predetermined time, exceeds a predetermined number of times When the threshold is reached, the URL is determined to be a hotspot URL.