CN110958186A

CN110958186A - Network equipment data processing method and system

Info

Publication number: CN110958186A
Application number: CN201911250619.7A
Authority: CN
Inventors: 谭小彬; 陈令安; 佟欣欣; 杨坚; 郑烇
Original assignee: University of Science and Technology of China USTC
Current assignee: University of Science and Technology of China USTC
Priority date: 2019-12-09
Filing date: 2019-12-09
Publication date: 2020-04-03

Abstract

The invention discloses a network equipment data processing method and a system, which can transparently and automatically optimize network load without matching other algorithms through automatic detection, cache and advanced response. The invention can optimize the problem of mass repeated data transmission in the current network, avoid extra special hardware overhead and does not need a complex load balancing algorithm. In addition, the invention does not need the software algorithm cooperation of the server and the client, does not need a control center like a CDN, does not need to manually configure a data mirror image, is finished on network equipment, and is transparent to the server and the client. The data traffic and the request traffic which can be detected can be automatically cached, repeated data requests can be automatically accelerated through caching, and all other traffic is forwarded as a normal switch, so that the method has good compatibility with the existing network software and hardware.

Description

Network equipment data processing method and system

Technical Field

The invention relates to the technical field of computer networks, in particular to a network equipment data processing method and system.

Background

In the last decade, with the development of the internet and mobile internet, the characteristics of network traffic have changed dramatically. At the beginning of internet development, different devices and hosts were connected as main targets, and data was transmitted between two hosts connected to each other. As the internet has grown deeper into the public, the internet has become the main way for the public to obtain media information, for example, the traffic of video transmission in the network has increased greatly, and only video service providers Youtube and Netflix occupy 50.3% of the downstream traffic in north america according to the relevant data display. The large amount of multimedia traffic generates a huge pressure and cost for network operators. A large part of these multimedia traffic is redundant traffic brought by content distribution type services, consuming a large amount of network resources.

In the conventional content distribution, a connection is established between a user and a server once when the user acquires a multimedia file, so that a large number of users need to establish a connection with the server at the same time when acquiring the same file, and the establishment of the connection requires resources of the server, thereby generating huge burden on the server. In addition, a large amount of content distribution type services in the current internet generate a large amount of redundant content, and the repeated transmission of the content also occupies a large amount of network bandwidth, which can reduce the network resource utilization efficiency and user experience. For the transmission of repeated traffic, the following methods exist:

(1) larger server clusters and higher bandwidth are employed to cope with the growing traffic demand: on the content provider side, large video websites tend to deploy video distribution services using server clusters, and the number of servers is increasing, higher performance servers are purchased, and higher network bandwidth is accessed for data centers. In the aspect of operators, new network nodes are deployed, and the throughput and bandwidth of the existing network are expanded by updating equipment and the like. This allows the content provider to connect (and serve) more users at the same time, providing users with higher bandwidth, higher quality multimedia video. However, the method of upgrading hardware may bring high cost to content providers and operators, which is not beneficial to the development of enterprises on one hand, and on the other hand, enterprises may also tend to share the cost with users, thereby affecting user experience.

(2) Use of Content Delivery Network (CDN): the CDN shortens the distance between the server and the user by deploying the server at the edge of each local network, stores a mirror image of a media file to be distributed on the edge server, and realizes efficient content distribution by modules such as load balancing and scheduling of the central platform. The CDN is a content delivery acceleration method widely used at present, and a third-party CDN service provider provides services. Although the CDN also needs a large number of servers, since the servers are deployed at the edge of the network and are close to the user, the CDN can provide a better content delivery service and save the network overhead of the backbone network. However, CDNs still require complex algorithms for load balancing and scheduling, and purchasing CDN services still entails higher costs for content providers.

(3) With new network architectures, such as Information Centric Networking (ICN): in the ICN, a data sender and a data receiver do not need to be directly connected, the sender only needs to send data to a network (instead of a specific user), the receiver requests data from the network, and both parties can complete data transmission without establishing a direct connection, thereby greatly reducing the transmission of repeated traffic and having higher efficiency compared with the CDN. However, the current internet is always developed on a TCP/IP protocol, a large amount of devices and software are designed for the TCP/IP protocol, and the deployment of the ICN requires a new upgrade of existing hardware devices and supporting software, which is a huge cost for the whole internet, and thus has not been pushed forcefully.

In summary, for the problem of mass and repeated content distribution in the internet, various solutions have advantages and disadvantages, and the problems that the cost is high, the deployment is difficult, and the algorithm is complex are difficult to overcome.

Disclosure of Invention

The invention aims to provide a network equipment data processing method and a network equipment data processing system, which can eliminate the transmission of redundant contents by utilizing the caching capacity of nodes, thereby improving the utilization rate of network resources.

The purpose of the invention is realized by the following technical scheme:

a network device data processing method comprises the following steps:

identifying input flow and determining a corresponding grouping type;

if the request packet is the request packet, extracting the unique content identification of the data from the flow, and matching in a content cache table; if the matching is successful, corresponding data is taken out from the content cache table, one or a series of response packets are constructed and sent to a sender of the request packets; if the matching fails, adding the unique content identification of the data into the request record table;

if the data packet is the data packet, the data packet is sent to a data cache module, the data cache module matches the unique content identifier of the data in the flow in the request record table, and if the matching is successful, the corresponding data is stored in a content cache table; and finally, the data cache module forwards the corresponding data outwards according to the destination address.

A network device data processing system, comprising: the system comprises a flow identification module, a data caching module, a forwarding and data acquiring module, a content caching table and a request recording table; wherein:

the flow identification module is used for identifying the input flow and determining the corresponding grouping category;

if the request packet is the request packet, the forwarding and data obtaining module extracts the unique content identification of the data in the flow and matches the unique content identification in the content cache table; if the matching is successful, corresponding data is taken out from the content cache table, one or a series of response packets are constructed and sent to a sender of the request packets; if the matching fails, adding the unique content identification of the data into the request record table;

According to the technical scheme provided by the invention, the flow is analyzed and identified, the flow can be classified into the request packet and the data packet, and the data packet is cached, so that the corresponding content data can be obtained from cache when the repeated request arrives, thereby reducing network transmission and improving the network resource utilization rate, and the purposes of not modifying a network protocol and client software and being transparent to a user are achieved; in addition, a great deal of cost can be saved compared with the existing scheme.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.

Fig. 1 is a flowchart of a data processing method of a network device according to an embodiment of the present invention;

fig. 2 is a schematic overall block diagram of a network device data processing system according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.

Traditional network multimedia file distribution requires a user to establish a connection with a server to transmit, which is the root cause of bandwidth occupation by a large amount of repeated traffic, and the push of a new generation of NDN network without connection faces heavy resistance. Considering that the root cause of duplicate content transmission in the internet today is that data streams are identified based on five-tuple with no content information, if content information is available on a network node, the caching capabilities of the node can be exploited to eliminate the transmission of redundant content. Therefore, on the basis of a new generation of intelligent network devices (intelligent switches), the invention provides a network device data processing method based on traffic identification and in-network caching, which is executed inside each network device, as shown in fig. 1, and mainly comprises the following steps:

step 1, identifying input flow and determining a corresponding grouping type.

It will be understood by those skilled in the art that traffic refers to all packets passing through a network device, and data refers to the content (picture, audio, video) to be transmitted or requested by a packet, that is, the data is part of the traffic.

In the embodiment of the invention, a Deep Packet Inspection (DPI) technology or a traffic identification technology based on machine learning can be adopted to identify the packet type of the traffic.

If the request packet is used for requesting specific data, the request packet is regarded as a request packet; for example a request for a URL (uniform resource locator) from a client to a server in the HTTP protocol.

If it is (one of) response data to a certain request, it is regarded as a data packet, which is a packet carrying file and media data; such as a response message from the server.

All but the above two packets are considered as other packets; for other packets, the processing is done in the conventional way, i.e. directly forwarded out according to the destination address.

Step 2, if the request grouping is performed, extracting the unique content identification of the data from the flow, and matching in a content cache table; if the matching is successful, corresponding data is taken out from the content cache table, one or a series of response packets are constructed and sent to a sender of the request packets; and if the matching fails, adding the unique content identification of the data into the request record table.

The content cache table is a data structure that caches files and media data that pass through the network nodes. The data caching module sends the data to be cached to a content caching table so as to extract the needed data from the content caching table to construct a response packet when needed, and the content caching table stores a plurality of data and unique content identifications of the corresponding data.

By parsing the request packet, a unique content identification, e.g. a URL or a corresponding hash value, can be extracted.

And matching the extracted unique content identifier, if the matching is successful, indicating that corresponding data is cached in the content cache table, constructing one or a series of response packets at the moment, and sending the response packets to a sender of the request packets, namely sending the response packets back to the source address. If the matching fails, the corresponding data is not cached in the content cache table, at this time, the unique content identification of the data is added into the request record table, and the request packet is forwarded according to the destination address of the request packet. On the one hand, the basic functions of the network device are guaranteed, and on the other hand, the corresponding data packets are saved when passing through the network device (step 3).

Step 3, if the data packet is a data packet, sending the data packet into a data cache module, matching the unique content identifier of the data in the flow in the request record table by the data cache module, and if the matching is successful, storing the corresponding data into a content cache table; and finally, the data cache module forwards the corresponding data outwards according to the destination address.

And requesting the main record in the record table and not matching the content cache table to the request packet, wherein the main content of the request record table is the unique content identification of the requested data.

The data caching module also analyzes the data packet, extracts the unique content identification in the data, and then performs matching in the request record table. If the matching is successful, the network device is used for indicating that the request corresponding to the data passes through the switch, and the probability that the corresponding data request passes through the switch again is high, so that the data in the data packet is extracted and stored in the content cache table. If there is no match, no action is taken. Finally, the packet is forwarded according to the destination address of the data packet to ensure the normal function of the network device. The data packets are not all stored in the content cache table in order to save unnecessary cache write overhead.

Those skilled in the art will appreciate that the network devices may be intelligent network devices, such as intelligent network devices with computing capabilities and cache space offered by network device providers represented by xinhua san, hua ye, etc.

The invention provides a network equipment data processing method, which can transparently and automatically optimize network load without matching other algorithms through automatic detection, cache and advanced response. The invention can optimize the problem of mass repeated data transmission in the current network, simultaneously avoids the overhead of extra special hardware (such as a server deployed at the edge of the network), does not need a complex load balancing algorithm, can reduce the network load, does not need the cooperation of software algorithms of a server end and a client end, does not need a control center like a CDN (content distribution network), does not need to manually configure a data mirror image, is finished on network equipment, and is transparent to the server end and the client end. The data traffic and the request traffic which can be detected can be automatically cached, repeated data requests can be automatically accelerated through caching, and all other traffic is forwarded as a normal switch, so that the method has good compatibility with the existing network software and hardware.

Another embodiment of the present invention further provides a network device data processing system, as shown in fig. 2, which mainly includes: a Traffic Identification module (TI), a Data Cache module (DC), a Forwarding and Data Acquisition module (FDA), a content Cache Table (CT), and a request record table (PRT); wherein:

In the embodiment of the invention, a deep packet detection technology or a traffic identification technology based on machine learning is adopted to identify the grouping category of the traffic.

In the embodiment of the invention, if the request packet is used for requesting specific data, the request packet is regarded as a request packet; if the request is response data for a certain request, the request is regarded as a data packet, and the data packet is a packet for carrying files and media; all but the above two packets are considered as other packets; for other packets, the packet is forwarded out directly according to the destination address.

In the embodiment of the invention, a plurality of data and unique content identifications of corresponding data are stored in the content cache table.

In the embodiment of the present invention, the unique content identifier is a uniform resource locator or a corresponding hash value.

In the embodiment of the invention, the system is arranged in each network device.

It will be clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be performed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to perform all or part of the above described functions.

Specific examples in application scenarios are given below in conjunction with the above-described schemes of the embodiments of the present invention.

Aiming at the problem of video distribution of a video website, after a user establishes TCP connection with the video website and clicks a video to be watched, a user browser initiates an HTTP request to the video website, and the URL of the request is the address. When the user request packet passes through the network equipment, firstly, the flow detection module detects that the packet belongs to the HTTP request through a machine learning algorithm, then, the FDA module extracts the URL of the request, and the URL uniquely locates the video required by the user. The FDA matches the URL to the cache in the content cache table, saves the URL to the PRT, assuming the video is not cached, and sends the request packet out at the normal destination address. When the video server responds to the request and the responding data packet passes through the intelligent switch, the intelligent switch can detect that the data packet is an HTTP response packet and send the HTTP response packet to the data caching module. The data caching module extracts the URL of the packet, searches the URL in the PRT, and if the matching is found, the data packet is forwarded normally and the payload of the data packet is cached to the CT. Thus, the CT buffers the most recently requested data.

When another user also logs in the video website to request the same video, the HTTP request sent by the browser passes through the intelligent network device, the flow detection module detects the request and then gives the request to the FDA, the FDA sends the URL of the HTTP request to the CT for retrieval, and finds that the requested URL is cached, so that a response message can be constructed instead of the target server according to information such as the target IP address and the port number of the request, and the response message is sent to the client. Thus, the burden of the server is reduced, the response time of the client is shortened, and repeated data transmission is avoided.

Through the above description of the embodiments, it is clear to those skilled in the art that the above embodiments can be implemented by software, and can also be implemented by software plus a necessary general hardware platform. With this understanding, the technical solutions of the embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.), and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods according to the embodiments of the present invention.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A network device data processing method is characterized by comprising the following steps:

identifying input flow and determining a corresponding grouping type;

2. The data processing method of claim 1, wherein a deep packet inspection technique or a traffic identification technique based on machine learning is used to identify the packet class of the traffic.

3. The network device data processing method according to claim 1 or 2,

if the request packet is used for requesting specific data, the request packet is regarded as a request packet;

if the request is response data for a certain request, the request is regarded as a data packet, and the data packet is a packet carrying files and media data;

all but the above two packets are considered as other packets; for other packets, the packet is forwarded out directly according to the destination address.

4. The method as claimed in claim 1, wherein the content cache table stores a plurality of data and unique content id of the corresponding data.

5. The method according to claim 1, wherein the unique content id is a uniform resource locator or a corresponding hash value.

6. A network device data processing system, comprising: the system comprises a flow identification module, a data caching module, a forwarding and data acquiring module, a content caching table and a request recording table; wherein:

7. The network device data processing system of claim 6, wherein the packet class of the traffic is identified using deep packet inspection techniques or machine learning based traffic identification techniques.

8. The network device data processing system of claim 6 or 7,

if the request is response data for a certain request, the request is regarded as a data packet, and the data packet is a packet for carrying files and media;

9. The network device data processing system of claim 6, wherein the content cache table stores a plurality of data and unique content identifiers of corresponding data.

10. The network device data processing system of claim 6, wherein the unique content identifier is a uniform resource locator or a corresponding hash value.