CN102685016B - Internet flow distinguishing method - Google Patents

Abstract

The invention discloses an internet flow distinguishing method. According to a small quantity of marked flow samples and by virtue of offline supervised learning classification, unmarked flows are identified according to the characteristics of classified flows, and application classes of generated flows can be predicted in the early stage of network flow generation, thereby ensuring the promptness of network supervision and classifying the network flows in an actual network environment further. Through further adding new application types in semi-supervised clustering, a correlation chart of application type marks and application types is perfected, and alleged flows in the network are effectively marked, therefore, flow data with accurate application type labels can be obtained in real time. Meanwhile, when the network environment changes, the change of the network environment is reflected in the semi-supervised clustering, and the requirement on the distinguishment of flows in a new network environment is further met.

Description

Internet traffic differentiating method

Technical field

The present invention relates to a kind of preparation method of net flow assorted, particularly relate to a kind of differentiating method of internet traffic.

Background technology

Internet traffic is distinguished mainly such as, according to the feature that network traffics present, data package size, the inter-packet gap time etc., predicts the network application type producing this flow.Thus network manager can carry out Inspect and control to the service condition of Internet resources according to classification results, and guarantees its service quality for the service provided.

The realization that existing network traffics are distinguished is mainly based on the intelligent method (corresponding to supervised classification) of supervised learning and the intelligent method (corresponding to semisupervised classification) based on semi-supervised learning.

Wherein, based on the network traffics differentiating method of supervised learning, its implementation procedure can be divided into two stages: training stage and cognitive phase.In the training stage, the flow sample in a large number with application type label is learnt, the Heuristics of net flow assorted is obtained by study, in other words, be exactly produce one group of rule that feature judges application type, i.e. disaggregated model according to the correspondence being obtained label by the study of the feature of the flow sample of described label.

At cognitive phase, just to the flow not having application type label, only can need divide decision boundary according to disaggregated model to the feature of flow, just can draw the application type of this flow.The advantage of the method: when having the flow sample abundance of application type label, disaggregated model can obtain abundant knowledge, can identify fast and accurately to not having the flow of application type label.Its shortcoming: being limited to the frequency that application type uses in real network environment, to make to obtain the flow sample with accurate application type label very difficult; The applicability of the method, by the constraint of its training sample, namely needs to have similitude between the network traffics of differentiation and the flow sample of train classification models; New application type can not be found, can only identify trained application type.

Realization based on the network traffics differentiating method of semi-supervised learning shows that the method uses a small amount of data on flows with application type label as tutorial message, a large amount of flows of application type label that do not have is distinguished, and can concrete application type be identified.Its implementation procedure also can be divided into two stages: clustering phase and mapping phase.At clustering phase, have the data on flows of application type label as tutorial message constrained clustering search procedure, the result obtained after cluster terminates is the set of multiple bunches.Wherein each bunch what comprise is multiple datas on flows with similar quality, between different bunches, there is larger otherness.At mapping phase, for bunch information that cluster obtains, have according to what bunch to comprise the application type that the sample of application type label is corresponding, by bunch information MAP to concrete application type, the flow sample of application type label that do not have namely comprised for this bunch uses the type to mark; For do not exist the flow sample with application type label bunch, the flow sample it comprised all is mapped as new application type.The advantage of the method: new application type can be found.The shortcoming of the method: its complexity calculated brings higher time delay and the expense of calculating.

Above content is the content that describes of internet traffic differentiating method mainly, and so the method for online intelligent recognition internet traffic is then normally simulated online condition and classified to off-line data (namely observing the data in a period of time of gathering).These class methods are passed through gathered network data, namely packet set aggregates into different network flows (sequence of data packet) according to the five-tuple (source IP address, source port number, object IP address, destination slogan, agreement) of packet, then former packets of this sequence of data packet are extracted, as the source observing feature, train and testing classification device with this, thus simulate online traffic classification.

The shortcoming of the method: do not complete classification task in real time in real network environment, the impact of change on categorizing system of network state is not taken into account, disposes actual online traffic classification system from real network environment and there is gap; Do not know owing to not having the data on flows of application type label which kind of application type it is specially and produces, the authenticity therefore for classification results needs to verify, but existing online classification technology but lacks the checking to classification results authenticity.

As supportive content, the identification of mixed traffic in network, for network operator and manager, has vital meaning.For this reason, research institution proposes a lot of sorting algorithm with regard to mixed traffic, as the sorting algorithm based on port with based on wrapping the sorting algorithm detected.Because increasing network application uses dynamic end slogan and encryption technology to send packet, therefore progressively popular along with dynamic end slogan in network application and encryption technology, make based on port numbers and lose validity based on the sorting algorithm that bag detects.Traffic classification method based on machine learning can overcome this problem, just becomes the emphasis of research.But the traffic classification algorithm based on machine learning needs the network flow data collection with accurate application type mark to come training classifier and testing classification device.

More be conducive to understanding relevant technological means herein to make those skilled in the art, at this, explanation is carried out to the object of some means wherein and application, only for related personnel, the continuity understanding of technical scheme is proposed to herein institute, do not form some technological means following and be considered as admission of prior art.

Flowing out for calling socket request the application type mark that the tcp data packet making of main frame produces this packet, needing to install Socket Hook on main frame and to drive and NDIS Hook drives.Use at the boundary of network and gather packet through border, networking based on the water flow collection device of FPGA, and the Packet Generation gathered to data processor.First data processor becomes stream the packet obtained according to five-tuple (source IP address, object IP address, source port number, destination slogan and the agreement) information integration of packet, then makes different data sets according to different demands.

In acquisition, there is the technology used in the process of the network flow data collection of accurate application type label as described below:

Socket Hook and NDIS Hook:

A kind of explanation of Hook is the one that provides in Windows in order to the system mechanism of " interruption " under replacing DOS, and Chinese is translated into " hook " or " hook ".After carrying out Hook to specific system event, once there is Hook event, the program of carrying out Hook to this event will be subject to the notice of system, and at this moment program just can make response in the very first time to this event.

The another kind explanation of Hook is then a platform of windows messaging treatment mechanism, and application program can arrange sub-journey to monitor certain message of specified window above, and the window monitored can be that other processes create.When message arrive after, target window process function pre-treatment it.That is, Hook Mechanism allows application program to intercept and capture process windows messaging or particular event.

Socket (Socket) interface is API (the Application Programming Interface of TCP/IP network, application programming interfaces), and Windows Sockets, namely Winsock is an API based on Socket model, it works in Windows application layer, provides the high level data irrelevant with bottom host-host protocol to transmit DLL (dynamic link library).In Windows system, use WinSock interface to provide the network access service based on TCP/IP agreement for application program, these services are that the function library provided by Wsock32.DLL dynamic link library has been come.

From illustrating above, any Windows must pass through WinSock interface accessing network based on the application program of TCP/IP.Application programming interfaces allow Application developer be called one group of routine function, and without the need to consider its bottom source code why or understand the details of working mechanism in it.We can remove control WinSock interface by Hook technology thus, and hook by the packet of WinSock interface, and then analyzes the packet of hook, obtain the application type information of the five-tuple information relevant with packet and this packet of generation.

NDIS (Network Driver Interface Specification, i.e. Network Driver Interface specification) has three types, is network interface card driver, intermediate driver, Protocol Driver program respectively.

Network interface card driver (Miniport Network Interface Card drivers) supervising the network interface card, NIC(Network Interface Card, network adapter, also known as network interface card) driver in its lower end direct net control interface card hardware, the interface providing the driving of a higher level to use in its upper end, this interface generally completes some following tasks: initialization network interface card, stop network interface card, transmit and receive data bag, operating parameter arranging network interface card etc.

Intermediate driver (Intermediate Protocol Driver) is operated between protocol driver and miniport driver.In the transport layer drivers of high level, intermediate driver resembles a miniport driver, and at the miniport driver of bottom, it resembles a protocol driver.Use the topmost reason of intermediate driver may be use new transport layer drivers and mutual format transformation in the miniport driver of unacquainted media formats a transport layer drivers existed and one, namely serve as the role of translation.

High-rise protocol driver (Upper Level Protocol Driver) is as various ICP/IP protocol, a protocol driver completes TDI(Transport Driver Interface, TDI) or other the application program interface that can identify be it user provides service.These driver distribution packets, the data copy that user is sent in packet, then by NDIS by the driver of Packet Generation to low layer, the driver of this low layer may be intermediate driver, also may be miniport driver.Certainly, it also provides a protocol layer interface in the lower end of oneself, is used for low layer driver mutual, and wherein topmost function is exactly receive the packet transmitted by low layer, and these communications are all completed by NDIS substantially.

From the above, NDIS Hook is exactly that based intermediate layer drives and realizes, and by hook protocol-driven and the mutual packet of Miniport Driver, obtains the relevant information of packet, and then obtain the application type information producing this packet, and application type information flag in the packet.

FPGA (Field Programmable Gate Array), i.e. field programmable gate array, it is the product further developed on the basis of the programming devices such as programmable logic array PAL (Programmable Array Logic), gate array logic GAL (Gate Array Logic), Programmable logic design (Programmable Logic Device).It occurs as a kind of semi-custom circuit in application-specific integrated circuit ASIC field, has both solved the deficiency of custom circuit, overcomes again the shortcoming that original programming device gate circuit number is limited.The feature such as FPGA has architecture and logical block is flexible, integrated level is high and the scope of application is wide.

Use FPGA device layout digital circuit, not only can design processes simplified, and volume and the cost of whole system can be reduced, increase the reliability of system.They manufacture plenty of time needed for integrated circuit and energy under spending traditional sense, avoid investment risk, become gang with fastest developing speed in electronic device industry.Use the major advantage of FPGA device layout digital system circuit as follows:

(1) flexible design

Use FPGA device, can not by standard series device in logic function limit, and amendment logic can be carried out in the either phase of system and use procedure, and need only can complete by carrying out reprogramming to FPGA device used, providing very large flexibility to system.

(2) function closeness is large

Function closeness refers to logic function quantity that can be integrated in given space.Assembly door number in programmable logic chip is high, and a slice FPGA can replace the digital integrated circuit chip of several, tens and even up to a hundred middle and small scales.Number of chips when realizing digital system with FPGA device is few, thus reduces the use number of chip, reduces printed substrate area and printed substrate number, finally causes comprehensive reduction of system scale.

(3) reliability is high

Reduce chip and printed panel number, can not only reduction system scale, and also it also greatly improves the reliability of system.The system of tool high integration degree is than having much higher reliability by the identical systems of the modular design of many low integrated levels.Use FPGA device decreases the core number required for the system of realization, and lead-in wire on a printed-wiring board and number of welds also reduce thereupon, so the reliability of system is improved.

(4) design cycle is short

Due to programmability and the flexibility of FPGA device, design a system required time with it and shorten dramatically than conventional method.FPGA device integration is high, and during use, the wiring of printed substrate circuit layout is simple.Meanwhile, after Prototype Design success, because developing instrument is advanced, automaticity is high, carries out logic Modification also very simple, convenient and rapid to it.Therefore, use FPGA device greatly can shorten the design cycle of system, accelerate the speed of launch products, improve product

Competitiveness.

(5) operating rate is fast

The operating rate of FPGA/CPLD device is fast, generally can reach hundreds of megahertz, be far longer than software.Meanwhile, the circuit progression realized after using FPGA device required for system is few again, and thus the operating rate of whole system can be improved.

(6) NetFPGA is with the RJ45 interface of four 1G and high-speed PCI bus, and Offhost carries out work and seldom takies the resource of main frame, and the operating efficiency of main frame so just can be made to be greatly improved.

Summary of the invention

Of the present inventionly be to provide a kind of internet traffic differentiating method with online intelligent recognition ability, can realize classifying to network traffics in real time online in real network environment.

In order to realize foregoing invention object, the technical scheme adopted is:

A kind of internet traffic differentiating method, comprises the following steps:

100. data flowing out part main frame in measured network are labeled according to the mapping relations of application program and predetermined application type label;

200. flow out measured network of network flow at measured network boundary place mirror image, and then the network traffics of this mirror image are forwarded two-way in order to process, and a road is used for semi-supervised clustering analysis, goes to step 310, and another road is used for supervised learning classification, goes to step 320;

310. semi-supervised clustering analyses: by the set of the clustering network flow cluster of mirror image, to bunch the marking of flow sample wherein including application type label, and bunch all flow samples comprised wherein not comprising the flow sample of application type label are all mapped as the new application type that semi-supervised clustering analysis draws; Then the result that semi-supervised clustering is analyzed is exported;

Application type new in the result that 400. foundation semi-supervised clustering are analyzed, adds the mapping relations item of new application program and application type label;

320. supervised learning classification:

321. use the flow sample be labeled in the network traffics of mirror image, existing label data training classifier;

322. classify to the flow be not labeled according to described grader, and sorted classification results is exported;

The flow that the result of 500. pairs of semi-supervised clustering analyses and step 322 classification results export carries out composition comparative analysis, for the generation and the semi-supervised clustering analytical method that instruct classification based training device.

According to above-mentioned internet traffic differentiating method of the present invention, the flow sample represented according to a small amount of mark, classified by the supervised learning of off-line, by the feature of the flow be classified, the flow be not labeled is identified, the early stage realization that can occur in network traffics is predicted the applicating category generating flow, ensure the promptness to network monitoring, thus realize the problem realizing net flow assorted under real network environment.

Add new application type by further semi-supervised clustering, the relation table of application type mark and application type, the flow claimed in network is effectively marked, thus the data on flows of accurate application type label can be had by Real-time Obtaining.Meanwhile, when network environment changes, this change can embody in semi-supervised clustering, and further meet to new network environment traffic differentiation.

Above-mentioned internet traffic differentiating method, step 100 comprises:

101. create a Socket Hash for the application type title and application type label mapping application program show according to predetermined;

102. for data flow, uses during socket call handle packet and utilizes Hook mechanism to intercept and capture corresponding packet, and at least obtain the application type title of the packet of outflow main frame wherein in application program;

103. and then, mate in described Socket Hash shows in steps 102 obtain application type title time, set up the corresponding relation of packet and respective application type mark; And add list item in the NDIS Hash table preset according to this corresponding relation;

104. to drive in procotol and carry out utilizing in data interaction Hook mechanism to intercept and capture the described packet flowing out main frame with Miniport Driver, this data packet matched have a NDIS Hash list item time, mark this packet.

Above-mentioned internet traffic differentiating method, the information obtained in step 102 also comprises the five-tuple flowing out and flow belonging to the packet of main frame;

Correspondingly, the tlv triple containing corresponding data bag in NDIS Hash list item.

Above-mentioned internet traffic differentiating method, is marked in packet IP head being labeled as of packet, thus when packet convergence flow, by obtaining described application type mark to the identification of header packet information.

Above-mentioned internet traffic differentiating method, is also forwarded by the interface card mirror image based on FPGA at network boundary place and flows out tested network of network flow.

Above-mentioned internet traffic differentiating method, described step supervised learning classification adopts the step of the flow sample generation grader be labeled as follows:

301. for all packets in flow sample, extract the five-tuple information of this packet, the stream record sheet then searching initial creation with judge whether to exist in the table the stream of acquisition five-tuple information match; If have, turn next step, otherwise add new stream record in described stream record sheet;

The packet meeting observation window requirement in 302. convection current record sheets sequentially carries out storing and carries out into stream according to the five-tuple information of correspondence and packet number and converges;

303. to equal in observation window bag number, to calculate the characteristic information of this stream in limited time when the packet numbers in a stream record, with the application type morphogenesis characters mated obtained and then be stored into file;

304. based on described file, trains according to selected supervised learning algorithm, generates and judges that flow is the rule of corresponding application type, and the set of rule forms grader.

Above-mentioned internet traffic differentiating method, according to the grader generated, the step that described step supervised learning classification is classified to the flow without label is as follows:

301 '. for without all packets in the flow of label, extract five-tuple, search current stream record sheet, go to step 302 ' when coupling has stream to record, otherwise create new stream record and add stream record sheet to;

302 '. one is added to the variable of the packet number of the stream of coupling, then judges whether this variable is less than the upper limit of observation window;

303 '. the packet meeting observation window requirement is sequentially stored and carries out into stream according to its five-tuple information and packet number and converge;

304 '. when the packet number of certain stream equals observation window upper limit N, calculate the characteristic information of this stream, for step 305 '; Repeat step 301 '-304 ', realize the Real time identification to network traffics.

305 '. use step 304 ' feature of this flow that obtains, judge according to described grader, draw to the application type of flow, and the application type feature of this stream and classification drawn carries out file storage, in order to calling.

Above-mentioned internet traffic differentiating method, described step 304 ' in, preset a characteristic information list, pass through obtained characteristic information and mate with the characteristic information in this characteristic information list the screening carrying out obtained characteristic information.

Describing more specifically technical scheme of the present invention below in conjunction with Figure of description, is that those skilled in the art better understands the present invention.

Accompanying drawing explanation

Fig. 1 has the network topological diagram of the internet traffic differentiating method of online intelligent recognition ability.

Fig. 2 has the overview flow chart of the internet traffic differentiating method of online intelligent recognition ability.

Fig. 3 generates the flow chart with accurate application type flag data.

Fig. 4 is based on the traffic forwarding flow chart of NetFPGA.

The flow chart of Fig. 5 grader training stage.

The flow chart of Fig. 6 grader online classification.

The flow chart of Fig. 7 semi-supervised clustering.

Fig. 8 classification results checking flow chart.

Embodiment

With reference to Figure of description 1, for having the network topological diagram of the internet traffic differentiating method institute subordinate network environment of online intelligent recognition ability, as shown in Figure 1.In measured network, select in minority network node deploy based on the module accurately applying mark, object produces to have label flow, these have label flow and majority of network node to produce without label flow and the flow entering this network, at network exit, network boundary is mirrored onto a stylobate in the network traffics transponder of FPGA in other words.All network traffics are forwarded to the server of server and the supervised learning classification running semi-supervised clustering analysis by this transponder.The latter will have label data to send into training classifier module, will send into online sort module without label flow.

According to foregoing, some technical problem is defined further, an object is the design of the online real-time grading module of network traffics, solve in prior art and cannot realize the real-time problem that network traffics are classified online in real network environment, the applicating category that the commitment that can occur in network traffics can realize generating flow is predicted, ensures the promptness to network monitoring.

Another object is the problem being difficult to acquisition for the data with accurate application type label, provides the method that can generate the data on flows with accurate application type label according to embodiments of the invention.

Moreover, according to the mapping table of the application type set up in advance and mark, the new application type that analysis expert as mentioned in this article draws and mapping table, the flow generated in network is marked effectively, thus the data on flows of accurate application type label can be had by Real-time Obtaining.

Another object is the problem retrained by the flow sample of train classification models for the applicability of supervised learning mode, the training sample being put into training classifier module is provided to be the data with accurate application type label of Real-time Obtaining from measured network, thus one can be had to be familiar with clearly to the situation of current network, and then can better effectively identify the flow do not indicated in current network, and when network changes, real-time update can be carried out to online classification model.

Another object is the problem that can not find new application type for supervised learning method, and the present invention uses the flow of off-line semi-supervised learning method to measured network to identify.This module is under the guidance with accurate application type label data, effectively identifies the flow of unknown applications type in measured network; Due to the characteristic of semi-supervised clustering, contribute to finding new application type, as the foundation of mapping table setting up application type and mark simultaneously.

Another object is the problem cannot effectively verified for the result of online classification, the result of off-line semi-supervised clustering and online classification result is used to carry out network traffics constituent analysis, then contrast verification is carried out, thus online classification result is verified, in order to ensure the true and reliable property of online classification.

First one, realization to the accurate marker having label flow and relevant auxiliary environment are described here, as shown in Figure of description 3, comprise following content:

First obtain that the method with the label flow of accurate marker shows as generally is to packet marking application type mark accurately, then the packet be labeled is extracted, network flow data collection can be made to the packet be extracted in further improved plan.As long as certainly simply distinguish to have label flow and just can meet without label flow here and realize requirement.Therefore, overall scheme can be understood like this, and it, primarily of three part compositions, is packet marking, data acquisition and data processing respectively.

Utilize Hook mechanism, the packet flowing out main frame is marked, network exit passes through router image, FPGA interface card gathers mirror image data stream, filters, markd for band Packet Generation is processed to processing server, the markd data set of band needed for finally generating on request.

First Hook mechanism is utilized to produce the application type mark of this packet to the tcp data packet making flowing out main frame based on socket call (Socket call also makes Socket ask, socket request); Then use at network boundary place the water flow collection device based on FPGA to gather the network traffics with accurate application type mark, and the network traffics with accurate application type mark gathered are sent to data processor; After the packet that data processor reception collector sends over, packet is pooled stream by the first five-tuple information according to packet and application type label information, and then be made into the data set that applicable different sorting algorithm requires, for training classifier or be classified device classification.

Wherein, the definition of the encyclopaedia of network boundary is which unsafe problem of network has, and the behave taked to these aspects.Its definition is actually and is described from the angle of the security protection of network, being described as more specifically realizing resource-sharing is the driving source that network occurs, development for many years makes Internet become a reality, and global computer can be linked to be network, is linked to be an entirety; But computer is more, network size is larger, and safety also becomes problem.No matter be the domestic consumer being connected into Internet by ADSL, or be connected into the enterprise customer of Internet by special line, or be connected into the industry user of dedicated network by special line, be all faced with the impact of increasing unsafe factor." drawing ground and control " is the general way solving safety problem in reality, country has territory with sovereign right, city has administrative region, enterprise has autonomous garden, resident a guy space, these main bodys all have physical space and border, the network of different level of security is connected, just creates network boundary.

About socket call, multiple TCP connects or multiple program process may need by same Transmission Control Protocol port transmission data.In order to distinguish different program process and connection, many computer operating systems are that application program and ICP/IP protocol provide the interface being called socket (Socket) alternately, before to this has been comparatively detailed description.

Every platform main frame is applied Hook mechanism and disposes relevant module, to intercept and capture relevant message.Here two parts content is related to, two interfaces in other words, one is socket (Socket) is NDIS, called after Socket Hook driving and NDIS Hook drive respectively, mode as such as interim driver is present in system, and the feature according to interim driver carries out the acquisition of relevant information.Also carried out clearly describing to this background technology part, those skilled in the art easily realizes based on Hook mechanism.

By the deployment of these two modules, the packet marking of Socket call outflow main frame is produced to the application type mark of this packet.Socket Hook drives and is used for intercepting and capturing the packet using socket call transmission, and obtains the five-tuple information of this packet according to different application and/or produce the application type information of this packet, preferably at least should contain application type information; Then the information obtained, drive use as the five-tuple of this packet and application type label information are transferred to NDIS Hook.

NDIS Hook drives the packet intercepted and captured and use socket call transmission, and obtain the triplet information of this packet, then drive the corresponding relation of five-tuple and the application type mark passing and come to compare with Socket Hook, if there is match information just application type to be marked on the TOS position of this packet IP head, if without match information, just do not process this packet, directly transmit this packet.

Compared to five-tuple, for subscriber's main station, local ip address is fixing, and we are again processing TCP packets, and so agreement also just there is no need to have compared, and the program both saved the processing time, also improved the efficiency of kernel processes.

In order to the realization making those skilled in the art more clearly understand Socket Hook and Socket Hook, a concrete implementing procedure as shown in Figure 3 and Figure 4.

Before this, the document marked for the application type title and application type that map application program, in other words a mapping table for an application type title and application type mark is set up, as table 1.Calling of the table preestablished according to this, creates a Socket Hash by leaving remainder method and open address method and shows in order to calling.

The mapping table of table 1 application type title and application type mark

Application type title	Application type marks
		Thunder.exe	1
eMule.exe	2
		360se.exe	3
TheWorld.exe	4
		QQ.exe	5
Msnmsgr.exe	6
		…	…
Other	255

Socket Hook drives the concrete steps of the corresponding relation of five-tuple and the application type mark obtaining and call the packet that Socket call transmits as follows:

A1., before startup Socket Hook, foregoing table 1 or a readable document has been established.

A2. use the leaving remainder method in Hash table and open address method that all the elements in the mapping table of application type title and application type mark are stored into during Socket Hash shows when starting Socket Hook, list item is as shown in table 1.

A3. when application program performs Socket call handle packet, Socket Hook drives will intercept and capture this packet, and according to the receiver function of Socket call with send the flow direction that function judges packet, if the packet flowing into main frame does not just process, directly transmit this packet; If flow out the packet of main frame, go to step a4.

A4. for the packet flowing out main frame, namely need the packet marked, Socket Hook drives the five-tuple information that just obtains this packet and produces the application type name information of this packet, and then compares according to the application type title in application type title and the Socket Hash table that creates before.

If a5. there is the information of coupling, just sets up the five-tuple of this packet and the corresponding relation of application type mark, if there is no the information of mating, just the application type of this packet is labeled as 255/

Note: 255 is self-defining values, represent the packet that those belong to the outflow main frame that TCP connects, but the application type producing this packet is not added in the mapping table of application type title and application type mark, also can be used for verifying the application that also there are those and be not labeled, thus the mapping table of further application typonym and application type mark.Table 1 can be revised accordingly, for follow-up amendment provides basis.

In addition, represent with eight see the diffserv field of table 1 due to IP packet header, removing TOS (terms of service, terms of service) position be 0 can not use outside the application that to be used to 255 to be marked at and to can not find in application type title and application type mark mapping table, also have 254 values can be used to tag application type, most application can be met.

A6. Socket Hook drives and uses the mode of METHOD_IN_DIRECT that the corresponding relation of the five-tuple of packet and application type mark is written in internal memory.

The mode of METHOD_IN_DIRECT, i.e. DMA(Direct Memory Accessory, direct memory access mode, normal title also has direct internal memory reading manner, direct memorymodel), the buffering area locking that Socket Hook can drive the DeviceIoControl function called to specify by operating system, then remap a sector address under kernel mode address, until kernel mode is finished drilling, work terminates, operating system just can discharge this section of buffering area, the memory address of Kernel Driver calling party pattern can be avoided like this, thus avoid data and be in use tampered.

In addition, for the packet flowing out main frame, can judge whether this packet is TCP Transmission Control Protocol further, transmission control protocol) packet, if not tcp data bag, just do not carry out any process, directly send this packet.If tcp data bag, Socket Hook driving is just extracted the five-tuple information of this packet and is obtained the application type title called this Socket and ask.Whether this differentiation is that the mode of tcp data bag can provide required packet for follow-up application.

NDIS Hook drive for flow out main frame packet marking accurately application type mark concrete steps as follows:

B1. can NDIS Hash that first initialization one is empty show when starting NDIS Hook and driving, when NDIS Hook drive in internal memory, obtain the information of the tlv triple of packet and application type mark in METHOD_IN_DIRECT mode time, will use the leaving remainder method in Hash table and open address method that the tlv triple of packet and application type mark are write during NDIS Hash shows.

The corresponding relation of the tlv triple of table 2 packet and application type mark

Hash Round Robin data partition	Source port number	Object IP address	Destination slogan	Application type marks
					417	18327	202.194.64.200	8000	5
56	22958	58.254.134.211	80	1
					1301	23727	212.63.206.35	4242	2
72	23452	119.118.15.225	53	3
					1806	23812	202.194.64.200	8000	4
932	23064	60.217.235.148	80	6
					…	…	…	…	…

NDIS Hook extracts the reason of triplet information: for subscriber's main station, local ip address is fixing, and we are again processing TCP packets, and so agreement also just there is no need to have compared, the program both saved the processing time, also improved the efficiency of kernel processes.The corresponding informance of triplet information and application type mark is stored in NDIS HASH table (namely the tlv triple of packet and the mapping table of application type mark, as shown in table 2).

B2. when NDIS Hook drives intercepting and capturing to call the packet of socket call, the flow direction of this packet is judged according to the transmission function of socket call and receiver function, if flow into the packet of main frame, just do not carry out any process, directly transmit this packet.

B3. if flow out the packet of main frame, just the type of this packet is judged according to this: be the packet of end TCP connection, ask to set up the packet of TCP connection or the packet of tcp data transmission.

Known, the control bit in packet TCP header information account for six in TCP head, is urgent pointer respectively: URG; Acknowledgement indicator: ACK; Push mark: PSH; Reseting mark: RST; Synchronous mark: SYN; Termination flag: FIN.

B4. first judge whether this packet is the packet terminating TCP connection, if terminate the packet of TCP connection, NDIS Hook drives the triplet information just obtaining this packet, and with the triplet information in NDIS Hash table compares, if there is the information of coupling, with regard to the record deleted and the triplet information of this packet matches; If without the information of coupling, be just left intact, directly transmit this packet.

Like this, can space be saved, the relevant information of the packet be disposed is deleted.

B5. if not the packet terminating TCP connection, just judge whether this packet is the SYN packet that TCP connection is set up in request, if SYN packet, is just left intact, and directly transmits this packet.

B6. the SYN packet of TCP connection is set up if not request, just judge whether this packet is the packet that tcp data transmits, if be with loaded data transfer rate bag, NDIS Hook drives the triplet information just obtaining this packet, and with the triplet information in NDIS Hash table compares, if without match information, just any process do not done to this packet and directly transmit this packet; If there is match information, just the application type of matching result mark added on COS region (i.e. TOS position), then recalculate the IP header checksum of these data, after having revised, just transmit this packet.

With reference to Figure of description 4, from the exit of monitored network, the gigabit ethernet port COM0 in figure on FPGA board is connected to obtain whole flows that mirror image forwards with the mirror port of network egress.All IP bag, by checking whether as IP bag, is undertaken being forwarded to semi-supervised clustering server by the COM port or COM device or COM1 of NetFPGA by the flow of coming for mirror image.IP packet analyzed further and is forwarded to supervised learning server, whether being 0 by TOS position in the header fields that checks each IP packet, packet being forwarded to different ports and flowing out.

TOS position be not 0 packet be have label data bag, be sent to gigabit ethernet port COM2, this port is connected with the network interface card 1 of supervised learning server, receive from network interface card 1 and have label data as the training sample of supervised classification method, train sorting technique, the disaggregated model drawn is for ONLINE RECOGNITION network traffics.TOS position be 0 packet be without label data bag, be sent to gigabit ethernet port COM3, this port is connected with the network interface card 2 of supervised learning server, receives without label data from network interface card 2, according to the disaggregated model that obtains of training, carries out online Real time identification.

Enter the flow of off-line training classifier modules, as training set, be combined with supervision machine learning intelligence Algorithm for Training grader.Send into the flow of online sort module, the top n packet based on each stream extracts feature, then uses grader to classify.Be forwarded to the flow of semi-supervised clustering Analysis server, use semi-supervised clustering analysis module to analyze, this module has the function finding new application.Finally, classification results and semi-supervised clustering result are done check analysis, complete whole online traffic classification task.

See Figure of description 4, be the traffic forwarding flow chart based on NetFPGA, the PHY in NetFPGA refers to four gigabit ethernet interfaces on NetFPGA board, and its twisted pair line connection by standard is in local area network (LAN).On the interface of physical layer, in order to internal system clock can be reduced, add a RGMII interface module in FPGA inside, coordinate outside BCM5464 chip to complete the physical layer interface of network.Its concrete implementation step is as follows:

C1. for the network traffics that mirror image is come, prime minister enters V2, namely in core FPGA chip by the PHY module of NetFPGA and RGMII interface module.

C2. judge in V2 chip, first the header information of NF2 packet is judged, it is 72b bit wide that FPGA board detects self-defining data format, the wherein packet header of front 8b position NF2 packet, when it is 8 ' hff, show the beginning of NF2 packet, now equipment starts to detect packet header, otherwise terminates for NF2 packet.

C3. judge whether packet is IP bag, if it is processes further, if not then abandoning this packet.

C4. for IP packet, judge whether its NF2 packet terminates, if not, be forwarded to FIFO1 mouth to data, if NF2 packet terminates, then FIFO is closed condition, can not carry out storage operation.Whether the TOS position simultaneously judging packet is zero, if be zero, then indicates without label data, judges whether NF2 data terminate, if not, be forwarded to FIFO3 mouth, if TOS position non-zero, then be expressed as label data, judged whether NF2 packet terminates, if otherwise be forwarded to FIFO2 mouth.

In D5.FIFO data successively by RGMII interface module and PHY parts by data retransmission.。

Reference: Chinese CN102253909A application for a patent for invention announce multipurpose pci interface under disclosed FPGA environment and data transmission method thereof and herein based on the board of FPGA except interface section is different, control section and basic circuit are consistent.

Two, for a tested Campus Networks, Figure of description 2 illustrates the overview flow chart of the internet traffic differentiating method of online intelligent recognition ability, and its concrete implementation step is as follows:

In 110. monitored Campus Networks, the set of network nodes that labels provides the data traffic of label based on accurate application type labeling method; The set of network nodes do not labelled normally outwards sends data.

111. at network exit, by all traffic mirrorings of measured Campus Network in the network traffics transponder that NetFPGA is housed.

112. are judged the header packet information of network traffics by NetFPGA, then forward according to different demands, whole traffic forwarding to semi-supervised clustering module, simultaneously, label data is had to be forwarded to the grader training stage, without label data by current grader Real time identification.

113. wherein, use has label data to carry out off-line training to grader, the disaggregated model obtained, for to without label data just Classification and Identification, along with the change of network, according to the disaggregated model obtained, real-time update is carried out to online classification device, obtain the grader required for step 112, make when needs upgrade grader, can real-time update.

114. and then, those skilled in the art more clearly knows, for the network traffics without label, observe the top n packet of each stream, and counting statistics feature on this basis, and predict according to the disaggregated model that step 113 obtains, to be predicted the outcome and traffic characteristic is stored into file, for analyzing in step 118.The timing in short period interval of this step performs, thus reaches Real time identification.

115. for whole flows of step 112 repeating, and adopt Novel semi-supervised to analyze, have label data as tutorial message in all flows, identify without label data flow, recognition result carries out file storage.If in analytic process, exist and according to the data traffic having label data identification, cannot analyze to step 116 according to event memory.

116. carry out manual analysis for the result of semi-supervised clustering, if any have label data cannot instruct identification without label network data, then according to (e.g., quaternary group information, the i.e. source IP address of data of the information about this flow of record, object IP address, source port number, destination slogan) review, analyzing it by that network behavior is produced, if new application behavior, then carry out signature analysis and store for subsequent use.

117. upgrade according to the mapping table of step 116 manual analysis result to application typonym and application type mark.

118. ratios of adding up various network application behavior in the classification results of online classification result and semi-supervised clustering respectively, and contrast, expert evaluates, if the flow distribution that two kinds of results draw is similar, then can correctly reflect current network condition, become phase-splitting difference too large if there is certain, then need expert to analyze, and analysis result information is used for instruct two kinds of Traffic identification technology.

The flow chart of the training stage of Fig. 5 presentation class device, first needs according to there being exemplar just to learn for supervised learning method, its disaggregated model exported, the rule of namely classifying; For sample to be identified, judge according to its feature, disaggregated model provides a recognition result, i.e. network application type.Training process is as follows:

D1. for the packet that each collects, extract five-tuple information from data packet header, then search current stream record sheet, judge whether the stream of five-tuple information matches; If had, forward step D2 to, otherwise create new stream record and fill in relevant data item information; Wherein flow the stream information that the current discrepancy of record sheet record is observed network, deposit stream record according to predetermined data structure.

D2. for the packet of the stream recorded information matched with five-tuple found, first the variable recording observation data bag number in the stream record of this coupling is added one, then judge whether this variable is less than the upper limit N of observation window.

D3. the packet meeting observation window requirement is carried out storing and carry out into stream according to its five-tuple information and packet number and converge.

D4. when the packet number observed of a stream equals the observation window upper limit ntime, according to the feature list analyzing the applicable online classification obtained, calculate the characteristic information of this stream, and from application type title and mark mapping table, find the label that corresponding application type name is referred to as this stream according to the occurrence of its TOS position., the correlated characteristic being communicated with this stream is stored into file.

D5. use the sample with application type label accurately of step D4 acquisition as training data, in conjunction with certain supervised learning Algorithm for Training grader, obtain disaggregated model and namely judge certain flow some rules as certain application type, for online network traffics identification.

Figure of description 6 is the flow chart of grader online classification, supervised learning method is used to identify unknown network flow for real-time online, first to calculate the feature of this flow, then use disaggregated model to judge according to the feature of this sample, the application type of this sample can be drawn.Its implementation procedure is as follows:

E1. for each packet got be not labeled, extract five-tuple information from data packet header, then searching the stream record sheet of the current stream record for depositing the packet be not labeled, judging whether the stream of five-tuple information matches; If had, forward step e 2 to, otherwise create new stream record and fill in relevant data item information.

E2. for the packet finding the stream recorded information matched with five-tuple, first to the variable of packet number of stream, one is added to hourly observation, then judge whether this variable is less than the upper limit N of observation window.

E3. the packet meeting observation window requirement is carried out storing and carry out into stream according to its five-tuple information and converge.

E4., when the packet number of certain stream equals observation window upper limit N, according to the feature list analyzing the applicable online classification obtained, the characteristic information of this stream is calculated, for step e 5.So circulation performs the process of step e 1-E3, just can realize the Real time identification to network traffics.

The feature of this flow E5. using step e 4 to obtain, judges, draw the application type of this flow, and the application type feature of this stream and classification drawn carries out file storage, for keeper in conjunction with the disaggregated model set up under off-line state.

Fig. 7 is the flow chart of semi-supervised clustering, the result that semi-supervised clustering module obtains contributes to finding new application type from unknown categorical data, for revising the mapping table generating the application type title that has in accurate application type identification data module and application type mark, thus obtain and more abundant have label data.

F1. all popular for the measured network collected, first according to the five-tuple information obtained, carries out convergences formation data flow.

Whether the TOS position F2. detecting the packet of composition stream is zero, if non-zero, represents that this stream is for there being label data, is stored into the set of label stream by this stream, if be zero, represent that this stream is without label data, is stored to the set without label stream.

F3. for closing without label adfluxion and having tag set stream to calculate the feature of stream respectively, collaborate for there being tally set, contrast according to the mapping table that the occurrence of its TOS position and the application type title used when labelling and application type mark, obtain its accurate application type and as its a feature.

F4. label data set will be had and without label data set, semi-supervised clustering is carried out according to the direct similitude of flow, namely use and have exemplar as tutorial message, accelerate process and the accuracy thereof of cluster, the object of cluster is that the network traffics with similar characteristic are brought together formation one bunch, after cluster terminates, obtain the relevant information of multiple bunches, form the set of multiple bunches.

F5. for cluster obtain bunch set, observe the sample whether comprising in each bunch and have application type label, if had, in having the type of exemplar to mark bunch, other do not have markd sample, are about to bunch information MAP to network application type; If the sample in bunch is without label data, then be labeled as " the unknown " type, then by manually analyzing, new application type is determined whether, the mapping table of the application type title used time if it is for revising and labelling and application type mark.The result obtained by semi-supervised clustering stores, and the result then obtained with Supervised classification carries out contrast verification.

Fig. 8 is then the schematic diagram of classification results checking, and by manual analysis, the recognition result according to two kinds of sorting techniques judges, feedback information is used to guide adjustment two kinds of sorting techniques.

First, the result that the result obtained semi-supervised clustering respectively and Supervised classification obtain carries out network traffics constituent analysis.

Then, the analysis result that two kinds of traffic classification methods obtain is contrasted, draws comparing result.

Moreover expert evaluates comparing result, if two kinds of results are similar, represent that on-line system classification is normal, if in comparing result, two kinds of too large then expert feedback information of result difference, to two kinds of method for recognizing flux, adjust it.

Claims (7)

1. an internet traffic differentiating method, is characterized in that, comprises the following steps:

100. data flowing out part main frame in measured network are labeled according to the mapping relations of application program and predetermined application type label;

200. flow out measured network of network flow at measured network boundary place mirror image, and then the network traffics of this mirror image are forwarded two-way in order to process, and a road is used for semi-supervised clustering analysis, goes to step 310, and another road is used for supervised learning classification, goes to step 320;

310. semi-supervised clustering analyses: by the set of the clustering network flow cluster of mirror image, to bunch the marking of flow sample wherein including application type label, and bunch all flow samples comprised wherein not comprising the flow sample of application type label are all mapped as the new application type that semi-supervised clustering analysis draws; Then the result that semi-supervised clustering is analyzed is exported;

Application type new in the result that 400. foundation semi-supervised clustering are analyzed, adds the mapping relations item of new application program and application type label;

320. supervised learning classification:

321. use the flow sample be labeled in the network traffics of mirror image, namely have label data training classifier;

322. classify to the flow be not labeled according to described grader, and sorted classification results is exported;

The flow that the result of 500. pairs of semi-supervised clustering analyses and step 322 classification results export carries out composition comparative analysis, for the generation and the semi-supervised clustering analytical method that instruct classification based training device;

Wherein, step 100 comprises:

101. create a Socket Hash for the application type title and application type label mapping application program show according to predetermined;

102. for data flow, uses during socket call handle packet and utilizes Hook mechanism to intercept and capture corresponding packet, and at least obtain the application type title of the packet of outflow main frame wherein in application program;

103. and then, mate in described Socket Hash shows in steps 102 obtain application type title time, set up the corresponding relation of packet and respective application type mark; And add list item in the NDIS Hash table preset according to this corresponding relation;

104. to drive in procotol and carry out utilizing in data interaction Hook mechanism to intercept and capture the described packet flowing out main frame with Miniport Driver, this data packet matched have a NDIS Hash list item time, mark this packet.

2. internet traffic differentiating method according to claim 1, is characterized in that, the information obtained in step 102 also comprises the five-tuple flowing out and flow belonging to the packet of main frame;

Correspondingly, the tlv triple containing corresponding data bag in NDIS Hash list item.

3. internet traffic differentiating method according to claim 1, is characterized in that, is marked in packet IP head being labeled as of packet, thus when packet convergence flow, by obtaining described application type mark to the identification of header packet information.

4. internet traffic differentiating method according to claim 1, is characterized in that, is also forwarded flow out tested network of network flow at network boundary place by the interface card mirror image based on FPGA.

5. internet traffic differentiating method according to claim 1, is characterized in that, described step supervised learning classification adopts the step of the flow sample generation grader be labeled as follows:

301. for all packets in flow sample, extract the five-tuple information of each packet, the stream record sheet then searching initial creation with judge whether to exist in the table the stream of acquisition five-tuple information match; If have, turn next step, otherwise add new stream record in described stream record sheet;

The packet meeting observation window requirement in 302. convection current record sheets sequentially carries out storing and carries out into stream according to the five-tuple information of correspondence and packet number and converges;

303. to equal in observation window bag number, to calculate the characteristic information of this stream in limited time when the packet numbers in a stream record, with the application type morphogenesis characters mated obtained and then be stored into file;

304. based on described file, trains according to selected supervised learning algorithm, generates and judges that flow is the rule of corresponding application type, and the set of rule forms grader.

6. internet traffic differentiating method according to claim 5, is characterized in that, according to the grader generated, the step that described step supervised learning classification is classified to the flow without label is as follows:

301 '. for without all packets in the flow of label, extract five-tuple, search current stream record sheet, go to step 302 ' when coupling has stream to record, otherwise create new stream record and add stream record sheet to;

302 '. one is added to the variable of the packet number of the stream of coupling, then judges whether this variable is less than the upper limit of observation window;

303 '. the packet meeting observation window requirement is sequentially stored and carries out into stream according to its five-tuple information and packet number and converge;

304 '. when the packet number of certain stream equals observation window upper limit N, calculate the characteristic information of this stream, for step 305 '; Repeat step 301 '-304 ', realize the Real time identification to network traffics;

305 '. use step 304 ' feature of this flow that obtains, judge according to described grader, draw to the application type of flow, and the application type feature of this stream and classification drawn carries out file storage, in order to calling.

7. internet traffic differentiating method according to claim 6, it is characterized in that, described step 304 ' in, preset a characteristic information list, pass through obtained characteristic information and mate with the characteristic information in this characteristic information list the screening carrying out obtained characteristic information.