The method and device that Data Concurrent reports in a kind of DPI system
Technical field
The present invention relates to DPI (Deep Packet Inspection, deep-packet detection) technical fields, and in particular to a kind of
The method and device that Data Concurrent reports in DPI system.
Background technique
DPI technology is a kind of flow detection and control technology based on application layer, when IP data packet, TCP
(Transmission Control Protocol, transmission control protocol) or UDP (User Datagram Protocol, user
Datagram protocol) data flow by bandwidth management system based on DPI technology when, the system is by deeply reading IP payload package
Content recombinates the application layer message in seven layer protocol of OSI, to obtain the content of entire application program, then according to
The management strategy that system defines carries out shaping operation to flow, and by a variety of detection techniques, carries out data to packet and stream information
Extraction, data traffic analysis, user behavior analysis and Customer subscription information collection etc..
Bandwidth management system place similar with anti-virus software system based on DPI technology is: the application that can be identified
Type is necessary for known to system;By taking BT known to user (Bit Torrent, bit stream) as an example, the agreement of Handshake
Tagged word is " BitTorrent Protocol ".In order to guarantee application type known to system identification, Anti-Virus backstage is needed
A huge anti-virus signature database is wanted, the bandwidth management system based on DPI technology will also safeguard that is applied a characteristic
Library.When flow pass through bandwidth management system when, bandwidth management system by after unpacking application message and backstage property data base into
Row relatively determines application type.
But when there is new application to occur, backstage using property data base need to have new application identification and
Control ability.When the existing bandwidth management system reported data based on DPI technology, by the way of " successively reported data ",
The efficiency that data are sent is lower, and the total duration that data are sent is longer, so that the performance of bandwidth management system declines, clothes
The load of business device is larger.
Summary of the invention
In view of the deficiencies in the prior art, present invention solves the technical problem that are as follows: data in a kind of DPI system are provided
The method and device concurrently reported, the present invention not only increase the performance that reports of DPI, and realize that data efficient concurrently reports, and
The performance of big data parsing is improved, the load of server is reduced.
To achieve the above objectives, the method that Data Concurrent reports in DPI system provided by the invention, this method includes following
Step:
A: the starting of deep-packet detection DPI system, initialization report reporting for thread pool number of threads and to report queue length,
Go to step B;
B: report in thread pool it is all report thread to enter wait state, when reporting task to be added to reporting queue
When, will report task distribute to report it is idle in thread pool report thread, go to step C;
C: according to the type of service for reporting task, corresponding TCP file connection descriptor list is selected, step D is gone to;
D: determine that there are effective descriptors in the connection descriptor list of TCP file, select the effective descriptor, turn
To step E;
E: calling send function from effective descriptor, carries out TCP data transmission;Check the return value of send function
Value: if value is the length for sending data packet, TCP data is transmitted, and goes to step B and other tasks is waited to trigger;
If value, less than 0, TCP data transmission failure will send after fail count count_fail adds 1 and go to step F, if value
Greater than 0 and it is less than the length for sending data packet, then TCP data transmission does not complete, and goes to step G;
F: judging whether count_fail is less than server number of units, if so, by the common mark of the effective descriptor
Step C is gone to after being set to down state -1, otherwise goes to step B;
G: judging whether value is less than single DPI maximum transmitted byte number, if so, sending obstruction, will send obstruction meter
Number count_block goes to step H after adding 1, otherwise continues to execute step E;
H: judging whether count_block is less than positive integer N, if so, re-executeing the steps E;It otherwise will be described effective
The common mark of descriptor goes to step C after being set to blocked state 2.
Based on the above technical solution, if there is no effectively retouch in TCP file connection descriptor list in step D
Symbol is stated, then state is sent according to history, selects suitable descriptor and commonly used to save after mark is set to 1 to corresponding TCP
File connects in descriptor list, re-execute the steps C.
Based on the above technical solution, described in step D according to history send state, select suitable descriptor,
And it is commonly used after mark is set to available mode 1 and saves the detailed process into the connection descriptor list of corresponding TCP file are as follows:
D01: the ip for defining server in deep-packet detection DPI system is x, and port y used in server is ipxy, y root
According to reporting the type of service of business to select, every kind of type of service corresponds to different Service-Ports;Ipxy.duration [0] is right
Answered before current time that TCP sends occupied total duration, ipxy.duration [1] corresponding current time in 10 minutes report cycles
TCP sends occupied total duration in preceding 5 minutes report cycles;Ipxy.duration's [0] and ipxy.duration [1] is initial
Value is 0;
D02: respectively to currently reporting each ipxy carried in business to create TCP connection, socket is stored in each
The descriptor of ipxy connection;
D03: the transmission scoring ipxyscore of each ipxy, calculation formula are as follows: ipxyscore=are calculated
ipxy.duration[0]·α+ipxy.duration[1]·β;Alpha+beta=1 in above-mentioned formula;
D04: judging whether all ipxyscore are identical, if so, the service of selection ipxy.duration [1] minimum value
Otherwise device ip selects the server ip of minimum value ipxyscore_min in all ipxyscore;
D05: determining the destination address that the server ipx selected and port y send for this, by the ipxy connection of selection
Common mark is set to 1, after the descriptor of other ipxy connections is set to original state 0, saves to corresponding TCP file and connects description
It accords in list.
Based on the above technical solution, it after carrying out TCP data transmission in step E, needs to report week according to transmission
Phase updates ipxy.duration [0] and ipxy.duration [1].
Based on the above technical solution, determination reports task to be added to the process for reporting queue in step B are as follows: judgement
Report thread reports whether queue is sky, if so, not reporting task to be added to reports queue, continues to execute step at this time
Otherwise B reports task to be added to and reports queue.
Based on the above technical solution, exist in the connection descriptor list of judgement TCP file described in step D effective
Descriptor process are as follows: judge TCP file connection descriptor list whether be empty and the connection descriptor list of TCP file in
The common mark of All Files descriptor is not 1, if so, there is no effectively retouch in the connection descriptor list of TCP file
Symbol is stated, state is sent according to history, selects suitable descriptor and commonly used to be saved after mark is set to 1 to corresponding TCP text
Part connects in descriptor list, re-execute the steps C;Otherwise there are effective descriptors in TCP file connection descriptor list.
Based on the above technical solution, the detailed process of the effective descriptor is selected in D are as follows: judge TCP text
The descriptor that it is 1 with the presence or absence of common mark in descriptor list that part, which connects:
If so, selecting first common descriptor indicated for 1 in the connection descriptor list of TCP file;Otherwise judge TCP
The descriptor that it is 0 with the presence or absence of common mark in descriptor list that file, which connects:
If so, select first common mark in the connection descriptor list of TCP file for 0 descriptor and commonly used
Mark is set to 1, goes to step E;Otherwise judge in the connection descriptor list of TCP file with the presence or absence of the common description indicated for 2
Symbol:
If so, select first common mark in the connection descriptor list of TCP file for 2 descriptor and commonly used
Mark is set to 1, goes to step E;Otherwise it determines in the connection descriptor list of TCP file and effective descriptor is not present.
The device that Data Concurrent reports in DPI system provided by the invention based on the above method, which includes depth
Packet detection DPI system initialization module reports task allocating module, descriptor list selection module, effective descriptor to determine mould
Block, TCP data transmission module, fail count determination module, obstruction determination module and obstruction count determination module;
Deep-packet detection DPI system initialization module is used for: controlling depth packet detects the starting of DPI system, and initialization reports
Reporting for thread pool and reports queue length at number of threads, to reporting task allocating module transmission that task is reported to distribute signal;
It reports task allocating module to be used for: receiving after reporting task to distribute signal, control reports on all in thread pool
Report thread enters wait state, when reporting task to be added to task when reporting queue, will be reported to distribute to reporting in thread pool
Idle reports thread, sends descriptor list selection signal to descriptor list selection module;
Descriptor list selection module is used for: after receiving descriptor list selection signal, according to the service class for reporting task
Type, selects corresponding TCP file connection descriptor list, and Xiang Youxiao descriptor determination module sends effective descriptor and determines letter
Number;
Effective descriptor determination module is used for: after receiving effective descriptor judgement signal, judging that TCP file connects descriptor
It whether there is effective descriptor in list, if so, sending TCP data to TCP data transmission module transmits signal;Otherwise basis
History sends state, selects suitable descriptor and commonly used to be saved after mark is set to 1 to the connection of corresponding TCP file to retouch
It states in symbol list, sends descriptor list selection signal to descriptor list selection module again;
TCP data transmission module is used for: after receiving TCP data transmission signal, send letter is called from effective descriptor
Number carries out TCP data transmission;Check the return value value of send function: if value is the length for sending data packet, TCP
Data are transmitted, and report task to distribute signal to reporting task allocating module to send;If value, less than 0, TCP data passes
Defeated failure will send and send fail count judgement signal to fail count determination module after fail count count_fail adds 1, if
Value is greater than 0 and is less than the length for sending data packet, then TCP data transmission does not complete, and sends obstruction to obstruction determination module and sentences
Determine signal;
Fail count determination module is used for: judging whether count_fail is less than server number of units, if so, by described effective
The common mark of descriptor be set to after down state -1 to descriptor list selection module and send descriptor list selection letter
Number, otherwise to report task allocating module transmission report task distribute signal;
Obstruction determination module is used for: after receiving obstruction judgement signal, judging whether value is less than single deep-packet detection
DPI maximum transmitted byte number will send to block to count to count after count_block adds 1 to obstruction and sentence if so, sending obstruction
Cover half block, which sends obstruction and counts, determines signal, otherwise continues to send TCP data transmission signal to TCP data transmission module;
Obstruction counts determination module and is used for: judging whether count_block is less than positive integer N, if so, again to TCP number
TCP data, which is sent, according to transmission module transmits signal;Otherwise after the common mark of the effective descriptor being set to blocked state 2
Descriptor list selection signal is sent to descriptor list selection module.
Compared with the prior art, the advantages of the present invention are as follows:
With in the prior art by the way of " successively reported data ", present invention incorporates DPI to handle big data, multi-service
Demand, provide a kind of method and device that Data Concurrent reports in DPI system.This method is using in multi-service classification
Report, multi-thread concurrent report, filec descriptor transmission state mechanism and balanced multi-server load design, and not only increase
DPI's reports performance, realizes that data efficient concurrently reports, and improve the performance of big data parsing, reduces server
Load.
Detailed description of the invention
The flow chart for the method that Fig. 1 reports for Data Concurrent in the DPI system in the embodiment of the present invention;
The signal flow schematic diagram for the device that Fig. 2 reports for Data Concurrent in the DPI system in the embodiment of the present invention.
Specific embodiment
Invention is further described in detail with reference to the accompanying drawings and embodiments.
Method shown in Figure 1, that Data Concurrent reports in the DPI system in the embodiment of the present invention, comprising the following steps:
S101: deep-packet detection DPI system boot, each initiation of services in system are completed to initialize according to business configuration
Work data modes to be processed such as (complete the Policy receipts such as reporting policy and store work) business enter;According to configuration (in business
Report scale and physical equipment processing capacity) initialization reports reporting for thread pool number of threads and to report queue length etc., it goes to
S102。
S102: report in thread pool it is all report thread to enter wait state, judgement reports the queue that reports of thread to be
No is sky, if so, task is not reported to be added to queue is reported, continues to execute S102, otherwise reports task to be added supreme
Report queue, will report task distribute to report it is idle in thread pool report thread, go to S103.
S103: according to the type of service for reporting task, corresponding TCP (Transmission Control is selected
Protocol, transmission control protocol) file connection descriptor list, go to S104.
S104: judging in the connection descriptor list of TCP file with the presence or absence of effective descriptor, if so, S106 is gone to, it is no
Then go to S105.
The detailed process that whether there is effective descriptor in the connection descriptor list of TCP file is judged in S104 are as follows: sentence
Whether disconnected TCP file connection descriptor list is the common of All Files descriptor in the connection descriptor list of empty and TCP file
Mark is not 1 (available mode), if so, effective descriptor is not present in the connection descriptor list of TCP file, otherwise
There are effective descriptors in the connection descriptor list of TCP file.
S105: sending state according to history, selects suitable descriptor and commonly used to save after mark is set to 1 to right
In the TCP file connection descriptor list answered, S103 is re-executed.
S106: effective descriptor in selection TCP file connection descriptor list goes to S107.
The detailed process of S106 are as follows: judge in the connection descriptor list of TCP file with the presence or absence of the common description indicated for 1
Symbol:
If so, selecting first common descriptor indicated for 1 in the connection descriptor list of TCP file;Otherwise judge TCP
File connects the descriptor in descriptor list with the presence or absence of common mark for 0 (original state):
If so, select first common mark in the connection descriptor list of TCP file for 0 descriptor and commonly used
Mark is set to 1, goes to S107;Otherwise judge in the connection descriptor list of TCP file with the presence or absence of common mark for 2 (obstruction shapes
State) descriptor:
If so, select first common mark in the connection descriptor list of TCP file for 2 descriptor and commonly used
Mark is set to 1, goes to S107;Otherwise S105 is gone to.
S107: socket is taken out from effective descriptor and calls send function, TCP data transmission is carried out, goes to S108.
S108: check the return value value of send function: if value is the length for sending data packet, TCP data is passed
It is finished into, goes to S102 and other tasks is waited to trigger;If value fails less than 0 (such as -1, -2 etc.), then TCP data transmission,
It will send after fail count count_fail adds 1 and go to S109, if value is greater than 0 and is less than the length for sending data packet,
TCP data transmission does not complete, and goes to S110.
S109: judging whether count_fail is less than server number of units (in the present embodiment server number of units for 3), if so,
S103 is gone to after the common mark of descriptor effective in S107 is set to -1 (down state), otherwise goes to S102.
S110: judging whether value is less than single DPI maximum transmitted byte number (such as 41024 bytes), if so,
Obstruction is sent, will send after obstruction counting count_block adds 1 and go to S111, and otherwise show that the data volume to be sent will be greater than DPI
Transmitting capacity, need to send several times, continue to execute S107.
S111: judge count_block whether be less than N (N is positive integer, and specific value is empirically determined according to testing, this
N is defined as 3 in embodiment, shows that the server load of TCP data transmission is heavier, needs load balancing), if so, re-executing
S107;Otherwise S103 is gone to after the common mark of descriptor effective in S107 being set to 2.
The detailed process of S105 are as follows:
S105a: the ip for defining server in deep-packet detection DPI system is x, and port y used in server is ipxy (y
According to reporting the type of service of business to select, every kind of type of service corresponds to different Service-Ports), each ipxy difference is unique
The total duration ipxy.duration that preceding 2 report cycles of a corresponding record (assuming that each report cycle is 5min) are sent
[2], TCP sends occupied total duration in 10 minutes report cycles before ipxy.duration [0] corresponding current time,
TCP sends occupied total duration in 5 minutes report cycles before ipxy.duration [1] corresponding current time.
The initial value of ipxy.duration [0] and ipxy.duration [1] are 0.
S105b: respectively to currently reporting each ipxy carried in business to create TCP connection, socket is stored in each
The descriptor of ipxy connection.
S105c: the transmission scoring ipxyscore of each ipxy, calculation formula are as follows: ipxyscore=are calculated
ipxy.duration[0]·α+ipxy.duration[1]·β;Alpha+beta=1 in above-mentioned formula, for different user α and β
Ratio might have difference, by saving convergence platform test to certain, obtain best practices value α to approach being 0.3, β approach being 0.7.
S105d: judging whether all ipxyscore are identical, if so, the clothes of selection ipxy.duration [1] minimum value
Be engaged in device ip, otherwise selects the server ip of minimum value ipxyscore_min in all ipxyscore.
S105e: determining the destination address that the server ipx selected and port y send for this, by the ipxy connection of selection
Common mark be set to 1, after the descriptor of other ipxy connections is set to 0, save to corresponding TCP file connect descriptor list
In.
It includes that general class reports TASK1 and flux and flow direction class to report TASK2, TASK1 corresponding end that the present embodiment, which reports business,
1, TASK2 of mouth corresponding ports 2, number of servers 3, ip is respectively ip1, ip2 and ip3.Ip1 and port 1, ip2 and port 1,
And the ipxyscore of ip3 and port 1, respectively ip11score, ip21score and ip31score.The present embodiment is implemented
When S105, when reporting business is TASK1 and TASK1, ip11score is minimum value, therefore, the service selected in s105d
Device ip is 1.
On this basis, it after carrying out TCP data transmission in S107, needs to be updated according to the report cycle of transmission
Ipxy.duration [0] and ipxy.duration [1].
Dress shown in Figure 2, that Data Concurrent reports in the DPI system based on the above method in the embodiment of the present invention
Set, the device include deep-packet detection DPI system initialization module, report task allocating module, descriptor list selection module,
Effective descriptor determination module, TCP data transmission module, fail count determination module, obstruction determination module and obstruction count and sentence
Cover half block;
Deep-packet detection DPI system initialization module is used for: controlling depth packet detects the starting of DPI system, and initialization reports
Reporting for thread pool and reports queue length at number of threads, to reporting task allocating module transmission that task is reported to distribute signal;
It reports task allocating module to be used for: receiving after reporting task to distribute signal, control reports on all in thread pool
Report thread enters wait state, when reporting task to be added to task when reporting queue, will be reported to distribute to reporting in thread pool
Idle reports thread, sends descriptor list selection signal to descriptor list selection module;
Descriptor list selection module is used for: after receiving descriptor list selection signal, according to the service class for reporting task
Type, selects corresponding TCP file connection descriptor list, and Xiang Youxiao descriptor determination module sends effective descriptor and determines letter
Number;
Effective descriptor determination module is used for: after receiving effective descriptor judgement signal, judging that TCP file connects descriptor
It whether there is effective descriptor in list, if so, sending TCP data to TCP data transmission module transmits signal;Otherwise basis
History sends state, selects suitable descriptor and commonly used to be saved after mark is set to 1 to the connection of corresponding TCP file to retouch
It states in symbol list, sends descriptor list selection signal to descriptor list selection module again;
TCP data transmission module is used for: after receiving TCP data transmission signal, send letter is called from effective descriptor
Number carries out TCP data transmission;Check the return value value of send function: if value is the length for sending data packet, TCP
Data are transmitted, and report task to distribute signal to reporting task allocating module to send;If value, less than 0, TCP data passes
Defeated failure will send and send fail count judgement signal to fail count determination module after fail count count_fail adds 1, if
Value is greater than 0 and is less than the length for sending data packet, then TCP data transmission does not complete, and sends obstruction to obstruction determination module and sentences
Determine signal;
Fail count determination module is used for: judging whether count_fail is less than server number of units, if so, by described effective
The common mark of descriptor be set to after down state -1 to descriptor list selection module and send descriptor list selection letter
Number, otherwise to report task allocating module transmission report task distribute signal;
Obstruction determination module is used for: after receiving obstruction judgement signal, judging whether value is less than single DPI maximum transmitted
Byte number will send obstruction and count to count determination module to obstruction after count_block adds 1 and send and hinder if so, sending obstruction
Plug, which counts, determines signal, otherwise continues to send TCP data transmission signal to TCP data transmission module;
Obstruction counts determination module and is used for: judging whether count_block is less than positive integer N, if so, again to TCP number
TCP data, which is sent, according to transmission module transmits signal;Otherwise after the common mark of the effective descriptor being set to blocked state 2
Descriptor list selection signal is sent to descriptor list selection module.
The present invention is not limited to the above-described embodiments, for those skilled in the art, is not departing from
Under the premise of the principle of the invention, several improvements and modifications can also be made, these improvements and modifications are also considered as protection of the invention
Within the scope of.The content being not described in detail in this specification belongs to the prior art well known to professional and technical personnel in the field.