CN109428774A - A kind of data processing method and relevant DPI equipment of DPI equipment - Google Patents
A kind of data processing method and relevant DPI equipment of DPI equipment Download PDFInfo
- Publication number
- CN109428774A CN109428774A CN201710725583.8A CN201710725583A CN109428774A CN 109428774 A CN109428774 A CN 109428774A CN 201710725583 A CN201710725583 A CN 201710725583A CN 109428774 A CN109428774 A CN 109428774A
- Authority
- CN
- China
- Prior art keywords
- record
- target user
- information
- data
- traffic flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/026—Capturing of monitoring data using flow identification
Abstract
The present embodiments relate to data processing field more particularly to the data processing methods and relevant DPI equipment of a kind of DPI equipment, for reducing the amount of storing data.In the embodiment of the present invention, obtains traffic flow information and determine the target user of traffic flow information；Determine the data flow record of target user；It is recorded if it is determined that existing in data flow record with the customer attribute information of traffic flow information consistent first, then updates the statistical attribute information in the first record according to traffic flow information.Due in the embodiment of the present invention, when determining in the data flow record of target user in the presence of the first record consistent with the user property of the traffic flow information got, the statistical attribute information in the first record is updated according to traffic flow information, the data for not needing will acquire are stored, and the data volume of storage is reduced；The data flow record of target user is using user as storage cell, and the quantity of user is much smaller than the quantity of data flow in the entire network；Therefore, it is further reduced the data volume of storage.
Description
Technical field
The present embodiments relate to data processing field more particularly to a kind of data processing methods and correlation of DPI equipment
DPI equipment.
Background technique
In recent years, the scale of network constantly expands, and the number of users of network is continuously increased, and network application and type of business are continuous
It is abundant, thus also bring the inconvenience to network analysis.How efficient analysis Network status and processing network crisis, quickly
User behavior, mining data value etc. are perceived, the important problem of network today analysis is become.
Currently, a kind of common network traffic data monitoring method is deep-packet detection (Deep Packet
Inspection, referred to as: DPI) technology, it is a kind of 7 layer protocols analysis, in addition to 4 layers or less (MAC Address, IP layers, transmission
Layer) data service analyzed outside, also add application layer analysis (application layer protocol, payload content etc.), the company of data packet
Connect state etc.；Various application types can be identified, as operator to a kind of supplementary means of network flow monitoring.Pass through DPI
Equipment can preserve link information, packet information, data analysis result information etc., and show in World Wide Web
(World Wide Web, abbreviation WEB).
DPI equipment often saves these information using database in the prior art, by the corresponding data flow of these information
It is saved by the way of being inserted into one by one.However, web database technology is more and more huger, DPI equipment saves with the development of network
Content is more, and the efficiency that will cause the preservation of DPI equipment is lower, and can not also accomplish in terms of business fine.Especially for big
Type network environment, using multiple network interface cards 10Gb, data flow reaches millions of ranks per second, if DPI equipment using database by
Inserted mode saves, and the data volume for needing to save is also quite huge, and needs to occupy a large amount of memory spaces.
Summary of the invention
The embodiment of the invention provides a kind of data processing method of DPI equipment and relevant DPI equipment, for reducing
The data volume of DPI equipment storage.
The embodiment of the present invention provides a kind of data processing method of DPI equipment, comprising: obtains traffic flow information and determines institute
State the target user of traffic flow information；Obtain the data flow record of the target user；If it is determined that being deposited in the data flow record
With consistent first record of the customer attribute information of the traffic flow information, then update described the according to the traffic flow information
Statistical attribute information in one record.
Optionally, however, it is determined that there is no the first record is stated in the data flow record, then existed according to the traffic flow information
Increase the second record in the data flow record of the target user newly, the statistical attribute information of second record is according to the data
The statistical attribute information in record is flowed to determine.
Optionally, the target user of the determination traffic flow information, comprising: according to network interface card type and default accordingly
Condition determines the target user of the traffic flow information；Wherein, the preset condition includes: to determine that the network interface card type is
In the case where uplink/downlink double netcard, institute will be determined as by the source network Protocol IP address in the traffic flow information of uplink network interface card
Target user is stated, target user will be determined as by the purpose IP address in the traffic flow information of downlink network interface card；Described in determination
In the case that network interface card type is Single NIC: if it is determined that the source IP address in the traffic flow information is net where the target user
Section in any IP address when, it is determined that the source IP address be target user；If it is determined that the purpose in the traffic flow information
IP address is any IP address in the network segment of the target user place, it is determined that the destination IP address is target user；It is described
Source IP address and the destination IP address be not in same network segment.
Optionally, after the statistical attribute information updated according to the traffic flow information in first record, also
It include: the target data stream record obtained from the data flow of target user record in statistical time section；For the mesh
At least one customer attribute information in data flow record is marked, executes: determining the corresponding statistical attribute of the customer attribute information
Accounting of the information in the summation for the statistical attribute information that the target data stream records.
Optionally, in preset period of time, the data flow record that will be stored in the target user of memory imports database；Its
In, the preset period of time is the period that network flow is lower than flow threshold.
The embodiment of the present invention provides a kind of DPI equipment for data processing, comprising: memory module, for storing each use
The data flow at family records, and records in the data flow record of each user comprising a plurality of data flow, the user of pieces of data stream record
Attribute information is not exactly the same；Processing module, for obtaining the target user of traffic flow information and the determining traffic flow information；
The data flow record of the target user is obtained from the memory module；If it is determined that exist in data flow record with it is described
Consistent first record of the customer attribute information of traffic flow information, then update in first record according to the traffic flow information
Statistical attribute information.
Optionally, the processing module, is also used to: if it is determined that there is no the first record is stated in data flow record, then
The second record, the statistics of second record are increased newly in the data flow record of the target user according to the traffic flow information
Attribute information is determined according to the statistical attribute information in data flow record.
Optionally, the processing module, is used for: determining that the data flow is believed according to network interface card type and corresponding preset condition
The target user of breath；Wherein, the preset condition include: determine the network interface card type be uplink/downlink double netcard in the case where,
It will be determined as the target user by the source network Protocol IP address in the traffic flow information of uplink network interface card, downlink net will be passed through
Purpose IP address in the traffic flow information of card is determined as target user；Determining the case where network interface card type is Single NIC
Under: if it is determined that when the source IP address in the traffic flow information is any IP address where the target user in network segment, then
Determine that the source IP address is target user；If it is determined that the purpose IP address in the traffic flow information is the target user institute
Any IP address in network segment, it is determined that the destination IP address is target user；The source IP address is with the destination IP
Location is not in same network segment.
Optionally, the processing module, is also used to: obtaining statistical time section from the data flow of target user record
Interior target data stream record；For at least one customer attribute information in target data stream record, executes: determining institute
The corresponding statistical attribute information of customer attribute information is stated in the summation for the statistical attribute information that the target data stream records
Accounting.
Optionally, processing module is also used to: in preset period of time, by the number of the target user stored in memory module
It is recorded according to stream and imports database；Wherein, the preset period of time is the period that network flow is lower than flow threshold.
Due in the embodiment of the present invention, getting the target user of traffic flow information and determining traffic flow information, in determination
When there is the first record consistent with the user property of the traffic flow information got in the data flow record of target user, according to
The traffic flow information update it is described first record in statistical attribute information, update be target user data stream statistics category
Property information, do not need to carry out existing subscriber's information and traffic flow information to repeat storage, and then reduce the data volume of storage；And
And it is directed to the data flow record of target user, it is using user as storage cell, the quantity of user is far small in the entire network
In the quantity of data flow；It therefore, is that search efficiency can be improved in index with user.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly introduced.
Fig. 1 is a kind of configuration diagram of communication system provided in an embodiment of the present invention；
Fig. 2 is a kind of data processing method flow diagram of DPI equipment provided in an embodiment of the present invention；
Fig. 3 is the data processing method flow diagram of another kind DPI equipment provided in an embodiment of the present invention；
Fig. 4 is the data processing method flow diagram of another kind DPI equipment provided in an embodiment of the present invention；
Fig. 5 is a kind of structural schematic diagram of data processing equipment provided in an embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.
Fig. 1 shows a kind of configuration diagram of communication system using the embodiment of the present invention.As shown in Figure 1, the system
Framework may include client 101, server 102 and data processing equipment, and data processing equipment includes DPI equipment, this hair
It is discussed for when data processing equipment is DPI equipment 103 in bright embodiment.DPI equipment can be set network exit,
Near interchanger, the positions such as in router attachment or router, convenient for obtaining the number communicated between client and server
According to stream.
Client 101 can be through wireless access network (Radio Access Network, abbreviation RAN) and one or more cores
The terminal device that heart net is communicated can refer to user equipment (User Equipment, abbreviation UE), access terminal, Yong Hudan
Member, subscriber station, movement station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless telecom equipment,
User agent or user apparatus.Access terminal can be cellular phone, wireless phone, session initiation protocol (Session
Initiation Protocol, abbreviation SIP) phone, wireless local loop (Wireless Local Loop, abbreviation WLL) stand,
Personal digital assistant (Personal Digital Assistant, abbreviation PDA), the handheld device with wireless communication function,
It calculates equipment or is connected to other processing equipments of radio modem, mobile unit, wearable device, in the following 5G network
Terminal device etc..
Server 102 can be any server communicated with client 101.
DPI equipment 103 is connected between client 101 and server 102.DPI equipment 103, which can be, is connected to client
Between corresponding router and the router of server, the flow direction of data flow is after sending out from client, by client
Router is transmitted to DPI equipment, then data flow is transmitted to server by the corresponding router of server by DPI equipment.DPI
After equipment gets traffic flow information, traffic flow information is analyzed, and under the preservation of analysis result, waiting when needed can
To be shown on web.Memory space includes memory and disk in DPI equipment, is saved mostly in the form of database in disk；Cause
This, analysis result can be saved in memory or be saved in the database of disk.DPI equipment 103 includes logging modle
103a, analysis detection module 103b, display module 103c.When data flow passes through DPI equipment, logging modle 103a records the number
According to the connection tracking information of stream, including five-tuple information: source IP address, purpose IP address, destination port, source port, transport layer；
Analysis detection module 103b is sent by data flow later, analysis detection module is when receiving data flow, to the number received
Flow analysis, protocol detection processing are done according to stream, that is, determines the application layer of uplink and downlink flow and/or total flow and data flow
Information, and the result after analysis is sent to data recordin module 103a；Pass through the logging modle 103a of DPI equipment, analysis
Detection module 103b can obtain traffic flow information to DPI equipment and parse, and determine that the corresponding target of traffic flow information is used
Family, customer attribute information and statistical attribute information.Display module 103c is obtained when needing display data stream information from backstage
Traffic flow information is shown, for example the statistical result of user property letter can be shown on web.It can be in the embodiment of the present invention
More separate threads are set up for DPI equipment, are tied on CPU, it is parallel to execute, the operational efficiency of DPI equipment, and DPI can be improved
Logging modle, analysis detection module in equipment, display module independent operating are independent of each other, it is ensured that the stabilization of DPI equipment operation
Property.
Based on system architecture shown in FIG. 1, Fig. 2 illustrates a kind of DPI equipment provided in an embodiment of the present invention
Data processing method flow diagram, as shown in Fig. 2, the data processing method of the DPI equipment the following steps are included:
Step 201, it obtains traffic flow information and determines the target user of the traffic flow information；
Step 202, the data flow record of the target user is obtained；
Step 203, however, it is determined that exist in the data flow record consistent with the customer attribute information of the traffic flow information
First record, then according to the traffic flow information update it is described first record in statistical attribute information.
Due in the embodiment of the present invention, getting the target user of traffic flow information and determining traffic flow information, in determination
When there is the first record consistent with the user property of the traffic flow information got in the data flow record of target user, according to
The traffic flow information update it is described first record in statistical attribute information, update be target user statistical attribute letter
Breath does not need to carry out repeating storage to existing subscriber's information and traffic flow information, and then reduces the data volume of storage；Moreover,
It is directed to the data flow record of target user, is using user as storage cell, the quantity of user is much smaller than in the entire network
The quantity of data flow；It therefore, is that search efficiency can be improved in index with user.
In the embodiment of the present invention, customer attribute information includes: source IP address, purpose IP address, user's application type, user
Any one of uniform resource locator (Uniform Resoure Locator, abbreviation URL), destination port, source port or
Appoint multinomial；Statistical attribute information includes: any one of uplink and downlink flow and/or total flow, uplink and downlink rate, online hours
Or appoint multinomial.
In the embodiment of the present invention, the optional method of the target user of determining traffic flow information a kind of is provided: according to network interface card
Type and corresponding preset condition determine the target user of the traffic flow information；Wherein, the preset condition includes: in determination
In the case that the network interface card type is uplink/downlink double netcard, the source network agreement in the traffic flow information of uplink network interface card will be passed through
IP address is determined as the target user, uses target is determined as by the purpose IP address in the traffic flow information of downlink network interface card
Family；In the case where determining the network interface card type is Single NIC: if it is determined that the source IP address in the traffic flow information is described
When any IP address in network segment where target user, it is determined that the source IP address is target user；If it is determined that the data
Purpose IP address in stream information is any IP address in the network segment of the target user place, it is determined that the destination IP address is
Target user；The source IP address and the destination IP address be not in same network segment.If source IP address or purpose IP address
In the matched IP address of network segment not corresponding with the target user, then be abandon the data flow.
In the embodiment of the present invention, after the target user for determining the traffic flow information, judge whether the target user deposits
If the target user is not present, the corresponding data flow record of the target user is being established, is including user in data flow record
Attribute information and statistical attribute information.If the target user exists, judge in data flow record with the presence or absence of with it is described
Consistent first record of the customer attribute information of traffic flow information, and if it exists, then update described the according to the traffic flow information
Statistical attribute information in one record；If it is determined that there is no the first record is stated in the data flow record, then according to the data
Stream information the target user data flow record in increase newly second record, it is described second record statistical attribute information according to
Statistical attribute information in data flow record determines.
Whether the traffic flow information that a kind of determining acquisition is provided in the embodiment of the present invention, which is stored in the traffic flow information, corresponds to
Target user data flow record in optional way, it is as follows:
Determine five metamessages of the traffic flow information obtained；Wherein, customer attribute information includes five-tuple information, according to obtaining
The five-tuple information of the data flow taken determines the data flow record of the target user with the presence or absence of the data flow obtained；Five-tuple letter
Breath can uniquely identify a data stream information.Therefore, therefore according to five-tuple information can accurately judge to receive
Data flow whether there is in the data flow record of the user.
In order to make it easy to understand, below with reference to a specific embodiment to the data processing method of DPI equipment furtherly
It is bright.Optionally, determine that the mode of statistical attribute information includes a variety of in the embodiment of the present invention, such as: summing after summation, weighting,
Certain multiple etc. is simultaneously amplified or reduced, is specifically determined according to the actual needs.Assuming that being stored with target user A and target
The customer attribute information of the data flow record of user B such as table 1, target user A and target user B include: user's application, destination IP
Address；Statistical attribute information includes: uplink and downlink flow, total flow.
The data flow of table 1, target user A and target user B records
Assuming that the data flow got is data flow 1, the target user of data flow 1 is determined, however, it is determined that the target of data flow 1
User is A, it is determined that the data flow of target user records, such as the corresponding data flow of target user A in table 1；Determine data flow 1
Customer attribute information and statistical attribute information.
Situation one: assuming that the customer attribute information for the data flow 1 determined are as follows: user's application is 360 search, destination IP
Location is IP2, uplink traffic is 1.5M, and downlink traffic is 20M, total flow 21.5M；Then determine the user property letter of data flow 1
It ceases data flow record corresponding with 360 search in the data flow of target user record to be consistent, by the number of target user
It is known as the first record according to the corresponding data flow record of 360 search in stream record, then institute is updated according to the traffic flow information
The statistical attribute information in the first record is stated, i.e., the uplink traffic in the first record is updated to 2.5M, downlink traffic is updated to
30M, total flow are updated to 32.5M；The data flow of the data stream 1 is recorded after update and is abandoned.
Situation two: assuming that the customer attribute information for the data flow 1 determined are as follows: user's application is QQ whirlwind, destination IP
Location is IP8, uplink traffic is 1M, and downlink traffic is 20M, total flow 21M；Then determine the data flow record of target user A not
It is recorded in the presence of with the customer attribute information of the traffic flow information consistent first, then according to the traffic flow information in the mesh
Increase the second record, the customer attribute information and the traffic flow information of second record in the data flow record of mark user newly
Customer attribute information is consistent, and the statistical attribute information of second record is according to the statistical attribute information in data flow record
It asks by way of summation and determines；The data flow record of target user A after the second newly-increased record is as shown in table 2.
The data flow record of target user A after table 2, newly-increased second record
In the embodiment of the present invention, by the rational design to the structure of storing data in DPI equipment, using target user as base
This storage cell is stored for the traffic flow information of each target user with customer attribute information and statistical attribute information.
In storage, the traffic flow information of target user is subjected to duplicate removal and updates the statistical attribute information of target user.In this way may be used
Largely to reduce the preservation quantity of data.Above method process is introduced in order to clearer, below for the data of acquisition
For when stream is 1 and N, illustrate respectively.
In the embodiment of the present invention, obtaining traffic flow information includes obtaining periodically or in real time traffic flow information, acquisition
Traffic flow information can be 1 or N item, and N is the integer greater than 1.
Fig. 3 illustrates the data processing method of another kind DPI equipment provided by the invention.It is obtained in the embodiment
Data flow be 1, as shown in figure 3, the storage method of data includes:
Step 301,1 data stream information is obtained；
Step 302, the target user of the data flow is determined；
Step 303, judge that the target user of the data flow whether there is in the memory of DPI equipment；If it exists, then it executes
Step 304；If it does not exist, 308 are thened follow the steps；
Step 304, the data flow record of the target user is determined；
Step 305, judge in the data flow record of target user with the presence or absence of the user property one with the traffic flow information
The first record caused, and if it exists, then follow the steps 306；If it does not exist, 307 are thened follow the steps；
Step 306, the statistical attribute information in first record is updated according to the traffic flow information；
Step 307, the second record, institute are increased newly in the data flow record of the target user according to the traffic flow information
The customer attribute information for stating the second record is consistent with the customer attribute information of the traffic flow information, the statistics of second record
Attribute information is determined according to the statistical attribute information in data flow record；
Step 308, the target user is established；
Step 309, corresponding data flow record is established under the target user of foundation.
After establishing the data flow of the user, to DPI equipment application memory headroom；Establish the data flow record of the target user
Afterwards, the storing process of data flow is same as mentioned above, and details are not described herein.
The method that Fig. 4 illustrates another data processing provided by the invention.The data obtained in the embodiment
Stream is N item, and N is the integer greater than 1；As shown in figure 4, the storage method of data includes:
Step 401, N traffic flow information is obtained；
Step 402, the target user of this N data stream information is determined；
In the embodiment of the present invention for ease of description, illustrate so that the target user of N data stream information is identical as an example；Such as
The target user of the traffic flow information of this N item of fruit at least 1 not identical, then the process executed, which combines, obtains data stream letter
Breath and two kinds of storage modes of N data stream information are available；
Step 403, judge that the target user whether there is in the memory of DPI equipment；If it exists, 404 are thened follow the steps；
If it does not exist, 408 are thened follow the steps；
Step 404, the data flow record of the target user is determined；
Step 405, it clusters customer attribute information is consistent in the N data stream information, obtains at least one collection
Group believes for the statistical attribute of the corresponding cluster of each statistical attribute information update in each cluster at least one cluster
Breath；
Wherein, customer attribute information includes that source IP address, purpose IP address, user's application type, user's unified resource are fixed
Any one of position device (Uniform Resoure Locator, abbreviation URL), destination port, source port are appointed multinomial；System
Meter attribute information includes: any one of uplink and downlink flow and/or total flow or appoints multinomial；
Step 406, for each cluster at least one cluster, judge that the data flow record of target user whether there is
With consistent first record of customer attribute information of the cluster；If it exists, 407 are thened follow the steps；If it does not exist, it thens follow the steps
408；
Step 407, the first note according to the statistical attribute information update in the corresponding traffic flow information of the described cluster
Statistical attribute information in record；
Step 408, the is increased newly in the data flow record of the target user according to the corresponding traffic flow information of the cluster
The customer attribute information of two records, second record is consistent with the customer attribute information of traffic flow information in the cluster,
The statistical attribute information of second record is determined according to the statistical attribute information in data flow record；
Step 409, the target user is established；
The target user based on foundation successively executes step 404 to 408 again later.
The storing process of data flow is same as mentioned above, and details are not described herein.
Optionally, when getting N data stream information, can also first determine every data stream whether there is with it is corresponding
In the data flow record of target user, clustered again for the data flow remained.
In the prior art, it generally requires to use classification, sequence when display data.For example wonder some clothes of access
It is engaged in the traffic conditions of device, needs to be traversed for the data flow record of all preservations, find out and all reach all of this server
Then data flow is ranked up, to obtain a result.If wanting to check current network user access data flow amount ranking, need to return
It receives out the total flow of each user, is then ranked up.In the case where data flow is huge, entire query process can be very slow,
Bring poor user experience.In the embodiment of the present invention, for the ease of showing the data of DPI equipment storage, a kind of specific reality is provided
Existing mode: after the statistical attribute information updated according to the traffic flow information in first record, further includes: determine
The target data stream of the corresponding target user of object time granularity records；Wherein, the target data stream record includes extremely
A few customer attribute information, the data flow record include that the target data stream records；Time granularity includes minute, hour
Or any one of day；It for each of at least one customer attribute information customer attribute information, executes: described in determining
The accounting for the statistical attribute information that the corresponding statistical attribute information of customer attribute information includes in target data stream record.
In the embodiment of the present invention, with the corresponding statistical attribute letter of the customer attribute information clearly fixed for specific example
Cease the accounting for the statistical attribute information for including in target data stream record.Assuming that object time granularity is minute, with 10
For minute；As shown in table 3, the user in the embodiment belongs to the target data stream record of the target user determined in 10 minutes
By taking user applies as an example, statistical attribute information illustrates property information by taking user's total flow as an example.
The target data stream of target user in table 3,10 minutes records
| Number | User's application | Connection number | Uplink traffic | Downlink traffic | Total flow |
| 1 | A sudden peal of thunder | 2 | 0.5M | 2M | 2.5 |
| 2 | 360 search | 10 | 1M | 10M | 11M |
| 3 | HTTP | 25 | 30M | 60M | 90M |
| 4 | Tencent | 1 | 0 | 0 | 0 |
The percentage that the total flow of a sudden peal of thunder accounts for the total flow that target data includes in user's application in 10 minutes is 2.5/
(2.5+11+90+0)=2.4%, the total flow of HTTP accounts for the percentage of the total flow that target data includes and is in user's application
90/ (2.5+11+90+0)=86.7%；The similarly accounting of available other application.
The accounting situation of other users attribute information can also be determined in the embodiment of the present invention, and statistical attribute information can
To be uplink and downlink flow, the others statistical attribute information such as total flow or online hours.Use is determined according to specific needs
The occurrence of family attribute information and statistics of attributes information.
It, can be by different customer attribute informations according to total in order to meet the displaying of display module in the embodiment of the present invention
Flow carries out ranking, for example will carry out ranking to user's application type in customer attribute information, arranges user URL
Name, ranking is carried out to purpose IP address corresponding server, can also be according to the total flow of target user to target user
Carry out ranking.It in this way, the result for accelerating DPI equipment is shown, and can quickly obtain the traffic conditions of user, promote user's body
It tests.
A kind of realization side that the data flow record by DPI equipment imports in database is provided in the embodiment of the present invention
Formula, i.e., in preset period of time, the data flow record that will be stored in the target user of memory imports database；Wherein, described pre-
If the period is the period that network flow is lower than flow threshold.Since the embodiment of the present invention is by being that basic storage is single with target user
Position carries out the operation such as duplicate removal, polymerization, update to the data flow of target user, largely reduces and deposit several data volumes, because
This, the quantity for importing database also consequently reduces, in this way, can not only save the time on write-in data road, but also can save
The expense of disk.
It,, can be in the importing number library for the property of can choose in the database for importing data to disk in the embodiment of the present invention
It is further reduced the data volume of write-in database.
Optionally, when preset period of time can be end in one day, such as 23: 50 to 24 points；It is saved in preset period of time
The data flow on the same day records, and network flow is lower than flow threshold, i.e., network flow at this time is smaller, and the requirement to equipment is not
The probability of original text, error is small.
Further, only in one day preset period of time by the data for the target user being stored between DPI device memory
Stream record imports the database of disk, does not need the operation for frequently carrying out database, facilitates the operation for promoting DPI equipment
Stability, and help avoid reducing the operational efficiency of DPI equipment.
It can be seen from the above: in the embodiment of the present invention, due to getting traffic flow information in the embodiment of the present invention
And determine the target user of traffic flow information, exist in the data flow record for determining target user and believes with the data flow got
When the user property of breath consistent first records, the statistical attribute in first record is updated according to the traffic flow information and is believed
Breath, update be target user statistical attribute information, do not need to carry out existing subscriber's information and traffic flow information to repeat to deposit
Storage, and then reduce the data volume of storage；It is single for storage with user moreover, being directed to the data flow record of target user
Position, the quantity of user is much smaller than the quantity of data flow in the entire network；It therefore, is that inquiry effect can be improved in index with user
Rate.
Based on same idea, Fig. 5 is that the embodiment of the invention provides a kind of structural schematic diagrams of data processing equipment, is such as schemed
Shown in 5, which includes memory module 501, processing module 502.Wherein:
Memory module, the data flow for storing each user record, and include a plurality of number in the data flow record of each user
It is recorded according to stream, the customer attribute information of pieces of data stream record is not exactly the same；Processing module, for obtaining traffic flow information simultaneously
Determine the target user of the traffic flow information；The data flow record of the target user is obtained from the memory module；If
It determines to exist in the data flow record and be recorded with the customer attribute information of the traffic flow information consistent first, then according to institute
State traffic flow information update it is described first record in statistical attribute information.
Optionally, the processing module, is also used to: if it is determined that there is no the first record is stated in data flow record, then
The second record, the statistics of second record are increased newly in the data flow record of the target user according to the traffic flow information
Attribute information is determined according to the statistical attribute information in data flow record.
Optionally, the processing module, is used for: determining that the data flow is believed according to network interface card type and corresponding preset condition
The target user of breath；Wherein, if the preset condition include: network interface card type be uplink/downlink double netcard when, uplink network interface card will be passed through
Traffic flow information in source network Protocol IP address be determined as the target user, will be by the traffic flow information of downlink network interface card
In purpose IP address be determined as target user；If network interface card type be Single NIC, by the traffic flow information source IP or
It is determined as target user with the matched IP address of network segment where the target user in destination IP.
Optionally, the processing module, is also used to: obtaining statistical time section from the data flow of target user record
Interior target data stream record；For at least one customer attribute information in target data stream record, executes: determining institute
The corresponding statistical attribute information of customer attribute information is stated in the summation for the statistical attribute information that the target data stream records
Accounting.
Optionally, processing module is used for: in preset period of time, by the data of the target user stored in memory module
Stream record imports database；Wherein, the preset period of time is the period that network flow is lower than flow threshold.
It can be seen from the above: in the embodiment of the present invention, due to getting traffic flow information in the embodiment of the present invention
And determine the target user of traffic flow information, exist in the data flow record for determining target user and believes with the data flow got
When the user property of breath consistent first records, the statistical attribute in first record is updated according to the traffic flow information and is believed
Breath, update be target user statistical attribute information, do not need to carry out existing subscriber's information and traffic flow information to repeat to deposit
Storage, and then reduce the data volume of storage；It is single for storage with user moreover, being directed to the data flow record of target user
Position, the quantity of user is much smaller than the quantity of data flow in the entire network；It therefore, is that inquiry effect can be improved in index with user
Rate.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method or computer program product.
Therefore, complete hardware embodiment, complete software embodiment or embodiment combining software and hardware aspects can be used in the present invention
Form.It is deposited moreover, the present invention can be used to can be used in the computer that one or more wherein includes computer usable program code
The shape for the computer program product implemented on storage media (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
Formula.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The equipment for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of equipment, the commander equipment realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to include these modifications and variations.
Claims (12)
1. a kind of data processing method of deep-packet detection DPI equipment characterized by comprising
It obtains traffic flow information and determines the target user of the traffic flow information；
Obtain the data flow record of the target user；
It is recorded if it is determined that existing in the data flow record with the customer attribute information of the traffic flow information consistent first, then
The statistical attribute information in first record is updated according to the traffic flow information.
2. the method as described in claim 1, which is characterized in that further include:
If it is determined that then being used according to the traffic flow information in the target in the data flow record there is no first record
Increase the second record in the data flow record at family newly, the statistical attribute information of second record is according in data flow record
Statistical attribute information determines.
3. the method as described in claim 1, which is characterized in that the target user of the determination traffic flow information, comprising:
The target user of the traffic flow information is determined according to network interface card type and corresponding preset condition；Wherein, the default item
Part includes:
It, will be by the traffic flow information of uplink network interface card in the case where determining the network interface card type is uplink/downlink double netcard
Source network Protocol IP address is determined as the target user, will be true by the purpose IP address in the traffic flow information of downlink network interface card
It is set to target user；
In the case where determining the network interface card type is Single NIC: if it is determined that the source IP address in the traffic flow information is described
When any IP address in network segment where target user, it is determined that the source IP address is target user；If it is determined that the data
Purpose IP address in stream information is any IP address in the network segment of the target user place, it is determined that the destination IP address is
Target user；The source IP address and the destination IP address be not in same network segment.
4. the method as described in claim 1, which is characterized in that described to update first record according to the traffic flow information
In statistical attribute information after, further includes:
The target data stream record in statistical time section is obtained from the data flow of target user record；
For at least one customer attribute information in target data stream record, execute:
Determine the statistical attribute information that the corresponding statistical attribute information of the customer attribute information is recorded in the target data stream
Summation in accounting.
5. such as method of any of claims 1-4 characterized by comprising
In preset period of time, the data flow record that will be stored in the target user of memory imports database；Wherein, described default
Period is the period that network flow is lower than flow threshold.
6. a kind of DPI equipment for data processing characterized by comprising
Memory module, the data flow for storing each user record, and include a plurality of data flow in the data flow record of each user
The customer attribute information of record, pieces of data stream record is not exactly the same；
Processing module, for obtaining the target user of traffic flow information and the determining traffic flow information；From the memory module
The middle data flow record for obtaining the target user；If it is determined that there is the use with the traffic flow information in the data flow record
Attribute information consistent first record in family then updates the statistical attribute in first record according to the traffic flow information and believes
Breath.
7. equipment as claimed in claim 6, which is characterized in that the processing module is also used to:
If it is determined that there is no the first record is stated in the data flow record, then according to the traffic flow information in the target user
Data flow record in increase newly second record, it is described second record statistical attribute information according to the data flow record in system
Attribute information is counted to determine.
8. equipment as claimed in claim 6, which is characterized in that the processing module is used for:
The target user of the traffic flow information is determined according to network interface card type and corresponding preset condition；Wherein, the default item
Part includes:
It, will be by the traffic flow information of uplink network interface card in the case where determining the network interface card type is uplink/downlink double netcard
Source network Protocol IP address is determined as the target user, will be true by the purpose IP address in the traffic flow information of downlink network interface card
It is set to target user；
In the case where determining the network interface card type is Single NIC: if it is determined that the source IP address in the traffic flow information is described
When any IP address in network segment where target user, it is determined that the source IP address is target user；If it is determined that the data
Purpose IP address in stream information is any IP address in the network segment of the target user place, it is determined that the destination IP address is
Target user；The source IP address and the destination IP address be not in same network segment.
9. equipment as claimed in claim 6, which is characterized in that the processing module is also used to:
The target data stream record in statistical time section is obtained from the data flow of target user record；
For at least one customer attribute information in target data stream record, execute:
Determine the statistical attribute information that the corresponding statistical attribute information of the customer attribute information is recorded in the target data stream
Summation in accounting.
10. the equipment as described in any one of claim 6-9, which is characterized in that the processing module is also used to:
In preset period of time, the data flow record of the target user stored in the memory module is imported into database；Wherein,
The preset period of time is the period that network flow is lower than flow threshold.
11. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer can
It executes instruction, the computer executable instructions are for requiring the computer perform claim described in 1 to 5 any claim
Method.
12. a kind of computer equipment characterized by comprising
Memory module, for storing program instruction；
Processing module is executed according to the program of acquisition as right is wanted for calling the program instruction stored in the memory module
Seek method described in 1 to 5 any claim.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710725583.8A CN109428774B (en) | 2017-08-22 | 2017-08-22 | Data processing method of DPI equipment and related DPI equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710725583.8A CN109428774B (en) | 2017-08-22 | 2017-08-22 | Data processing method of DPI equipment and related DPI equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109428774A true CN109428774A (en) | 2019-03-05 |
| CN109428774B CN109428774B (en) | 2020-12-22 |
Family
ID=65497376
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710725583.8A Active CN109428774B (en) | 2017-08-22 | 2017-08-22 | Data processing method of DPI equipment and related DPI equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109428774B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114095386A (en) * | 2020-07-01 | 2022-02-25 | 阿里巴巴集团控股有限公司 | Data stream statistical method, device and storage medium |
| CN115150171A (en) * | 2022-06-30 | 2022-10-04 | 北京天融信网络安全技术有限公司 | Flow statistical method and device, electronic equipment and storage medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1716867A (en) * | 2004-06-29 | 2006-01-04 | 杭州华为三康技术有限公司 | Data flow statistic method and device |
| CN101505276A (en) * | 2009-03-23 | 2009-08-12 | 杭州华三通信技术有限公司 | Network application flow recognition method and apparatus and network application flow management apparatus |
| CN101888303A (en) * | 2009-05-13 | 2010-11-17 | 中国移动通信集团上海有限公司 | Recording method of network traffic information and related device |
| CN102025623A (en) * | 2010-12-07 | 2011-04-20 | 苏州迈科网络安全技术股份有限公司 | Intelligent network flow control method |
| CN202696628U (en) * | 2012-07-16 | 2013-01-23 | 北京国创富盛通信股份有限公司 | Network optimization system |
| CN103051725A (en) * | 2012-12-31 | 2013-04-17 | 华为技术有限公司 | Application identification method, data mining method, device and system |
| CN103916294A (en) * | 2014-04-29 | 2014-07-09 | 华为技术有限公司 | Identification method and device for protocol type |
| CN104243237A (en) * | 2014-09-17 | 2014-12-24 | 杭州华三通信技术有限公司 | P2P flow detection method and device |
| CN104486143A (en) * | 2014-12-01 | 2015-04-01 | 中国联合网络通信集团有限公司 | Deep packet inspection (DPI) method and deep packet inspection system |
| US9113400B2 (en) * | 2013-03-08 | 2015-08-18 | Tellabs Operations, Inc | Method and apparatus for offloading packet traffic from LTE network to WLAN using DPI |
| CN107241701A (en) * | 2016-03-28 | 2017-10-10 | 中国移动通信有限公司研究院 | A kind of data transmission method and device |
| CN107360062A (en) * | 2017-08-28 | 2017-11-17 | 上海国云信息科技有限公司 | Verification method, system and the DPI equipment of DPI equipment recognition results |
-
2017
- 2017-08-22 CN CN201710725583.8A patent/CN109428774B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1716867A (en) * | 2004-06-29 | 2006-01-04 | 杭州华为三康技术有限公司 | Data flow statistic method and device |
| CN101505276A (en) * | 2009-03-23 | 2009-08-12 | 杭州华三通信技术有限公司 | Network application flow recognition method and apparatus and network application flow management apparatus |
| CN101888303A (en) * | 2009-05-13 | 2010-11-17 | 中国移动通信集团上海有限公司 | Recording method of network traffic information and related device |
| CN102025623A (en) * | 2010-12-07 | 2011-04-20 | 苏州迈科网络安全技术股份有限公司 | Intelligent network flow control method |
| CN202696628U (en) * | 2012-07-16 | 2013-01-23 | 北京国创富盛通信股份有限公司 | Network optimization system |
| CN103051725A (en) * | 2012-12-31 | 2013-04-17 | 华为技术有限公司 | Application identification method, data mining method, device and system |
| US9113400B2 (en) * | 2013-03-08 | 2015-08-18 | Tellabs Operations, Inc | Method and apparatus for offloading packet traffic from LTE network to WLAN using DPI |
| CN103916294A (en) * | 2014-04-29 | 2014-07-09 | 华为技术有限公司 | Identification method and device for protocol type |
| CN104243237A (en) * | 2014-09-17 | 2014-12-24 | 杭州华三通信技术有限公司 | P2P flow detection method and device |
| CN104486143A (en) * | 2014-12-01 | 2015-04-01 | 中国联合网络通信集团有限公司 | Deep packet inspection (DPI) method and deep packet inspection system |
| CN107241701A (en) * | 2016-03-28 | 2017-10-10 | 中国移动通信有限公司研究院 | A kind of data transmission method and device |
| CN107360062A (en) * | 2017-08-28 | 2017-11-17 | 上海国云信息科技有限公司 | Verification method, system and the DPI equipment of DPI equipment recognition results |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114095386A (en) * | 2020-07-01 | 2022-02-25 | 阿里巴巴集团控股有限公司 | Data stream statistical method, device and storage medium |
| CN114095386B (en) * | 2020-07-01 | 2024-03-26 | 阿里巴巴集团控股有限公司 | Data stream statistics method, device and storage medium |
| CN115150171A (en) * | 2022-06-30 | 2022-10-04 | 北京天融信网络安全技术有限公司 | Flow statistical method and device, electronic equipment and storage medium |
| CN115150171B (en) * | 2022-06-30 | 2023-11-10 | 北京天融信网络安全技术有限公司 | Flow statistics method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109428774B (en) | 2020-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Baştuğ et al. | Big data meets telcos: A proactive caching perspective | |
| US11870649B2 (en) | Multi-access edge computing based visibility network | |
| CN104396188B (en) | System and method for carrying out basic reason analysis to mobile network property problem | |
| US9503465B2 (en) | Methods and apparatus to identify malicious activity in a network | |
| US10958546B2 (en) | System and method for estimation of quality of experience (QoE) for web browsing using passive measurements | |
| US10332005B1 (en) | System and method for extracting signatures from controlled execution of applications and using them on traffic traces | |
| CN111339436B (en) | Data identification method, device, equipment and readable storage medium | |
| CN110300084B (en) | IP address-based portrait method and apparatus, electronic device, and readable medium | |
| CN107196848B (en) | Information push method and device | |
| CN108023788A (en) | Monitoring data method for uploading, device, equipment, system and storage medium | |
| CN111148018B (en) | Method and device for identifying and positioning regional value based on communication data | |
| Balakrishnan et al. | Diverse client selection for federated learning: Submodularity and convergence analysis | |
| CN104615765A (en) | Data processing method and data processing device for browsing internet records of mobile subscribers | |
| CN111385122A (en) | Distributed system link tracking method and device, computer equipment and storage medium | |
| Bao et al. | User behavior and user experience analysis for social network services | |
| CN114513850B (en) | Positioning method, positioning device, computer equipment and medium | |
| CN109428774A (en) | A kind of data processing method and relevant DPI equipment of DPI equipment | |
| CN107371179A (en) | Measurement result report method, measurement result method of reseptance, relevant device and system | |
| CN106648722A (en) | Flume receiving side data processing method and device based on big data | |
| CN104168174A (en) | Method and apparatus for information transmission | |
| US20240022507A1 (en) | Information flow recognition method, network chip, and network device | |
| CN116578911A (en) | Data processing method, device, electronic equipment and computer storage medium | |
| CN108471387B (en) | Log flow decentralized control method and system | |
| Li et al. | Characterizing service providers traffic of mobile internet services in cellular data network | |
| WO2022001480A1 (en) | Popular application identification method, network system, network device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |