CN106230823A - A kind of flow statistical method and device - Google Patents
A kind of flow statistical method and device Download PDFInfo
- Publication number
- CN106230823A CN106230823A CN201610622351.5A CN201610622351A CN106230823A CN 106230823 A CN106230823 A CN 106230823A CN 201610622351 A CN201610622351 A CN 201610622351A CN 106230823 A CN106230823 A CN 106230823A
- Authority
- CN
- China
- Prior art keywords
- data
- thread
- statistical result
- dimension
- memory source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
- H04L63/1458—Denial of Service
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
- H04L63/1466—Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to network safety filed, particularly relate to a kind of flow statistical method and device, for while ensureing traffic statistics performance, reduce EMS memory occupation amount.The method is: the data traffic that cleaning device utilizes some threads to obtain respectively carries out counting messages, multiple threads are bundled under different physical cpus, it is respectively allocated a memory source to each physical cpu simultaneously, and respective statistical result is stored in the memory source that the physical cpu of binding is corresponding by each thread；As such, it is possible to be not necessarily each thread to distribute single internal memory, thus substantially reduce EMS memory occupation amount；On the other hand, although each thread under every a memory source remains a need for when storing respective statistical result queuing up accessing, but the cache hit rate of internal memory is high under same physical cpu, access speed is fast, therefore, even if getting rid of access not interfere with storage speed, thus it is effectively increased statistical efficiency, it is ensured that traffic statistics performance.
Description
Technical field
The present invention relates to network safety filed, particularly relate to a kind of flow statistical method and device.
Background technology
Distributed denial of service (Distributed Denial of Service, DDoS) attack has become as the tightest
One of Cyberthreat of weight.Ddos attack is a kind of based on refusal the service distribution of (Denial of Service, DoS), cooperation
Attack in force mode, it directly or indirectly through other in check computers on the Internet as puppet's machine target of attack
System, this aggressive behavior takies a large amount of Internet resources by the most legal request, to reach the purpose of paralysis network.DDoS
Attack and be mainly made up of assailant, control puppet's machine, attack puppet's machine and the person of being hacked four part, with reference to Fig. 1, control puppet's machine
It is not involved in the attack of reality, only issues manipulation order, attack puppet's machine and send actual attack bag.
In order to prevent ddos attack, generally add flow cleaning equipment, flow cleaning equipment energy in the front end of the person of being hacked
The data traffic of the person of being hacked is monitored in real time, finds in time and clean abnormal flow, although the packet of ddos attack
Having carried out certain camouflage, but these abnormal flows all exist some attack characteristics, in order to find these attack characteristics, it is right to need
Data traffic is added up.
In the middle of prior art, traffic statistics are had two kinds of solutions.
Scheme one: each thread individually carries out traffic statistics, and each thread individually takies a internal memory and preserves respective
Statistical result, is finally collected the statistical result of each thread by an independent process.
Although employing scheme one can realize preferable traffic statistics effect, performance is fabulous, but can take substantial amounts of
Memory source.In order to ensure the abundance of memory source, generally requiring the hardware configuration constantly promoting cleaning equipment, this can increase
Add O&M cost.
Scheme two: each thread individually carries out traffic statistics, all of Line sharing portion internal memory preserves respective statistics
As a result, finally by an independent process, the statistical result of each thread is collected.
Although employing scheme two can reduce the occupancy of memory source, but, need to use former between each thread
Sub-lock side formula is queued up and is accessed memory source, so will take a substantial amount of time, reduce the execution efficiency of traffic statistics, therefore
Performance extreme difference.
In view of this, need to design a kind of new traffic statistics scheme to overcome drawbacks described above.
Summary of the invention
The embodiment of the present invention provides a kind of flow statistical method and device, while ensureing traffic statistics performance, reduces
EMS memory occupation amount.
The concrete technical scheme that the embodiment of the present invention provides is as follows:
A kind of flow statistical method, including:
Obtain data traffic, use some threads respectively described data traffic to be carried out counting messages, it is thus achieved that respective system
Meter result；
The statistical result obtained by each thread respectively, preserves in the memory source that the physical cpu extremely bound is assigned to,
Wherein, between each physical cpu and at least one thread, there is binding relationship, and each physical cpu is assigned accordingly
A memory source；
According to predetermined manner, the statistical result of each memory source record is collected, it is thus achieved that final statistical result.
Optionally, use any one thread that described data traffic is carried out counting messages, it is thus achieved that corresponding statistical result,
Including:
Any one thread described is used to carry out message extraction in described data traffic；
The message extracted is resolved, determines the characteristic parameter that described message comprises；
Using described characteristic parameter as corresponding statistical result.
Optionally, the statistical result that any one thread is obtained, preserve the internal memory money being assigned to the physical cpu bound
In source, including:
Determine the physical cpu of described any one thread binding, and determine the internal memory money that described physical cpu is assigned to
Source；
In described memory source, all statistical results obtained by any one thread described, with Multidimensional numerical form
Storage, or, all statistical results that any one thread described is obtained, carry out separate storage respectively.
Optionally, any one statistical result that any one thread described is obtained, store with Multidimensional numerical form, bag
Include:
The N number of characteristic parameter specified is extracted from any one statistical result described, and respectively by each characteristic parameter
It is set as the data of data ... the N-dimensional degree of the data of the first dimension, the data of the second dimension, third dimension；
Judge in corresponding memory source, if the data of corresponding described first dimension preserve corresponding many dimensions
Group；
The most then by the data of described second dimension, the data of data ... the N-dimensional degree of third dimension, preserve successively
Corresponding storage position to described Multidimensional numerical；
Otherwise, the Multidimensional numerical that the data creation of corresponding described first dimension is new, and by the data of described second dimension, the
The data of data ... the N-dimensional degree of three dimensionality, preserve the corresponding storage position to described new Multidimensional numerical successively.
Optionally, according to predetermined manner, the statistical result of each memory source record is collected, it is thus achieved that finally add up
As a result, including:
Preserve the statistical result of each thread according to Multidimensional numerical form, then, from each memory source, extract
The Multidimensional numerical that the data of dimension are identical merges, to obtain final statistical result；Or,
If the statistical result of each thread to be carried out respectively separate storage, then according to default extraction standard, extract bag
Statistical result containing the characteristic parameter specified merges, to obtain final statistical result.
A kind of flow statistic device, including:
Statistic unit, is used for obtaining data traffic, uses some threads respectively described data traffic to be carried out counting messages,
Obtain respective statistical result；
Memory element, for the statistical result obtained by each thread respectively, the physical cpu preserving extremely binding is assigned to
Memory source in, wherein, between each physical cpu and at least one thread, there is binding relationship, and each physical cpu
It is assigned corresponding a memory source；
Collection unit, for collecting the statistical result of each memory source record according to predetermined manner, it is thus achieved that
Whole statistical result.
Optionally, use any one thread that described data traffic is carried out counting messages, it is thus achieved that corresponding statistical result
Time, described statistic unit is used for:
Any one thread described is used to carry out message extraction in described data traffic；
The message extracted is resolved, determines the characteristic parameter that described message comprises；
Using described characteristic parameter as corresponding statistical result.
Optionally, the statistical result that any one thread is obtained, preserve the internal memory money being assigned to the physical cpu bound
Time in source, described memory element is used for:
Determine the physical cpu of described any one thread binding, and determine the internal memory money that described physical cpu is assigned to
Source；
In described memory source, all statistical results obtained by any one thread described, with Multidimensional numerical form
Storage, or, all statistical results that any one thread described is obtained, carry out separate storage respectively.
Optionally, any one statistical result that any one thread described is obtained, when storing with Multidimensional numerical form,
Described memory element is used for:
The N number of characteristic parameter specified is extracted from any one statistical result described, and respectively by each characteristic parameter
It is set as the data of data ... the N-dimensional degree of the data of the first dimension, the data of the second dimension, third dimension；
Judge in corresponding memory source, if the data of corresponding described first dimension preserve corresponding many dimensions
Group；
The most then by the data of described second dimension, the data of data ... the N-dimensional degree of third dimension, preserve successively
Corresponding storage position to described Multidimensional numerical；
Otherwise, the Multidimensional numerical that the data creation of corresponding described first dimension is new, and by the data of described second dimension, the
The data of data ... the N-dimensional degree of three dimensionality, preserve the corresponding storage position to described new Multidimensional numerical successively.
Optionally, according to predetermined manner, the statistical result of each memory source record is collected, it is thus achieved that finally add up
During result, described collection unit is used for:
Preserve the statistical result of each thread according to Multidimensional numerical form, then, from each memory source, extract
The Multidimensional numerical that the data of dimension are identical merges, to obtain final statistical result；Or,
If the statistical result of each thread to be carried out respectively separate storage, then according to default extraction standard, extract bag
Statistical result containing the characteristic parameter specified merges, to obtain final statistical result.
The present invention has the beneficial effect that:
In the embodiment of the present invention, the data traffic that cleaning device utilizes some threads to obtain respectively carries out counting messages, will
Multiple threads are bundled under different physical cpus, are respectively allocated a memory source to each physical cpu simultaneously, and each line
Respective statistical result is stored in the memory source that the physical cpu of binding is corresponding by journey；As such, it is possible to be not necessarily each line
Journey distributes single internal memory, thus substantially reduces EMS memory occupation amount；On the other hand, although often each under portion memory source
Thread remains a need for when storing respective statistical result queuing up accessing, but under same physical cpu, the cache hit rate of internal memory is high,
Access speed is fast, therefore, even if getting rid of access not interfere with storage speed, thus is effectively increased statistical efficiency, it is ensured that stream
Amount statistic property.
Accompanying drawing explanation
Fig. 1 is ddos attack principle schematic under prior art；
Fig. 2 is Multidimensional numerical storage mode schematic diagram in the embodiment of the present invention；
Fig. 3 is traffic statistics flow chart in the embodiment of the present invention；
Fig. 4 is cleaning apparatus function structural representation in the embodiment of the present invention；
Fig. 5 is flow statistic device schematic diagram in the embodiment of the present invention.
Detailed description of the invention
In order to, while ensureing traffic statistics performance, reduce EMS memory occupation amount, in the embodiment of the present invention, refitting devises
A kind of traffic statistics scheme, the program is: each thread one physical cpu of binding, each thread is as the criterion with physical cpu and carries out
Packet, each physical cpu is respectively provided with a memory source, and the statistical result of self is stored in self binding by each thread
In the memory source that physical cpu is corresponding, the thread in the most same group could be shared with a memory source, the line between different groups
Journey uses different memory sources.
Below in conjunction with the accompanying drawings the embodiment that the present invention is preferential is described in detail.
In the embodiment of the present invention, it is necessary first to clear and definite physical cpu and the concept of logic CPU, physical cpu i.e. refers on server
Actual CPU number, logic CPU then utilizes Hyper-Threading that physical cpu is invented Logical processing unit, a physical cpu
Multiple logic CPU can be divided into.
In actual application, each thread can bind corresponding logic CPU, and between logic-based CPU and physical cpu
Division relation, each logic CPU can associate again a corresponding physical cpu, therefore, it can regard that each thread is all bound as
One corresponding physical cpu.In the present embodiment, will take this as the standard and carry out associated description.
Refering to shown in Fig. 2, in actual application, equipment can exist at least one physical cpu, and on each physical cpu
Multiple logic CPU can be marked off, in the present embodiment, it is assumed that there is n physical cpu on equipment, and each physical cpu is all drawn
Separate two logics CPU.Each logic CPU has all bound the thread of some, and each thread all can carry out independent stream
Amount statistics.
So-called traffic statistics, i.e. refer to that thread carries out message extraction at original data stream, and to each message extracted
Resolve respectively, determine the characteristic parameter comprised in message, and these characteristic parameters i.e. can by as corresponding statistical result,
Wherein, these characteristic parameters can comprise " purpose IP ", " source IP ", " attack type " etc., but is not limited to this three category feature ginseng
Amount.
In the embodiment of the present invention, store statistical result with Multidimensional numerical form, from statistical result, extract appointment
N number of characteristic parameter, and respectively each characteristic parameter is set as the data of the first dimension, the data of the second dimension, the third dimension
The data of data ... the N-dimensional degree of degree；
Judge in corresponding memory source, if the data of corresponding described first dimension preserve corresponding many dimensions
Group；
The most then by the data of described second dimension, the data of data ... the N-dimensional degree of third dimension, preserve successively
Corresponding storage position to described Multidimensional numerical；
Otherwise, the Multidimensional numerical that the data creation of corresponding described first dimension is new, and by the data of described second dimension, the
The data of data ... the N-dimensional degree of three dimensionality, preserve the corresponding storage position to described new Multidimensional numerical successively.
So-called Multidimensional numerical, i.e. refers to that an array has multiple statistics dimension, as in figure 2 it is shown, in the present embodiment, it is assumed that
Multi-dimensional data has three kinds of statistics dimensions, and the i.e. first dimension, the second dimension and third dimension, three kinds of dimensions are with nested form
Record.
Below by the first dimension be " purpose IP ", the second dimension be as a example by " source IP ", third dimension are " attack type ", right
Multidimensional data is introduced.
In the present embodiment, each thread binds a corresponding physical cpu, and each physical cpu is assigned one
Fixed memory source, each thread belonging to same physical cpu can share same memory source.
For example, it is assumed that physical cpu 1 has bound thread a and thread b, and physical cpu 1 distribution has obtained memory source 1,
So, the statistical result each obtained after thread a and thread b will will carry out traffic statistics is stored in memory source 1.
Assume each bar statistical result preserves on the basis of the first dimension, then, as in figure 2 it is shown, at each internal memory
In resource, the statistical result that the first dimension value is identical will be recorded at one.
Such as, the first dimension is " purpose IP ", i.e. refers to that the statistical result of each message makes a distinction with purpose IP,
The statistical result that purpose IP is identical will merged record.
As, the data 1 of the first dimension have recorded whole statistical results of the message of " purpose IP is 1.1.1.1 ", and
The data 2 of dimension have recorded whole statistical results of the message of " purpose IP is 1.1.1.2 ".
Further, owing to being various dimensions array results, the data of the most multiple dimensions are nested type records, such as Fig. 2
Shown in, with the data instance of first dimension, the data of the first dimension have recorded the first dimension status information, e.g., the
Can include data 1, data 2 and data 3 in one-dimensional status information, wherein, data 1 represent, a certain purpose IP is (e.g.,
1.1.1.1) the corresponding total input flow rate of message, data 2 represent that the message of above-mentioned a certain purpose IP (e.g., 1.1.1.1) correspondence is total
Output flow (total flow that i.e. cleaning equipment is let pass for this purpose IP), (e.g., data 3 represent above-mentioned a certain purpose IP
1.1.1.1) corresponding message data flow path direction.
Additionally, the data of the first dimension are gone back the nesting data of some second dimensions, it is assumed that the second dimension is " source IP ",
So, under same purpose IP, the further statistical result of each message makes a distinction with source IP, the statistics that source IP is identical
Result will merged record.
As, the data 1 of the second dimension under the data 1 of the first dimension have recorded the message of " source IP is 1.1.1.3 "
All statistical result, and the data 2 of the second dimension under the data 1 of the first dimension have recorded the report of " source IP is 1.1.1.4 "
Whole statistical results of literary composition
As in figure 2 it is shown, with the data instance of second dimension, have recorded two-dimensional state in the data of the second dimension
Information, e.g., can include data 1 ', data 2 ' and data 3 ' in the second dimension status information, and wherein, data 1 ' represent a certain purpose
The total input flow rate of message corresponding for a certain source IP (e.g., 1.1.1.3) under IP (e.g., 1.1.1.1), data 2 ' represent above-mentioned a certain mesh
IP (e.g., 1.1.1.1) under the total output flow of message corresponding for a certain source IP (e.g., 1.1.1.3) (i.e. cleaning equipment is for this
The total flow that source IP lets pass), data 3 ' represent the above-mentioned a certain source IP under above-mentioned a certain purpose IP (e.g., 1.1.1.1) (e.g.,
1.1.1.3) corresponding message data flow path direction.
The present embodiment is illustrate as a example by three-dimensional array, therefore, in the data of the second dimension also nested the
The data of three dimensionality, it is assumed that third dimension is " attack type ", then, under the same source IP under same purpose IP, attack class
The statistical result that type is identical will merged record.
As, the data 1 of the third dimension under the data 1 of the second dimension under the data 1 of the first dimension have recorded and " attack
Type is that router is attacked " whole statistical results of message, under the data 1 of the second dimension under the data 1 of the first dimension
The data 2 of third dimension have recorded whole statistical results of the message of " attack type is that bandwidth is attacked ".
As in figure 2 it is shown, with the data instance of a third dimension, the data of third dimension have recorded third dimension
Status information, e.g., can include data 1 in third dimension status information ", data 2 " and data 3 ", wherein, data 1 " represent a certain
The report that under purpose IP (e.g., 1.1.1.1), under a certain source IP (e.g., 1.1.1.3), a certain attack type (e.g., router is attacked) is corresponding
The total input flow rate of literary composition, data 2 " represent a certain under a certain source IP (e.g., 1.1.1.3) under above-mentioned a certain purpose IP (e.g., 1.1.1.1)
Attack type (e.g., router is attacked) the corresponding total output flow of message (i.e. let pass for this attack type by cleaning equipment
Total flow), data 3 " represent under the above-mentioned a certain source IP (e.g., 1.1.1.3) under above-mentioned a certain purpose IP (e.g., 1.1.1.1)
The message data flow path direction that a certain attack type (e.g., router is attacked) is corresponding.
So far, array loop nesting recording process terminates.Certainly, in the present embodiment, only it is situated between as a example by three-dimensional array
Continue, if the most nested record data of fourth dimension, the data of the 5th dimension in the data of the third dimension ..., the most all it is suitable for this
The technical scheme provided in bright embodiment, for the ease of describing, does not repeats them here.
As in figure 2 it is shown, the statistical result that self obtains all can be deposited by each thread according to the mode of above-mentioned three-dimensional array
Enter in the memory source that the physical cpu bound is corresponding, owing to each thread works alone, therefore, it is possible to different internal memory money
Preserving the statistical result that the data of the first dimension are identical in source, such as, thread a binds with physical cpu 1, and therefore, thread a will add up
The ASSOCIATE STATISTICS result record that data are " purpose IP1.1.1.1 " of the first dimension obtained is in physical cpu 1 is assigned to
Depositing in resource 1, and thread c and physical cpu 2 are bound, therefore, the data of the first dimension that compiling is obtained by thread c are also for " mesh
IP1.1.1.1 " ASSOCIATE STATISTICS result record in the memory source 2 that physical cpu 2 is assigned to.
Finally, statistical result identical for the data of the first dimension in each memory source is merged by cleaning equipment, as
Shown in Fig. 4, thus the data of each the first dimension corresponding obtain complete statistical result.
Conceive based on above-mentioned technology, refering to shown in Fig. 3, in the embodiment of the present invention, carry out the detailed process of traffic statistics such as
Under:
Step 300: cleaning equipment obtains data traffic.
Concrete, as it is shown in figure 1, cleaning equipment is arranged on the access person's of being hacked network of network front end, in data traffic
Traffic statistics are carried out, to reach the purpose of flow cleaning before the entrance person of being hacked.
Step 310: cleaning equipment carries out message extraction to the data traffic obtained, and uses some threads to carry out respectively
Counting messages, it is thus achieved that respective statistical result.
In the present embodiment, so-called counting messages, is i.e. to resolve message, really according to the parameter of three-dimensional array definition
The characteristic parameter that fixed described message comprises, these characteristic parameters are exactly the analysis result obtained, also known as statistical result.
Such as: message 1 contains " purpose IP ", " source IP ", " destination interface ", " source port ", " attack type ", " association
View " etc. characteristic parameter, message is resolved by cleaning equipment according to the parameter defined in three-dimensional array, extracts containing " purpose
IP ", " source IP " and the statistical result of " attack type " these three characteristic parameter.
Step 320: cleaning equipment determines the physical cpu that each thread is bound respectively, and determines each physical cpu
Distribution memory source extremely.
In the implementation case, so-called memory source, is i.e. the shared drive block shown in Fig. 2, and e.g., memory source 1 is
Shared drive block 1, memory source 2 is shared drive block 2, the like, memory source n is shared drive block n.
As described in content before, in the embodiment of the present invention, with a logic CPU binding on each threading logic, and patrol
Collect CPU logical partitioning from physical cpu out, to therefore, it can be considered as each thread and all bind with a physical cpu.
On the other hand, in the embodiment of the present invention, each physical cpu is assigned certain memory source, is bundled in same
Thread on physical cpu shares same memory source.
Step 330: the statistical result that cleaning equipment will be obtained by each thread respectively, is stored in corresponding physics
In the memory source that CPU is assigned to.
In the implementation case, each thread can bind a physical cpu, and each physical cpu can correspondence one
Part memory source, therefore, the statistical result of each thread will be saved under the memory source that the physical cpu of association is corresponding,
And the statistical result of each thread is not to be as the criterion with thread to store in memory source, but it is as the criterion with dimension and stores.
Such as, respective statistical result is saved in physical cpu 1 correspondence by the thread 1 under physical cpu 1, thread 2, thread 3
Memory source 1 in.
Assume that the statistical result of thread 1 correspondence is for " purpose IP is 1.1.1.1, source IP is 1.1.2.1, attack type is road
Attacked by device ", the statistical result of thread 2 correspondence is for " purpose IP is 1.1.1.1, source IP is 1.1.2.2, attack type is route
Device is attacked ", the statistical result of thread 3 correspondence is for " purpose IP is 1.1.1.2, source IP is 1.1.2.2, attack type is router
Attack ".
Being the criteria for classifying by the first dimension, in memory source 1, in the data 1 of the first dimension, the statistical result of storage is then line
Journey 1 and thread is 2-in-1 and after statistical result, in the data 2 of the first dimension storage the statistical result that statistical result is thread 3.
Certainly, in the embodiment of the present invention, the statistical result preserving each thread according to Multidimensional numerical mode as shown in Figure 2
Mode be only for example, in actual application, it is also possible in memory source block, directly the statistical result of each thread is carried out solely
Vertical preservation, and carry out again extracting merging in follow-up aggregation stages.
Step 340: the statistical result of record in different memory source blocks is collected by cleaning equipment, it is thus achieved that finally add up
Result.
Concrete, in the embodiment of the present invention, when collecting, process system to each memory source is individually enabled
Meter result collects, and process can regularly scan each three-dimensional array of record in each memory source, if finding different
There is the three-dimensional array that the first dimension is identical between memory source, then carry out extracting and merging collecting.
Such as, the statistical result of each memory source is deposited with the storage mode of three-dimensional array, and the i.e. first dimension is " purpose
IP ", the second dimension is " source IP " and third dimension is " attack type ", with two memory sources, and under two memory sources again
Illustrate as a example by each own two statistical results.
Assume that the data 1 of the first dimension in memory source 1 are for " purpose IP is 1.1.1.1, source IP is 1.1.2.1, attacks class
Type is that router is attacked " statistical result, in memory source 1, the data 2 of the first dimension are for " purpose IP is 1.1.1.2, source IP is
1.1.2.1, attack type be that broadband is attacked " statistical result, in memory source 2, the data 1 of the first dimension are for " purpose IP is
1.1.1.1, source IP be 1.1.2.2, attack type be router attack ", in memory source 2, the data 2 of the first dimension are " purpose
IP is 1.1.1.2, source IP is 1.1.2.3, attack type is that router is attacked ", if by the first dimension i.e. " purpose IP " for dividing
Standard, process when collecting, each memory source merge after data 1 ' for " purpose IP is 1.1.1.1, source IP is
1.1.2.1, attack type is that router is attacked " and " purpose IP is 1.1.1.1, source IP is 1.1.2.2, attack type is for routeing
Device is attacked " two statistical results, the data 2 ' after merging are for " purpose IP is 1.1.1.2, source IP is 1.1.2.1, attack type is
Broadband is attacked " and " purpose IP is 1.1.1.2, source IP is 1.1.2.3, attack type is router attack " two statistical results.
In the present embodiment, illustrate as a example by Multidimensional numerical form stores by statistical result.In actual application, if
Data traffic is not a lot, or, the disposal ability of cleaning equipment is the most powerful, then preserve statistics in each memory source
During result, it is also possible to do not use Multidimensional numerical form to preserve, but directly each statistical result is carried out independent preservation, and
In the data summarization stage, then by a special process, use the extraction standard preset, carry out each statistical result extracting and converge
Always.
For example, it is assumed that the characteristic parameter of each statistical result comprises " purpose IP ", " source IP " and " attack type ", if preset
Extraction standard is as criteria for classification with " purpose IP ", then statistical result identical to " purpose IP " in each memory source is closed by process
And together, if the extraction standard preset is as criteria for classification with " attack type ", then process " will be attacked in each memory source
Type " identical statistical result combines, by that analogy.
Wherein, the extraction standard preset is not limited to " purpose IP ", " source IP " and " attack type " three class, it is also possible to be
" destination interface ", " source port ", " agreement " etc..
Refering to shown in Fig. 5, in the embodiment of the present invention, clean device and at least include statistic unit 50, memory element 51 and converge
Total unit 52, wherein,
Statistic unit 50, is used for obtaining data traffic, uses some threads respectively described data traffic to be carried out message system
Meter, it is thus achieved that respective statistical result；
Memory element 51, for the statistical result obtained by each thread respectively, preserves the physical cpu distribution to binding
To memory source in, wherein, between each physical cpu and at least one thread, there is binding relationship, and each physics
CPU is assigned corresponding a memory source；
Collection unit 52, for collecting the statistical result of each memory source record according to predetermined manner, it is thus achieved that
Final statistical result.
Optionally, use any one thread that described data traffic is carried out counting messages, it is thus achieved that corresponding statistical result
Time, described statistic unit is used for:
Any one thread described is used to carry out message extraction in described data traffic；
The message extracted is resolved, determines the characteristic parameter that described message comprises；
Using described characteristic parameter as corresponding statistical result.
Optionally, the statistical result that any one thread is obtained, preserve the internal memory money being assigned to the physical cpu bound
Time in source, described memory element is used for:
Determine the physical cpu of described any one thread binding, and determine the internal memory money that described physical cpu is assigned to
Source；
In described memory source, all statistical results obtained by any one thread described, with Multidimensional numerical form
Storage, or, all statistical results that any one thread described is obtained, carry out separate storage respectively.
Optionally, any one statistical result that any one thread described is obtained, when storing with Multidimensional numerical form,
Described memory element is used for:
The N number of characteristic parameter specified is extracted from any one statistical result described, and respectively by each characteristic parameter
It is set as the data of data ... the N-dimensional degree of the data of the first dimension, the data of the second dimension, third dimension；
Judge in corresponding memory source, if the data of corresponding described first dimension preserve corresponding many dimensions
Group；
The most then by the data of described second dimension, the data of data ... the N-dimensional degree of third dimension, preserve successively
Corresponding storage position to described Multidimensional numerical；
Otherwise, the Multidimensional numerical that the data creation of corresponding described first dimension is new, and by the data of described second dimension, the
The data of data ... the N-dimensional degree of three dimensionality, preserve the corresponding storage position to described new Multidimensional numerical successively.
Optionally, according to predetermined manner, the statistical result of each memory source record is collected, it is thus achieved that finally add up
During result, described collection unit is used for:
Preserve the statistical result of each thread according to Multidimensional numerical form, then, from each memory source, extract
The Multidimensional numerical that the data of dimension are identical merges, to obtain final statistical result；Or,
If the statistical result of each thread to be carried out respectively separate storage, then according to default extraction standard, extract bag
Statistical result containing the characteristic parameter specified merges, to obtain final statistical result.
In sum, in the embodiment of the present invention, the data traffic that cleaning device utilizes some threads to obtain respectively is reported
Multiple threads are bundled under different physical cpus by literary composition statistics, are respectively allocated a memory source to each physical cpu simultaneously,
And respective statistical result is stored in the memory source that the physical cpu of binding is corresponding by each thread；As such, it is possible to be not necessarily
Each thread distributes single internal memory, thus substantially reduces EMS memory occupation amount；On the other hand, although every a memory source
Under each thread remain a need for when storing respective statistical result queuing up accessing, but the caching of internal memory under same physical cpu
Shooting straight, access speed is fast, therefore, even if getting rid of access not interfere with storage speed, thus is effectively increased statistical efficiency,
Ensure that traffic statistics performance.
Those skilled in the art are it should be appreciated that embodiments of the invention can be provided as method, system or computer program
Product.Therefore, the reality in terms of the present invention can use complete hardware embodiment, complete software implementation or combine software and hardware
Execute the form of example.And, the present invention can use at one or more computers wherein including computer usable program code
The upper computer program product implemented of usable storage medium (including but not limited to disk memory, CD-ROM, optical memory etc.)
The form of product.
The present invention is with reference to method, equipment (system) and the flow process of computer program according to embodiments of the present invention
Figure and/or block diagram describe.It should be understood that can the most first-class by computer program instructions flowchart and/or block diagram
Flow process in journey and/or square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
Instruction arrives the processor of general purpose computer, special-purpose computer, Embedded Processor or other programmable data processing device to produce
A raw machine so that the instruction performed by the processor of computer or other programmable data processing device is produced for real
The device of the function specified in one flow process of flow chart or multiple flow process and/or one square frame of block diagram or multiple square frame now.
These computer program instructions may be alternatively stored in and computer or other programmable data processing device can be guided with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in this computer-readable memory produces and includes referring to
Make the manufacture of device, this command device realize at one flow process of flow chart or multiple flow process and/or one square frame of block diagram or
The function specified in multiple square frames.
These computer program instructions also can be loaded in computer or other programmable data processing device so that at meter
Perform sequence of operations step on calculation machine or other programmable devices to produce computer implemented process, thus at computer or
The instruction performed on other programmable devices provides for realizing at one flow process of flow chart or multiple flow process and/or block diagram one
The step of the function specified in individual square frame or multiple square frame.
Although preferred embodiments of the present invention have been described, but those skilled in the art once know basic creation
Property concept, then can make other change and amendment to these embodiments.So, claims are intended to be construed to include excellent
Select embodiment and fall into all changes and the amendment of the scope of the invention.
Obviously, those skilled in the art can carry out various change and modification without deviating from this to the embodiment of the present invention
The spirit and scope of bright embodiment.So, if these amendments of the embodiment of the present invention and modification belong to the claims in the present invention
And within the scope of equivalent technologies, then the present invention is also intended to comprise these change and modification.
Claims (10)
1. a flow statistical method, it is characterised in that including:
Obtain data traffic, use some threads respectively described data traffic to be carried out counting messages, it is thus achieved that respective statistics is tied
Really；
The statistical result obtained by each thread respectively, preserves the internal memory being assigned to the physical central processor CPU bound
In resource, wherein, between each physical cpu and at least one thread, there is binding relationship, and each physical cpu is assigned
Corresponding a memory source；
According to predetermined manner, the statistical result of each memory source record is collected, it is thus achieved that final statistical result.
2. the method for claim 1, it is characterised in that use any one thread that described data traffic is carried out message
Statistics, it is thus achieved that corresponding statistical result, including:
Any one thread described is used to carry out message extraction in described data traffic；
The message extracted is resolved, determines the characteristic parameter that described message comprises；
Using described characteristic parameter as corresponding statistical result.
3. method as claimed in claim 1 or 2, it is characterised in that the statistical result obtained by any one thread, preserves extremely
In the memory source that the physical cpu of binding is assigned to, including:
Determine the physical cpu of described any one thread binding, and determine the memory source that described physical cpu is assigned to；
In described memory source, all statistical results that any one thread described is obtained, store with Multidimensional numerical form,
Or, all statistical results that any one thread described is obtained, carry out separate storage respectively.
4. method as claimed in claim 3, it is characterised in that any one statistics knot that any one thread described is obtained
Really, store with Multidimensional numerical form, including:
From any one statistical result described, extract the N number of characteristic parameter specified, and respectively each characteristic parameter is set
It is the data of data ... the N-dimensional degree of the data of the first dimension, the data of the second dimension, third dimension；
Judge in corresponding memory source, if the data of corresponding described first dimension preserve corresponding Multidimensional numerical；
The most then by the data of described second dimension, the data of data ... the N-dimensional degree of third dimension, preserve successively to institute
State the corresponding storage position of Multidimensional numerical；
Otherwise, the Multidimensional numerical that the data creation of corresponding described first dimension is new, and by the data of described second dimension, the third dimension
The data of data ... the N-dimensional degree of degree, preserve the corresponding storage position to described new Multidimensional numerical successively.
5. method as claimed in claim 4, it is characterised in that the statistics of each memory source record is tied according to predetermined manner
Fruit collects, it is thus achieved that final statistical result, including:
Preserve the statistical result of each thread according to Multidimensional numerical form, then, from each memory source, extract the first dimension
The Multidimensional numerical that the data of degree are identical merges, to obtain final statistical result；Or,
If the statistical result of each thread to be carried out respectively separate storage, then according to default extraction standard, extract and include
The statistical result of the characteristic parameter specified merges, to obtain final statistical result.
6. a flow statistic device, it is characterised in that including:
Statistic unit, is used for obtaining data traffic, uses some threads respectively described data traffic to be carried out counting messages, it is thus achieved that
Respective statistical result；
Memory element, for the statistical result obtained by each thread respectively, preserves the physical central processor CPU to binding
In the memory source being assigned to, wherein, there is binding relationship, and each between each physical cpu and at least one thread
Physical cpu is assigned corresponding a memory source；
Collection unit, for collecting the statistical result of each memory source record according to predetermined manner, it is thus achieved that finish-unification
Meter result.
7. device as claimed in claim 6, it is characterised in that use any one thread that described data traffic is carried out message
Statistics, it is thus achieved that during corresponding statistical result, described statistic unit is used for:
Any one thread described is used to carry out message extraction in described data traffic；
The message extracted is resolved, determines the characteristic parameter that described message comprises；
Using described characteristic parameter as corresponding statistical result.
Device the most as claimed in claims 6 or 7, it is characterised in that the statistical result obtained by any one thread, preserves extremely
Time in the memory source that the physical cpu of binding is assigned to, described memory element is used for:
Determine the physical cpu of described any one thread binding, and determine the memory source that described physical cpu is assigned to；
In described memory source, all statistical results that any one thread described is obtained, store with Multidimensional numerical form,
Or, all statistical results that any one thread described is obtained, carry out separate storage respectively.
9. device as claimed in claim 8, it is characterised in that any one statistics knot that any one thread described is obtained
Really, when storing with Multidimensional numerical form, described memory element is used for:
From any one statistical result described, extract the N number of characteristic parameter specified, and respectively each characteristic parameter is set
It is the data of data ... the N-dimensional degree of the data of the first dimension, the data of the second dimension, third dimension；
Judge in corresponding memory source, if the data of corresponding described first dimension preserve corresponding Multidimensional numerical；
The most then by the data of described second dimension, the data of data ... the N-dimensional degree of third dimension, preserve successively to institute
State the corresponding storage position of Multidimensional numerical；
Otherwise, the Multidimensional numerical that the data creation of corresponding described first dimension is new, and by the data of described second dimension, the third dimension
The data of data ... the N-dimensional degree of degree, preserve the corresponding storage position to described new Multidimensional numerical successively.
10. device as claimed in claim 9, it is characterised in that according to the predetermined manner statistics to each memory source record
Result collects, it is thus achieved that during final statistical result, described collection unit is used for:
Preserve the statistical result of each thread according to Multidimensional numerical form, then, from each memory source, extract the first dimension
The Multidimensional numerical that the data of degree are identical merges, to obtain final statistical result；Or,
If the statistical result of each thread to be carried out respectively separate storage, then according to default extraction standard, extract and include
The statistical result of the characteristic parameter specified merges, to obtain final statistical result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610622351.5A CN106230823A (en) | 2016-08-01 | 2016-08-01 | A kind of flow statistical method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610622351.5A CN106230823A (en) | 2016-08-01 | 2016-08-01 | A kind of flow statistical method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106230823A true CN106230823A (en) | 2016-12-14 |
Family
ID=57534880
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610622351.5A Pending CN106230823A (en) | 2016-08-01 | 2016-08-01 | A kind of flow statistical method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106230823A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109327441A (en) * | 2018-10-10 | 2019-02-12 | 光通天下网络科技股份有限公司 | Attack data integration method, integrating apparatus and the electronic equipment of distributed DDoS system of defense |
| CN110311838A (en) * | 2019-07-24 | 2019-10-08 | 北京神州绿盟信息安全科技股份有限公司 | A kind of method and device of security service traffic statistics |
| CN110708258A (en) * | 2019-09-29 | 2020-01-17 | Oppo广东移动通信有限公司 | Flow control method, flow control device, server and storage medium |
| CN112437074A (en) * | 2020-11-17 | 2021-03-02 | 北京金山云网络技术有限公司 | Counting processing method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309840A (en) * | 2013-07-08 | 2013-09-18 | 天津汉柏汉安信息技术有限公司 | Connection establishment method and device |
| CN104461735A (en) * | 2014-11-28 | 2015-03-25 | 杭州华为数字技术有限公司 | Method and device for distributing CPU resources in virtual scene |
| CN104572295A (en) * | 2014-12-12 | 2015-04-29 | 北京应用物理与计算数学研究所 | Structured grid data management method matched with high-performance computer system structure |
| CN105282152A (en) * | 2015-09-28 | 2016-01-27 | 广东睿江科技有限公司 | Abnormal flow detection method |
-
2016
- 2016-08-01 CN CN201610622351.5A patent/CN106230823A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309840A (en) * | 2013-07-08 | 2013-09-18 | 天津汉柏汉安信息技术有限公司 | Connection establishment method and device |
| CN104461735A (en) * | 2014-11-28 | 2015-03-25 | 杭州华为数字技术有限公司 | Method and device for distributing CPU resources in virtual scene |
| CN104572295A (en) * | 2014-12-12 | 2015-04-29 | 北京应用物理与计算数学研究所 | Structured grid data management method matched with high-performance computer system structure |
| CN105282152A (en) * | 2015-09-28 | 2016-01-27 | 广东睿江科技有限公司 | Abnormal flow detection method |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109327441A (en) * | 2018-10-10 | 2019-02-12 | 光通天下网络科技股份有限公司 | Attack data integration method, integrating apparatus and the electronic equipment of distributed DDoS system of defense |
| CN109327441B (en) * | 2018-10-10 | 2021-01-05 | 光通天下网络科技股份有限公司 | Attack data integration method and integration device of distributed DDoS defense system and electronic equipment |
| CN110311838A (en) * | 2019-07-24 | 2019-10-08 | 北京神州绿盟信息安全科技股份有限公司 | A kind of method and device of security service traffic statistics |
| CN110708258A (en) * | 2019-09-29 | 2020-01-17 | Oppo广东移动通信有限公司 | Flow control method, flow control device, server and storage medium |
| CN110708258B (en) * | 2019-09-29 | 2023-04-07 | Oppo广东移动通信有限公司 | Flow control method, device, server and storage medium |
| CN112437074A (en) * | 2020-11-17 | 2021-03-02 | 北京金山云网络技术有限公司 | Counting processing method and device, electronic equipment and storage medium |
| CN112437074B (en) * | 2020-11-17 | 2022-11-22 | 北京金山云网络技术有限公司 | Counting processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11061942B2 (en) | Unstructured data fusion by content-aware concurrent data processing pipeline | |
| US8069210B2 (en) | Graph based bot-user detection | |
| US10467433B2 (en) | Event processing system | |
| CN103930887B (en) | The inquiry stored using raw column data collects generation | |
| CN106230823A (en) | A kind of flow statistical method and device | |
| US11258703B1 (en) | Data plane for learning flows, collecting metadata regarding learned flows and exporting metadata regarding learned flows | |
| CN106487596A (en) | Distributed Services follow the tracks of implementation method | |
| CN1633111B (en) | High-speed network traffic flow classification method | |
| Kotenko et al. | Attack detection in IoT critical infrastructures: a machine learning and big data processing approach | |
| CN103414608B (en) | Rapid web flow collection statistical system and method | |
| CN110175154A (en) | A kind of processing method of log recording, server and storage medium | |
| CN107040405A (en) | Passive type various dimensions main frame Fingerprint Model construction method and its device under network environment | |
| Wang et al. | Honeynet construction based on intrusion detection | |
| CN109344137A (en) | A kind of log storing method and system | |
| CN106909624B (en) | Real-time sequencing optimization method for mass data | |
| CN104243348A (en) | Data processing method and device | |
| Kim et al. | Implementation of hybrid P2P networking distributed web crawler using AWS for smart work news big data | |
| CN117440053B (en) | Multistage cross die access method and system | |
| JP5104529B2 (en) | Analysis support program, analysis support method, and analysis support apparatus | |
| Bang et al. | Design and implementation of storage system for real-time blockchain network monitoring system | |
| Kousar et al. | DDoS attack detection system using Apache spark | |
| Aslam et al. | Pre‐filtering based summarization for data partitioning in distributed stream processing | |
| TW201710943A (en) | System and method for high frequency heuristic data acquisition and analytics of information security events | |
| Wang et al. | DataNet: A data distribution-aware method for sub-dataset analysis on distributed file systems | |
| KR20120085400A (en) | Packet Processing System and Method by Prarllel Computation Based on Hadoop |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161214 |