CN116471212B - Service type-based network traffic data processing method and system - Google Patents
Service type-based network traffic data processing method and system Download PDFInfo
- Publication number
- CN116471212B CN116471212B CN202310409270.7A CN202310409270A CN116471212B CN 116471212 B CN116471212 B CN 116471212B CN 202310409270 A CN202310409270 A CN 202310409270A CN 116471212 B CN116471212 B CN 116471212B
- Authority
- CN
- China
- Prior art keywords
- data
- network
- service
- network flow
- business
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 7
- 230000005540 biological transmission Effects 0.000 claims abstract description 44
- 238000000034 method Methods 0.000 claims abstract description 41
- 238000004458 analytical method Methods 0.000 claims abstract description 23
- 230000006399 behavior Effects 0.000 claims abstract description 23
- 238000007781 pre-processing Methods 0.000 claims abstract description 18
- 238000003860 storage Methods 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims description 37
- 238000004891 communication Methods 0.000 claims description 15
- 238000005192 partition Methods 0.000 claims description 10
- 230000002159 abnormal effect Effects 0.000 claims description 8
- 230000011218 segmentation Effects 0.000 claims description 8
- 238000013500 data storage Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 3
- 238000007405 data analysis Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 239000002699 waste material Substances 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000014759 maintenance of location Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 210000001520 comb Anatomy 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005206 flow analysis Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 230000008521 reorganization Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/026—Capturing of monitoring data using flow identification
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
- H04L43/0847—Transmission error
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
- H04L43/106—Active monitoring, e.g. heartbeat, ping or trace-route using time related information in packets, e.g. by adding timestamps
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/18—Protocol analysers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application discloses a network flow data processing method and system based on service type, comprising the following steps: collecting network flow data and carrying out full protocol analysis on the network flow data; preprocessing the analyzed network flow data according to the service area and the non-service area based on the service behavior learning model, marking the service sequence numbers, and establishing a segmented index for the partitioned data according to the service sequence numbers and the time stamps; the network flow data is stored in a distributed mode according to the service serial numbers and the time stamps; the data is exported according to a preset format or sent to a designated data receiving system. By analyzing the data packet and marking the data packet in a mode of combining the service session control sequence number and the timestamp on the basis of collecting the data packet by using the PF_RING protocol, network delay and memory consumption caused by switching between a user mode and a kernel mode of a general bypass flow method are avoided, and the influence of a large amount of dirty data on efficiency in collection, transmission, storage and data analysis is avoided.
Description
Technical Field
The present application relates to the field of network data transmission, and in particular, to a method and system for processing network traffic data based on a service type.
Background
In recent years, network security events are frequent, and network security is facing serious challenges, and the root is that part of core technologies and devices are subject to people, so that timely discovery, timely analysis and timely disposal cannot be achieved when network risks are faced. Along with the promulgation of data security laws, network session flow data are more and more paid attention, and in the past, most of users purchase hardware devices of third-party manufacturers to introduce network flow in a switch mirror image mode, and then analyze the network flow in a mode of a preset model, a rule base or a blacklist and the like. The method is effective, but in the practical application process, the model and the rule are not updated timely, the acquired data is incomplete, the acquisition efficiency is low, and therefore, the method cannot be analyzed and early-warned timely and effectively. The network traffic analysis level needs to be improved, and once a network security event occurs, the security and stable operation of the network and the information system may be affected.
In the prior art, some methods for collecting and residing data packets are disclosed, such as a buffer packet capturing method based on libpcap, a wireframe method and the like for capturing traffic data, and the method for capturing the data packets is frequently switched between a user mode and a kernel mode, so that network delay and memory consumption are caused. Meanwhile, in the data acquisition, too much occupation is generated on resources for the acquisition, transmission and storage of dirty data which does not belong to the service; and the utilization of transmission resources and the guarantee of breakpoint continuous transmission after transmission interruption are carried out during data transmission, and the waste of invalid data on storage space and the influence of hardware resources such as a disk, a memory and the like on retrieval efficiency are eliminated during data storage.
In order to solve the resource waste caused by frequent interaction between the user mode and the kernel mode during collection, the patent CN115604207A mirrors network traffic in a way of switch port mirroring or optical fiber splitting, and then carries out traffic collection through an open source protocol such as PF_RING, and then reduces the influence of disk IO on collection efficiency in a conversation ID way. The above-mentioned way of introducing network traffic through port mirroring or fiber splitting; firstly, the port of a switch is occupied, and meanwhile, the partial network traffic is lost for convenience of the network without coverage or wiring conditions; secondly, the port mirroring mode needs to modify the configuration of the switch, and at this time, the performance of the switch or a service system may be affected inevitably; thirdly, if the total flow is collected in a mirror image mode, when the network flow is large, the switch is overloaded, the performance is reduced, and the service continuity and stability are further affected; finally, most of the images can perform irregular filtration on the data packet, if full flow retention analysis cannot be performed, the situation of no evidence checking can possibly occur in the post analysis and evidence obtaining links of the network security event. The patent CN111490976a is mainly directed to a traditional network flow collection mode of an industrial control environment, but still collects full flow during collection, performs analysis work on data again after data preprocessing, and combs out data deviating from a baseline mask by taking a service system as a unit for network flow data, thereby forming an alarm. The industrial control environment collects network flow; firstly, compared with complex network environments of multi-network integration such as Internet, office network, production network and the like, the industrial control environment has more rules in data form, has no characteristics of large data volume and multiple types, and is easy to "lead into a room" when a network link is erected in a bidirectional manner in the safer and closed industrial control environment; secondly, the business systems of most enterprises are not single independent, and frequent node changes or frequent inter-call among interfaces can cause inaccurate and effective alarming generated by a base line, so that false alarm is greatly increased; thirdly, when the baseline data are combed, the baseline communication flow data in a fixed time period or a certain time period are inevitably mixed into abnormal data through machine learning, and if the baseline communication flow data are subjected to manual combing in front of a large amount of data, the labor cost is greatly increased; finally, under the condition that the base line is difficult to comb and constantly changes, alarm data is formed through the data deviating from the base line mask, more missed reports and false alarms are brought, and on the aspect of processing the missed reports and false alarms, the analysis and processing cost of management personnel is increased, so that real and effective alarms are omitted.
Disclosure of Invention
The embodiment of the application aims to provide a network flow data processing method and system based on service types, which avoid network delay and memory consumption caused by switching between a user mode and a kernel mode of a general bypass flow method and influence of a large amount of dirty data on efficiency in acquisition, transmission, storage and data analysis by analyzing a data packet and marking the data packet in a mode of combining a service session control sequence number with a timestamp; the service sequence numbers are marked for full-flow preprocessing, so that network flow data can be efficiently stored, transmitted and indexed.
In order to solve the above technical problem, a first aspect of an embodiment of the present application provides a method for processing network traffic data based on a service type, including the following steps:
collecting network flow data and carrying out full protocol analysis on the network flow data;
based on a business behavior learning model, carrying out partition preprocessing on the analyzed network traffic data according to a business region and a non-business region, marking a business sequence number, and establishing a segmentation index on the partitioned data according to the business sequence number and a time stamp;
the network flow data are stored in a distributed mode according to the service serial numbers and the time stamps;
the data is exported according to a preset format or sent to a designated data receiving system.
Further, before the analyzing the network traffic data based on the business behavior learning model and performing the partition preprocessing on the network traffic data according to the business area and the non-business area, the method comprises the following steps:
constructing a business behavior learning model, forming a normal communication session control library of the business region, and marking sequence numbers through business session control sequence numbers;
and marking the communication behavior of the non-service area as a default index library, and marking the sequence number through the service session control sequence number.
Further, after establishing the segment index according to the service session control sequence number and the time stamp, the method includes:
dividing the analyzed network flow data into a plurality of preset levels according to risk levels.
Further, the collecting network traffic data includes:
and collecting the network flow data mirror image through an FP_RING protocol.
Further, before the data is exported according to the preset format or sent to the designated data receiving system, the method includes:
judging whether network transmission is normal or not in a form of sending heartbeat packets;
if the network transmission is normal, the network flow data are sequentially sent according to the time stamp queue sequence in a segmented form;
if the network transmission is abnormal, continuing to send the untransmitted data at the interruption time point after the network transmission is normal.
Accordingly, a second aspect of an embodiment of the present application provides a network traffic data processing system based on a service type, including:
the data acquisition module is used for acquiring network flow data and carrying out full protocol analysis on the network flow data;
the data processing module is used for carrying out partition preprocessing on the analyzed network flow data according to the service area and the non-service area based on the service behavior learning model, marking service serial numbers, and establishing a segmentation index on the partitioned data according to the service serial numbers and the time stamps;
the data storage module is used for carrying out distributed storage on the network flow data according to the service serial numbers and the time stamps;
and the data transmitting module is used for exporting data according to a preset format or transmitting the data to a specified data receiving system.
Further, the network traffic data processing system based on the service type further comprises:
the model construction module is used for constructing a business behavior learning model, forming a normal communication session control library of the business region and marking a serial number through a business session control serial number;
the model building module marks the communication behavior of the non-service area as a default index library and marks the sequence number through the service session control sequence number.
Further, the data processing module is further configured to divide the parsed network traffic data into a plurality of preset levels according to risk levels.
Further, the data acquisition module acquires the network traffic data mirror image through an FP_RING protocol.
Further, the network traffic data processing system based on the service type further comprises: the network judging module comprises:
a network judging unit for judging whether the network transmission is normal in the form of sending a heartbeat packet;
the network control unit is used for sequentially transmitting the network flow data according to the time stamp queue sequence in a segmented form when the network transmission is normal;
the network control unit is further configured to, when the network transmission is abnormal, continue to send untransmitted data at an interruption time point after the network transmission is normal.
Accordingly, a third aspect of the embodiment of the present application provides an electronic device, including: at least one processor; and a memory coupled to the at least one processor; wherein the memory stores instructions executable by the one processor to cause the at least one processor to perform the traffic type-based network traffic data processing method described above.
Accordingly, a fourth aspect of embodiments of the present application provides a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the above-described method for processing network traffic data based on traffic type.
The technical scheme provided by the embodiment of the application has the following beneficial technical effects:
by analyzing the data packet and marking the data packet in a mode of combining the service session control sequence number and the timestamp on the basis of collecting the data packet by using the PF_RING protocol, network delay and memory consumption caused by switching between a user mode and a kernel mode of a general bypass flow method are avoided, and the influence of a large amount of dirty data on efficiency in collection, transmission, storage and data analysis is avoided; the service session control sequence numbers are marked for full-flow preprocessing, so that network flow data can be efficiently stored, transmitted and indexed.
Drawings
Fig. 1 is a flowchart of a method for processing network traffic data based on a service type according to an embodiment of the present application;
FIG. 2 is a block diagram of a network traffic data processing system based on traffic type according to an embodiment of the present application;
fig. 3 is a block diagram of a network judgment module according to an embodiment of the present application.
Reference numerals:
1. the system comprises a data acquisition module, a data processing module, a data storage module, a data transmission module, a model construction module, a network judgment unit, a network control unit and a network control unit.
Detailed Description
The objects, technical solutions and advantages of the present application will become more apparent by the following detailed description of the present application with reference to the accompanying drawings. It should be understood that the description is only illustrative and is not intended to limit the scope of the application. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the present application.
Referring to fig. 1, a first aspect of the embodiment of the present application provides a method for processing network traffic data based on a service type, including the following steps:
step S100, collecting network flow data and carrying out full protocol analysis on the network flow data.
Step S300, based on the business behavior learning model, the analyzed network flow data is subjected to partition preprocessing according to the business area and the non-business area, business serial numbers are marked, and a segmentation index is built for the partitioned data according to the business serial numbers and the time stamps.
Step S500, the network flow data is stored in a distributed mode according to the service serial numbers and the time stamps.
Step S700, exporting data according to a preset format or sending the exported data to a specified data receiving system.
In the prior art, unpacking, analysis, recombination and other operations are performed on the network data after the network data is acquired, and the technical scheme is that data preprocessing is added after a network traffic data acquisition stage, and a full data index with a service serial number and a timestamp as index serial numbers is established, so that not only is the fine granularity of session restoration increased, but also the efficiency of data re-analyzed and utilized is greatly increased, and meanwhile, the efficiency of transmission is also increased due to the differentiation of invalid data; in addition, for third party maintenance personnel of the system, only the log efficiency of the system is audited to be greatly increased.
Further, before the analyzing the network traffic data based on the business behavior learning model in step S300 performs the partition preprocessing on the business area and the non-business area, the method includes:
step S200, a business behavior learning model is built, a normal communication session control library of a business area is formed, and sequence number marking is carried out through a business session control sequence number (business session sequence number); and marking the communication behavior of the non-service area as a default index library, and marking the sequence number through the service session control sequence number.
Further, after establishing the segment index according to the service session control sequence number and the timestamp, the step S300 includes:
step S310, dividing the analyzed network flow data into a plurality of preset levels according to the risk levels. First, when collecting network data, the source, content, usage and the like of the data are classified for the data of different service systems by a service analysis model. For different levels of service systems that need protection, the data of important service systems is also relatively higher. Through value analysis of data, sensitivity degree analysis of content, range analysis of data influence and range analysis of data distribution, when analyzing different business system data, business marking can be carried out on the data, for example, a grade protection 2.0 three-level system is marked as C class, a grade protection 2.0 two-level system is marked as B class, and sensitive data in the business system, such as sensitive information of personal identity information, family residence, contact information, bank card account number and the like, can be marked as M1, and identity authentication information of a user account number, password and the like is marked as M2. In addition, other data information in the network traffic is also marked. Meanwhile, for the same service area, data with different categories and different levels of the same data segment can automatically learn weights, and the data segment is classified, graded and scored and calculated, specifically as follows:
classification score = c1×service area 1+c2×service area 2+b1×service area 3;
grading and scoring: m1×the sensitive data of the level of the segment/the sensitive data of the segment+m2×the sensitive data of the level of the segment/the sensitive data of the segment+m3..a.and so on;
risk rating score = classification score x score validity factor + classification score x score validity factor for each piece of data;
finally, according to the risk grade scores of each piece of data, carrying out risk grade division on each piece of data.
Further, collecting network traffic data in step S100 includes: and collecting network traffic data mirror images through the FP_RING protocol.
In the prior art, network traffic is collected through sniffer, tcpdump or wireshare, and the consumption of resources by the traditional method is relatively high, so that the collection flow is not efficient by adopting the FP_RING protocol in the patent. And collecting network flow data mirror image by adopting an FP_RING protocol, wherein the transfer_mode is set to 2, and the packet is only copied to the FP_RI NG by a driver without kernel processing.
Further, before exporting data according to a preset format or sending the data to a specified data receiving system in step S700, the method includes:
step S610, judging whether the network transmission is normal in the form of sending heartbeat packets;
step S620, if the network transmission is normal, the network flow data are sequentially transmitted according to the time stamp queue sequence in a segmented form;
in step S630, if the network transmission is abnormal, the data that is not transmitted is continuously transmitted at the interruption time point after the network transmission is normal.
In the data transmission process, firstly, judging whether network transmission is normal in a form of sending heartbeat packets; if the network is normal, firstly, the buffered data are sequentially sent in a segmented form by taking the time stamp as a queue, and if the network is not connected, the network is waited to be connected, and then, the data which are not transmitted are continuously sent at the time point of memory interruption. By means of the data segmentation mode, the instant congestion of the network caused by the transmission of large data packets is avoided, and meanwhile, the high bandwidth occupation caused by the large concurrent transmission is avoided.
Three docking modes are provided for a specified data receiving system (i.e., a third party platform): 1. establishing a full index query interface, and passively querying by a third party platform; 2. actively transmitting the data to a designated platform in a syslog form; 3. the export file form is imported to a third party platform.
The application is mainly used for solving the following problems of the traditional network flow analysis system:
firstly, most manufacturers announce that network data are completely saved, most manufacturers adopt traditional protocols to calculate the traffic data according to high-frequency equipment, and the storage efficiency of the network traffic is controlled by collecting the frequency, so that the method cannot always completely save the network traffic data; secondly, the traditional method is not used for effectively cleaning the data during the operations of data acquisition, unpacking, reorganization and the like, and screening out invalid data, so that the waste of resources is avoided; thirdly, in the network data transmission process, if network connection is interrupted, the traditional method can cause data loss, and the full data retention can not be ensured; finally, when the traditional method is used for storage, the analysis efficiency is increased only through buffering, and the analysis efficiency is increased through establishing an index.
In the technical scheme of the application, firstly, a business analysis baseline is established during data preprocessing, a full-disc index is established according to a business serial number and a time stamp, and meanwhile, the data is classified and graded; secondly, the network connectivity is confirmed in the form of sending heartbeat packets in the data transmission process. When the network connectivity is abnormal, firstly caching data by using a service index sequence number and a time stamp, establishing a pre-transmission queue, and continuing to transmit the data after the network connectivity; finally, the data segment 200M is stored in one segment, wherein each segment of data is divided into a plurality of service indexes, each service index comprises a plurality of protocols, the source IP, the destination IP, the source port, the destination port, the protocols and the like analyzed by each session are subjected to threat score identification, and if the source IP is a public network address and the destination port is a high-risk port such as 80, the threat score is higher than that of the normal intranet service communication.
By analyzing the data packet and marking the data packet in a mode of combining the service session control sequence number and the timestamp on the basis of collecting the data packet by using the PF_RING protocol, network delay and memory consumption caused by switching between a user mode and a kernel mode of a general bypass flow method are avoided, and the influence of a large amount of dirty data on efficiency in collection, transmission, storage and data analysis is avoided; the service session sequence numbers are marked for full-flow preprocessing, so that network flow data can be efficiently stored, transmitted and indexed.
Accordingly, referring to fig. 2, a second aspect of the embodiment of the present application provides a network traffic data processing system based on a service type, including:
the data acquisition module 1 is used for acquiring network flow data and carrying out full protocol analysis on the network flow data;
the data processing module 2 is used for carrying out partition preprocessing on the analyzed network flow data according to the service area and the non-service area based on the service behavior learning model, marking service serial numbers, and establishing a segmentation index on the partitioned data according to the service serial numbers and the time stamps;
the data storage module 3 is used for carrying out distributed storage on the network traffic data according to the service serial numbers and the time stamps;
and the data transmitting module 4 is used for exporting data according to a preset format or transmitting the data to a specified data receiving system.
Further, the network traffic data processing system based on the service type further comprises:
the model building module 5 is used for building a business behavior learning model, forming a normal communication session control library of a business area and marking a serial number through a business session control serial number;
the model building module 5 also marks the communication behavior of the non-service area as a default index library, and marks the sequence number through the service session control sequence number.
Further, the data processing module 2 is further configured to divide the parsed network traffic data into a plurality of preset levels according to risk levels.
Further, the data acquisition module 1 acquires the network traffic data mirror image through the fp_ring protocol.
Further, referring to fig. 3, the network traffic data processing system based on the service type further includes: the network judgment module 6, the network judgment module 6 includes:
a network judging unit 61 for judging whether or not the network transmission is normal in the form of sending a heartbeat packet;
a network control unit 62, configured to sequentially send network traffic data according to a time stamp queue order in a segmented form when network transmission is normal;
the network control unit 62 is further configured to, when the network transmission is abnormal, continue to send the untransmitted data at the interruption time point after the network transmission is normal.
Accordingly, a third aspect of the embodiment of the present application provides an electronic device, including: at least one processor; and a memory coupled to the at least one processor; the memory stores instructions executable by a processor, the instructions being executable by the processor, to cause the at least one processor to perform the method for processing network traffic data based on traffic type.
Accordingly, a fourth aspect of embodiments of the present application provides a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the above-described method for processing network traffic data based on traffic type.
The embodiment of the application aims to protect a network traffic data processing method and system based on service types, and the method comprises the following steps: collecting network flow data and carrying out full protocol analysis on the network flow data; based on a business behavior learning model, carrying out partition preprocessing on the analyzed network flow data according to a business region and a non-business region, marking a business sequence number, and establishing a segmentation index on the partitioned data according to the business sequence number and a time stamp; the network flow data is stored in a distributed mode according to the service serial numbers and the time stamps; the data is exported according to a preset format or sent to a designated data receiving system. The technical scheme has the following effects:
by analyzing the data packet and marking the data packet in a mode of combining the service session control sequence number and the timestamp on the basis of collecting the data packet by using the PF_RING protocol, network delay and memory consumption caused by switching between a user mode and a kernel mode of a general bypass flow method are avoided, and the influence of a large amount of dirty data on efficiency in collection, transmission, storage and data analysis is avoided; the service session sequence numbers are marked for full-flow preprocessing, so that network flow data can be efficiently stored, transmitted and indexed.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical aspects of the present application and not for limiting the same, and although the present application has been described in detail with reference to the above embodiments, it should be understood by those of ordinary skill in the art that: modifications and equivalents may be made to the specific embodiments of the application without departing from the spirit and scope of the application, which is intended to be covered by the claims.
Claims (10)
1. The network flow data processing method based on the service type is characterized by comprising the following steps:
collecting network flow data and carrying out full protocol analysis on the network flow data;
based on a business behavior learning model, carrying out partition preprocessing on the analyzed network traffic data according to a business region and a non-business region, marking a business sequence number, and establishing a segmentation index on the partitioned data according to the business sequence number and a time stamp;
the network flow data are stored in a distributed mode according to the service serial numbers and the time stamps;
the data is exported according to a preset format or sent to a designated data receiving system.
2. The method for processing network traffic data based on service type according to claim 1, wherein before the analyzing the network traffic data based on the service behavior learning model performs the partition preprocessing according to the service area and the non-service area, the method comprises:
constructing a business behavior learning model, forming a normal communication session control library of the business area, and marking serial numbers through the business serial numbers;
and marking the communication behavior of the non-service area as a default index library, and marking the sequence number through the service sequence number.
3. The method for processing network traffic data based on service type according to claim 1, wherein after establishing the segment index according to the service sequence number and the time stamp, the method comprises:
dividing the partitioned network flow data into a plurality of preset levels according to risk levels.
4. The method for processing network traffic data based on service type according to claim 1, wherein the collecting network traffic data comprises:
and collecting the network flow data mirror image through an FP_RING protocol.
5. The method for processing network traffic data based on service type according to any one of claims 1 to 4, wherein before the data is exported according to a preset format or sent to a designated data receiving system, the method comprises:
judging whether network transmission is normal or not in a form of sending heartbeat packets;
if the network transmission is normal, the network flow data are sequentially sent according to the time stamp queue sequence in a segmented form;
if the network transmission is abnormal, continuing to send the untransmitted data at the interruption time point after the network transmission is normal.
6. A network traffic data processing system based on traffic type, comprising:
the data acquisition module is used for acquiring network flow data and carrying out full protocol analysis on the network flow data;
the data processing module is used for carrying out partition preprocessing on the analyzed network flow data according to the service area and the non-service area based on the service behavior learning model, marking service serial numbers, and establishing a segmentation index on the partitioned data according to the service serial numbers and the time stamps;
the data storage module is used for carrying out distributed storage on the network flow data according to the service serial numbers and the time stamps;
and the data transmitting module is used for exporting data according to a preset format or transmitting the data to a specified data receiving system.
7. The traffic-type based network traffic data processing system of claim 6, further comprising:
the model construction module is used for constructing a business behavior learning model, forming a normal communication session control library of the business area and marking serial numbers through the business serial numbers;
the model building module marks the communication behavior of the non-service area as a default index library and marks the sequence number through the service sequence number.
8. The traffic-type based network traffic data processing system of claim 6 wherein,
the data processing module is further used for dividing the partitioned network flow data into a plurality of preset levels according to risk levels.
9. The traffic-type based network traffic data processing system of claim 6 wherein,
and the data acquisition module acquires the network flow data mirror image through an FP_RING protocol.
10. A traffic-type based network traffic data processing system according to any of claims 6-9, further comprising: the network judging module comprises:
a network judging unit for judging whether the network transmission is normal in the form of sending a heartbeat packet;
the network control unit is used for sequentially transmitting the network flow data according to the time stamp queue sequence in a segmented form when the network transmission is normal;
the network control unit is further configured to, when the network transmission is abnormal, continue to send untransmitted data at an interruption time point after the network transmission is normal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310409270.7A CN116471212B (en) | 2023-04-17 | 2023-04-17 | Service type-based network traffic data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310409270.7A CN116471212B (en) | 2023-04-17 | 2023-04-17 | Service type-based network traffic data processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116471212A CN116471212A (en) | 2023-07-21 |
| CN116471212B true CN116471212B (en) | 2023-11-14 |
Family
ID=87178455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310409270.7A Active CN116471212B (en) | 2023-04-17 | 2023-04-17 | Service type-based network traffic data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116471212B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1747581A (en) * | 2004-09-07 | 2006-03-15 | 三星电子株式会社 | The method and apparatus of portable terminal notice customer service district and type of service |
| US7979439B1 (en) * | 2006-03-14 | 2011-07-12 | Amazon Technologies, Inc. | Method and system for collecting and analyzing time-series data |
| CN108259371A (en) * | 2016-12-28 | 2018-07-06 | 亿阳信通股份有限公司 | A kind of network flow data analysis method and device based on stream process |
| CN115550034A (en) * | 2022-09-29 | 2022-12-30 | 国网重庆市电力公司电力科学研究院 | Service flow monitoring method and device for distribution network power monitoring system |
| CN115914022A (en) * | 2022-11-22 | 2023-04-04 | 山西合力创新科技股份有限公司 | Application relation analysis method, system, equipment and medium based on network flow |
-
2023
- 2023-04-17 CN CN202310409270.7A patent/CN116471212B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1747581A (en) * | 2004-09-07 | 2006-03-15 | 三星电子株式会社 | The method and apparatus of portable terminal notice customer service district and type of service |
| US7979439B1 (en) * | 2006-03-14 | 2011-07-12 | Amazon Technologies, Inc. | Method and system for collecting and analyzing time-series data |
| CN108259371A (en) * | 2016-12-28 | 2018-07-06 | 亿阳信通股份有限公司 | A kind of network flow data analysis method and device based on stream process |
| CN115550034A (en) * | 2022-09-29 | 2022-12-30 | 国网重庆市电力公司电力科学研究院 | Service flow monitoring method and device for distribution network power monitoring system |
| CN115914022A (en) * | 2022-11-22 | 2023-04-04 | 山西合力创新科技股份有限公司 | Application relation analysis method, system, equipment and medium based on network flow |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116471212A (en) | 2023-07-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114584405B (en) | Electric power terminal safety protection method and system | |
| CN109271793B (en) | Internet of things cloud platform equipment category identification method and system | |
| CN1866951B (en) | Method and system for detecting shared access host machine in network | |
| CN108337652B (en) | Method and device for detecting flow fraud | |
| CN109391613A (en) | A kind of intelligent substation method for auditing safely based on SCD parsing | |
| CN110401624A (en) | The detection method and system of source net G system mutual message exception | |
| CN115134099B (en) | Network attack behavior analysis method and device based on full flow | |
| CN111049843A (en) | Intelligent substation network abnormal flow analysis method | |
| CN100362805C (en) | Multifunctional management system for detecting erotic images and unhealthy information in network | |
| CN111654486A (en) | Server equipment judgment and identification method | |
| CN113141368A (en) | System supporting real-time security threat association analysis of mass data | |
| CN115378711A (en) | Industrial control network intrusion detection method and system | |
| CN111131332A (en) | Network service interconnection and flow acquisition, analysis and recording system | |
| CN110365673B (en) | Method, server and system for isolating network attack plane | |
| CN116471212B (en) | Service type-based network traffic data processing method and system | |
| CN115766471B (en) | Network service quality analysis method based on multicast flow | |
| CN111490976A (en) | Dynamic baseline management and monitoring method for industrial control network | |
| CN109190408B (en) | Data information security processing method and system | |
| Brahmi et al. | A Snort-based mobile agent for a distributed intrusion detection system | |
| CN115442159A (en) | Household routing-based risk management and control method, system and storage medium | |
| CN109309679A (en) | A kind of Network scan detection method and detection system based on TCP flow state | |
| CN113792076A (en) | Data auditing system | |
| CN112887316A (en) | Access control list conflict detection system and method based on classification | |
| CN112134845A (en) | Rejection service system | |
| CN118054957B (en) | Computer network security analysis system based on security signal matching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |