CN117640257A - Data processing method and system for network security operation based on big data - Google Patents
Data processing method and system for network security operation based on big data Download PDFInfo
- Publication number
- CN117640257A CN117640257A CN202410103156.6A CN202410103156A CN117640257A CN 117640257 A CN117640257 A CN 117640257A CN 202410103156 A CN202410103156 A CN 202410103156A CN 117640257 A CN117640257 A CN 117640257A
- Authority
- CN
- China
- Prior art keywords
- flow
- data set
- address
- data
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 9
- 230000005856 abnormality Effects 0.000 claims abstract description 58
- 238000007781 pre-processing Methods 0.000 claims abstract description 13
- 230000002159 abnormal effect Effects 0.000 claims description 72
- 238000012545 processing Methods 0.000 claims description 35
- 238000011156 evaluation Methods 0.000 claims description 30
- 230000005540 biological transmission Effects 0.000 claims description 27
- 238000012544 monitoring process Methods 0.000 claims description 24
- 238000004364 calculation method Methods 0.000 claims description 18
- 238000007405 data analysis Methods 0.000 claims description 18
- 230000004044 response Effects 0.000 claims description 16
- 238000010606 normalization Methods 0.000 claims description 15
- 238000004891 communication Methods 0.000 claims description 13
- 238000012937 correction Methods 0.000 claims description 12
- 238000001514 detection method Methods 0.000 claims description 11
- 238000000034 method Methods 0.000 claims description 10
- 238000005111 flow chemistry technique Methods 0.000 claims description 9
- 241000700605 Viruses Species 0.000 claims description 6
- 230000010354 integration Effects 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000012550 audit Methods 0.000 claims description 3
- 230000000903 blocking effect Effects 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 230000001681 protective effect Effects 0.000 claims description 3
- 230000002596 correlated effect Effects 0.000 abstract description 5
- 238000007796 conventional method Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 2
- 238000001994 activation Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000000875 corresponding effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000009792 diffusion process Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 102000003712 Complement factor B Human genes 0.000 description 1
- 108090000056 Complement factor B Proteins 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000002547 anomalous effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a data processing method and a system for network security operation based on big data, which relate to the technical field of network security, and the system is operated by collecting real-time network flow data and IP address related information to form a flow data set and an IP address data set, preprocessing the flow data set and the IP address data set to obtain flow related information to form a first data set, obtaining IP address related information to form a second data set, matching the flow related information and the IP address related information to obtain one-to-many or one-to-one relation matching quantity of IP addresses, recording to form a third data set, and then calculating to obtain: the abnormality index Yczs is matched with a preset abnormality alert threshold Y, an abnormality level policy scheme is obtained, specific execution is performed, and when the traffic abnormality and the IP abnormality are correlated, the traffic abnormality can be effectively monitored, and real-time protection measures can be made.
Description
Technical Field
The invention relates to the technical field of network security, in particular to a data processing method and system for network security operation based on big data.
Background
Network security operation is one of important fields in the brand-new angle along with the rapid development of the Internet, in the wide field, the application of big data technology gradually becomes an indispensable tool for solving network threat and protecting information security, and the application of big data in the network security field is not only for storing and processing massive data, but also for providing comprehensive network security situation awareness by deeply analyzing network traffic, user behaviors and threat information.
However, in conventional network security operations, monitoring and responding to abnormal traffic and IP are often limited and insufficient, and conventional methods often rely on limited data and simple rules, so that potential threats contained in a large amount of network data cannot be fully mined, and in particular, when a situation that traffic abnormality and IP abnormality are correlated is faced, conventional means are worry, for example, a network attack confuses conventional detection rules by simulating normal traffic, so that conventional methods cannot quickly and accurately identify abnormal requests and IP visitors, and such situations cause network security operations to lack flexibility and instantaneity when facing evolving network threats.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a data processing method and a system for network security operation based on big data, which solve the problems in the background art.
In order to achieve the above purpose, the invention is realized by the following technical scheme: the system comprises a data processing system for network security operation based on big data, a flow acquisition module, an IP acquisition module, a data analysis module, a detection module, an evaluation module and a response module;
the flow acquisition module monitors network flow information to acquire real-time network flow data and form a flow data set;
the IP acquisition module monitors network communication content to acquire real-time IP address related information to form an IP address data set;
the data analysis module preprocesses the flow data set and the IP address data set to obtain flow related information, forms a first data set, obtains IP address related information, forms a second data set, matches the flow related information with the IP address related information, obtains one-to-many or one-to-one relation matching quantity of the IP addresses, and records the relation matching quantity to form a third data set;
the monitoring module performs normalization processing on the first data set, the second data set and the third data set, performs calculation, and obtains: abnormality index Yczs;
the abnormality index Yczs is obtained by the following formula:
；
wherein Llxs represents a network flow coefficient, fwxs represents an address access coefficient, A and B represent proportionality coefficients of the network flow coefficient Llxs and the address access coefficient Fwxs respectively, and C represents a first correction constant;
the network flow coefficient Llxs is obtained through calculation of a first data set and a third data set, and is compared with a preset network flow threshold L to obtain a network flow evaluation scheme;
the address access coefficient Fwxs is obtained through calculation of the second data set and the third data set, and is compared with a preset address access threshold Z to obtain an address access evaluation scheme;
the evaluation module is matched with the preset abnormal alert threshold Y and the abnormal index Yczs to acquire an abnormal level strategy scheme;
and the response module performs specific execution through the content of the abnormal level strategy scheme.
Preferably, the flow acquisition module comprises a monitoring unit and an integration unit;
the monitoring unit is used for monitoring data packets in network communication in real time, extracting relevant information in the traffic packets, including source addresses, target addresses, data packet lengths, time stamps, the number of the data packets, loads, service types and port numbers, and marking the acquired information;
the integration unit integrates the marked flow packet information to form a structured data form so as to form a flow data set.
Preferably, the IP acquisition module includes a classification unit and an extraction unit;
the classifying unit classifies the protocol field and the data packet type in the protocol network communication content to obtain the IP address related information, wherein the protocol field comprises: TCP, UDP, and ICMP, packet types include: HTTP, FTP, and DNS;
the extracting unit extracts the classified IP address related information to obtain: the source IP address, the target IP address, the number of active times, the time stamp information, the transmission rate, the data packet quantity value and the connection duration form an IP address data group.
Preferably, the data analysis module comprises a flow processing unit, an IP processing unit and a matching association unit;
the flow processing unit performs checksum preprocessing on the flow data set to obtain flow packet related information, and the flow processing unit forms a first data set, and comprises: packet length Sjb, traffic transmission duration Llsc, load Sjfz, and traffic packet number Llb;
the IP processing unit performs checksum preprocessing on the IP address data set to acquire IP address related information, and forms a second data set, and the method comprises the following steps: address packet Dzb, connection duration Ljsc, and active number Hycs;
the source address and the target address in the flow data set and the IP address data set of the matching association unit are matched, and an association relation between the flow and the IP address is established and marked, wherein the association relation comprises the following steps: one-to-many and one-to-one, recording the number of incidence relations of the marks, and integrating the number of incidence relations and the frequency value of occurrence of the same IP address to form a third data set, wherein the third data set comprises: the IP traffic relation quantity Gxsl and the IP traffic frequency value Plz.
Preferably, the monitoring module comprises a normalization unit and a calculation unit;
the normalization unit performs normalization processing on the first data set, the second data set and the third data set to enable the first data set, the second data set and the third data set to be in the same dimension;
the computing unit performs first computation on the normalized first data set, the normalized second data set and the normalized third data set to obtain: and (3) carrying out second calculation on the network flow coefficient Llxs and the address access coefficient Fwxs to obtain: abnormality index Yczs.
Preferably, the network traffic coefficient Llxs is obtained by the following formula:
；
in the formula, the absolute ratio value of the data packet length Sjb to the flow transmission duration Llsc, the absolute ratio value of the load Sjfz to the flow transmission duration Llsc, the absolute ratio value of the flow data packet quantity Llb to the flow transmission duration Llsc are calculated, and then the calculated value is compared with the calculated result of the IP flow relation quantity Gxsl and the IP flow frequency value Plz to obtain a network flow coefficient Llxs, wherein E represents a second correction constant;
and, the network flow coefficient Llxs is compared with a preset network flow threshold L, and a network flow evaluation scheme is obtained:
the network flow coefficient Llxs is smaller than the network flow threshold L, and the network flow transmission is not abnormal;
the network flow coefficient Llxs is more than or equal to the network flow threshold L, the network flow transmission is abnormal, the full checking and killing are carried out, and the abnormal flow packet is deleted and marked.
Preferably, the address access coefficient Fwxs is obtained by:
；
in the formula, the absolute ratio value of the address data packet Dzb to the connection duration Ljsc is used for calculating the active times Hycs to the connection duration Ljsc, and then comparing the calculated result with the calculated result of the IP flow relation quantity Gxsl and the IP flow frequency value Plz to obtain an address access coefficient Fwxs, wherein F represents a third correction constant;
and, address access coefficient Fwxs is compared with a preset address access threshold Z, and an address access evaluation scheme is obtained:
the address access coefficient Fwxs is smaller than the address access threshold Z, and the access address is not abnormal;
the address access coefficient Fwxs is more than or equal to the address access threshold Z, the access address is abnormal, the abnormal address is marked and a blacklist is added, and meanwhile, the address of the IP section is set to be the upper limit of the flow.
Preferably, the evaluation module comprises a threshold storage unit and a matching unit;
the threshold storage unit is used for storing an abnormal alert threshold Y, a network flow threshold L, an address access threshold Z, an abnormal level policy scheme, a network flow evaluation scheme, an address access evaluation scheme and the contact modes of related notification personnel;
the matching unit compares the abnormality index Yczs with an abnormality alert threshold Y to obtain an abnormality level policy scheme:
the abnormality index Yczs is less than the abnormality alert threshold Y, and the traffic packet and the source IP address are not abnormal;
the abnormality index Yczs is more than or equal to an abnormality warning threshold Y, the flow packet and the source IP address are abnormal, virus searching and killing are carried out on the flow packet, a blacklist is added to the source IP address contained in the flow packet, the upper flow limit is increased by the IP section, and meanwhile workers are informed of tracing or searching and killing the flow packet.
Preferably, the response module comprises an execution unit and a recording unit;
the execution unit executes corresponding predefined operations and informs related personnel according to specific protective measures in the abnormal level strategy scheme content, wherein the predefined operations comprise: blocking the source IP of attack, increasing the defending level, triggering alarm notification and limiting the upper and lower limits of traffic, wherein the notification modes comprise: broadcasting, short messages, presetting call recording and internal application communication;
the recording unit records log information generated in the execution process and is used for post audit and analysis, and the recorded information comprises: secure operation, execution time, execution result, and notification personnel.
A data processing method for network security operation based on big data comprises the following steps:
step one: acquiring real-time network flow data through a flow acquisition module to form a flow data set;
step two: acquiring real-time IP address related information through an IP acquisition module to form an IP address data set;
step three: preprocessing a flow data set and an IP address data set through a data analysis module to obtain a first data set, a second data set and a third data set;
step four: and carrying out normalization processing on the first data set, the second data set and the third data set through a monitoring module, and calculating to obtain: abnormality index Yczs;
step five: matching the abnormality index Yczs with a preset abnormality warning threshold Y through an evaluation module to acquire an abnormality grade strategy scheme;
step six: and the response module is used for executing the specific execution of the abnormal level strategy scheme content.
The invention provides a data processing method and a system for network security operation based on big data, which have the following beneficial effects:
(1) When the system operates, real-time network flow data and IP address related information are acquired through a flow acquisition module and an IP acquisition module, a flow data set and an IP address data set are formed, preprocessing is performed through a data analysis module, flow related information is acquired, a first data set is formed, IP address related information is acquired, a second data set is formed, the flow related information and the IP address related information are matched through the data analysis module, one-to-many or one-to-one relation matching quantity of the IP addresses is acquired, and is recorded, a third data set is formed, and the first data set, the second data set and the third data set are calculated through a monitoring module, so that the system is obtained: the anomaly index Yczs is matched with a preset anomaly alert threshold Y to obtain an anomaly level strategy scheme, and finally, the content of the anomaly level strategy scheme is specifically executed through a response module, so that under the condition that flow anomalies and IP anomalies are correlated with each other, real-time protection measures can be effectively monitored, and through more comprehensive data analysis and multidimensional matching, the perception level of network threat is improved, and the overall level of network security operation is improved.
(2) By judging whether the network traffic transmission is abnormal, the real-time monitoring of the network traffic state is realized, the abnormal detection precision of the network traffic transmission and the identification of various abnormal states are improved, the access and the times of the abnormal addresses are more flexibly identified through the detection of the related information of the IP addresses, the control of the abnormal addresses is further enhanced through the modes of marking and adding a blacklist and setting upper and lower limits, finally, a comprehensive abnormal level strategy scheme is provided, the secondary detection of the overall network traffic and the IP is carried out, and meanwhile, the automatic predefined protection measures are provided, so that the diffusion and the occurrence of the abnormal states are controlled at the first time.
(3) In the invention, the real-time network flow data is acquired to form a flow data set, the real-time IP address related information is formed into an IP address data set, the IP address data set is preprocessed to form a first data set and a second data set, the matching number of the IP address relation is acquired, the matching number is recorded, a third data set is formed, and the calculation is performed to obtain: the anomaly index Yczs is matched with a preset anomaly alert threshold Y at the same time, an anomaly level strategy scheme is obtained, specific execution is carried out according to the content of the anomaly level strategy scheme, the matching of the flow related information and the IP address related information is carried out, and the generated anomaly level strategy scheme provides guidelines for subsequent response, so that the purposes of improving network safety and property safety are achieved, the use condition of manpower and material resources is reduced, and the processing efficiency of network safety information is improved.
Drawings
FIG. 1 is a block diagram of a data processing system for secure operation of a network based on big data according to the present invention;
fig. 2 is a schematic diagram of steps of a data processing method for network security operation based on big data in the present invention.
Detailed Description
The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments, and all other embodiments obtained by those skilled in the art without making any inventive effort based on the embodiments of the present invention are within the scope of protection of the present invention.
Network security operation is one of important fields in the brand-new angle along with the rapid development of the Internet, in the wide field, the application of big data technology gradually becomes an indispensable tool for solving network threat and protecting information security, and the application of big data in the network security field is not only for storing and processing massive data, but also for providing comprehensive network security situation awareness by deeply analyzing network traffic, user behaviors and threat information.
However, in conventional network security operations, monitoring and responding to abnormal traffic and IP are often limited and insufficient, and conventional methods often rely on limited data and simple rules, so that potential threats contained in a large amount of network data cannot be fully mined, and in particular, when a situation that traffic abnormality and IP abnormality are correlated is faced, conventional means are worry, for example, a network attack confuses conventional detection rules by simulating normal traffic, so that conventional methods cannot quickly and accurately identify abnormal requests and IP visitors, and such situations cause network security operations to lack flexibility and instantaneity when facing evolving network threats.
Examples
The invention provides a data processing system for network security operation based on big data, referring to fig. 1, a flow acquisition module, an IP acquisition module, a data analysis module, a detection module, an evaluation module and a response module;
the flow acquisition module monitors network flow information to acquire real-time network flow data and form a flow data set;
the IP acquisition module monitors network communication content to acquire real-time IP address related information to form an IP address data set;
the data analysis module preprocesses the flow data set and the IP address data set to obtain flow related information, forms a first data set, obtains IP address related information, forms a second data set, matches the flow related information with the IP address related information, obtains one-to-many or one-to-one relation matching quantity of the IP addresses, and records the relation matching quantity to form a third data set;
the monitoring module performs normalization processing on the first data set, the second data set and the third data set, performs calculation, and obtains: abnormality index Yczs;
the abnormality index Yczs is obtained by the following formula:
；
wherein Llxs represents a network flow coefficient, fwxs represents an address access coefficient, A and B represent proportionality coefficients of the network flow coefficient Llxs and the address access coefficient Fwxs respectively, and C represents a first correction constant;
wherein,，/>and (2) is (are) of>；
The network flow coefficient Llxs is obtained through calculation of a first data set and a third data set, and is compared with a preset network flow threshold L to obtain a network flow evaluation scheme;
the address access coefficient Fwxs is obtained through calculation of the second data set and the third data set, and is compared with a preset address access threshold Z to obtain an address access evaluation scheme;
the evaluation module is matched with the preset abnormal alert threshold Y and the abnormal index Yczs to acquire an abnormal level strategy scheme;
and the response module performs specific execution through the content of the abnormal level strategy scheme.
In this embodiment, real-time network traffic data and IP address related information are collected through a traffic collection module and an IP collection module, a traffic data set and an IP address data set are formed, preprocessing is performed through a data analysis module, traffic related information is obtained, a first data set is formed, IP address related information is obtained, a second data set is formed, the traffic related information and the IP address related information are matched through the data analysis module, one-to-many or one-to-one relationship matching number of the IP addresses is obtained, and recording is performed, a third data set is formed, and the first data set, the second data set and the third data set are calculated through a monitoring module, so that: the anomaly index Yczs is matched with a preset anomaly alert threshold Y to obtain an anomaly level strategy scheme, and finally, the content of the anomaly level strategy scheme is specifically executed through a response module, so that under the condition that flow anomalies and IP anomalies are correlated with each other, real-time protection measures can be effectively monitored, and through more comprehensive data analysis and multidimensional matching, the perception level of network threat is improved, and the overall level of network security operation is improved.
Example 2
This embodiment is explained in embodiment 1, please refer to fig. 1, specifically: the flow acquisition module comprises a monitoring unit and an integration unit;
the monitoring unit is used for monitoring data packets in network communication in real time, extracting relevant information in the traffic packets, including source addresses, target addresses, data packet lengths, time stamps, the number of the data packets, loads, service types and port numbers, and marking the acquired information;
the integration unit integrates the marked flow packet information to form a structured data form so as to form a flow data set.
The IP acquisition module comprises a classification unit and an extraction unit;
the classifying unit classifies the protocol field and the data packet type in the protocol network communication content to obtain the IP address related information, wherein the protocol field comprises: TCP, UDP, and ICMP, packet types include: HTTP, FTP, and DNS;
the extracting unit extracts the classified IP address related information to obtain: the source IP address, the target IP address, the number of active times, the time stamp information, the transmission rate, the data packet quantity value and the connection duration form an IP address data group.
The data analysis module comprises a flow processing unit, an IP processing unit and a matching association unit;
the flow processing unit performs checksum preprocessing on the flow data set to obtain flow packet related information, and the flow processing unit forms a first data set, and comprises: packet length Sjb, traffic transmission duration Llsc, load Sjfz, and traffic packet number Llb;
the IP processing unit performs checksum preprocessing on the IP address data set to acquire IP address related information, and forms a second data set, and the method comprises the following steps: address packet Dzb, connection duration Ljsc, and active number Hycs;
number of activations Hycs: the number of active communication times of the IP address in a fixed period is represented, and the interaction frequency of the IP address and the system is reflected;
the source address and the target address in the flow data set and the IP address data set of the matching association unit are matched, and an association relation between the flow and the IP address is established and marked, wherein the association relation comprises the following steps: one-to-many and one-to-one, recording the number of incidence relations of the marks, and integrating the number of incidence relations and the frequency value of occurrence of the same IP address to form a third data set, wherein the third data set comprises: the number of IP traffic relations Gxsl and the IP traffic frequency value Plz, wherein the IP address is recorded only when three or more associated traffic packets appear at the same IP address.
The monitoring module comprises a normalization unit and a calculation unit;
the normalization unit performs normalization processing on the first data set, the second data set and the third data set to enable the first data set, the second data set and the third data set to be in the same dimension;
the computing unit performs first computation on the normalized first data set, the normalized second data set and the normalized third data set to obtain: and (3) carrying out second calculation on the network flow coefficient Llxs and the address access coefficient Fwxs to obtain: abnormality index Yczs.
Example 3
This embodiment is explained in embodiment 1, please refer to fig. 1, specifically: the network flow coefficient lxs is obtained by the following formula:
；
in the formula, the absolute ratio value of the data packet length Sjb to the flow transmission duration Llsc, the absolute ratio value of the load Sjfz to the flow transmission duration Llsc, the absolute ratio value of the flow data packet quantity Llb to the flow transmission duration Llsc are calculated, and then the calculated value is compared with the calculated result of the IP flow relation quantity Gxsl and the IP flow frequency value Plz to obtain a network flow coefficient Llxs, wherein E represents a second correction constant;
and, the network flow coefficient Llxs is compared with a preset network flow threshold L, and a network flow evaluation scheme is obtained:
the network flow coefficient Llxs is smaller than the network flow threshold L, and the network flow transmission is not abnormal;
the network flow coefficient Llxs is more than or equal to the network flow threshold L, the network flow transmission is abnormal, the full checking and killing are carried out, and the abnormal flow packet is deleted and marked.
The address access coefficient Fwxs is obtained by:
；
in the formula, the absolute ratio value of the address data packet Dzb to the connection duration Ljsc is used for calculating the active times Hycs to the connection duration Ljsc, and then comparing the calculated result with the calculated result of the IP flow relation quantity Gxsl and the IP flow frequency value Plz to obtain an address access coefficient Fwxs, wherein F represents a third correction constant;
and, address access coefficient Fwxs is compared with a preset address access threshold Z, and an address access evaluation scheme is obtained:
the address access coefficient Fwxs is smaller than the address access threshold Z, and the access address is not abnormal;
the address access coefficient Fwxs is more than or equal to the address access threshold Z, the access address is abnormal, the abnormal address is marked and a blacklist is added, and meanwhile, the address of the IP section is set to be the upper limit of the flow.
The evaluation module comprises a threshold storage unit and a matching unit;
the threshold storage unit is used for storing an abnormal alert threshold Y, a network flow threshold L, an address access threshold Z, an abnormal level policy scheme, a network flow evaluation scheme, an address access evaluation scheme and the contact modes of related notification personnel;
the matching unit compares the abnormality index Yczs with an abnormality alert threshold Y to obtain an abnormality level policy scheme:
the abnormality index Yczs is less than the abnormality alert threshold Y, and the traffic packet and the source IP address are not abnormal;
the abnormality index Yczs is more than or equal to an abnormality warning threshold Y, the flow packet and the source IP address are abnormal, virus searching and killing are carried out on the flow packet, a blacklist is added to the source IP address contained in the flow packet, the upper flow limit is increased by the IP section, and meanwhile staff is informed to trace or search and kill the flow packet;
virus checking and killing: carrying out deep scanning on the abnormal flow packet, detecting whether the abnormal flow packet contains malicious software or virus, and immediately carrying out virus searching and killing operation to prevent the spread of the malicious code;
blacklist addition: the method comprises the steps that a source IP address of traffic abnormality is marked and added to a blacklist, and the traffic abnormality is prevented from continuously accessing a system, so that potential security threat is reduced;
the upper limit of the IP section flow increases: and for the abnormal IP section, the traffic upper limit is increased so as to better manage and isolate potential attacks and ensure the normal operation of the network.
The response module comprises an execution unit and a recording unit;
the execution unit executes corresponding predefined operations and informs related personnel according to specific protective measures in the abnormal level strategy scheme content, wherein the predefined operations comprise: blocking the source IP of attack, increasing the defending level, triggering alarm notification and limiting the upper and lower limits of traffic, wherein the notification modes comprise: broadcasting, short messages, presetting call recording and internal application communication;
the recording unit records log information generated in the execution process and is used for post audit and analysis, and the recorded information comprises: safe operation, execution time, execution result and personnel notification;
and (3) safety operation record: recording specific security operations executed each time, wherein the specific security operations comprise specific protection measures taken and specific processing steps aiming at abnormality;
performing time recording: recording the execution time of each operation, including the start time and the end time, helps to analyze the safety conditions of the system during different time periods, as well as abnormal conditions occurring at certain points in time;
and (3) recording an execution result: the execution results of the safety operation are recorded in detail, including whether the operation is successful or not and whether abnormal conditions occur or not, and the records provide real-time feedback on the running condition of the system for system maintenance personnel.
In this embodiment, by judging whether the network traffic transmission is abnormal, the real-time monitoring of the network traffic state is realized, the abnormality detection precision of the network traffic transmission and the recognition of multiple abnormal states are improved, the access and the number of times of identifying the abnormal address are more flexibly achieved through the detection of the related information of the IP address, the control of the abnormal address is further enhanced through the ways of marking and adding a blacklist and setting upper and lower limits, and finally, a comprehensive abnormal grade strategy scheme is provided to carry out the secondary detection of the global network traffic and the IP, and meanwhile, an automatic predefined safeguard measure is provided, so that the diffusion and the occurrence of the abnormal state are controlled at the first time.
Example 4
Referring to fig. 2, a specific method for processing data based on network security operation of big data is shown: the method comprises the following steps:
step one: acquiring real-time network flow data through a flow acquisition module to form a flow data set;
step two: acquiring real-time IP address related information through an IP acquisition module to form an IP address data set;
step three: preprocessing a flow data set and an IP address data set through a data analysis module to obtain flow related information, forming a first data set, obtaining IP address related information, forming a second data set, matching the flow related information and the IP address related information, obtaining one-to-many or one-to-one relation matching quantity of the IP addresses, and recording to form a third data set;
step four: and carrying out normalization processing on the first data set, the second data set and the third data set through a monitoring module, and calculating to obtain: abnormality index Yczs;
step five: matching the abnormality index Yczs with a preset abnormality warning threshold Y through an evaluation module to acquire an abnormality grade strategy scheme;
step six: and the response module is used for executing the specific execution of the abnormal level strategy scheme content.
In this embodiment, by acquiring real-time network traffic data, forming a traffic data set, real-time IP address related information, forming an IP address data set, preprocessing, forming a first data set and a second data set, =acquiring the matching number of IP address relationships, recording, forming a third data set, and calculating to acquire: the anomaly index Yczs is matched with a preset anomaly alert threshold Y at the same time, an anomaly level strategy scheme is obtained, specific execution is carried out according to the content of the anomaly level strategy scheme, the matching of the flow related information and the IP address related information is carried out, and the generated anomaly level strategy scheme provides guidelines for subsequent response, so that the purposes of improving network safety and property safety are achieved, the use condition of manpower and material resources is reduced, and the processing efficiency of network safety information is improved.
Specific examples: a data processing system for big data based network security operations used by a certain security operator will use specific parameters and values to demonstrate how to calculate: the abnormality index Yczs, the network traffic coefficient Llxs, and the address access coefficient Fwxs;
it is assumed that the following parameters are owned:
a first data set: packet length Sjb: 1200. traffic transmission duration Llsc: 300. load Sjfz:50 and number of traffic packets Llb:150;
a second data set: address packet Dzb: 500. connection time length Ljsc:150 and the number of activations Hycs:300;
third data set: IP traffic relation number Gxsl:3 and IP traffic frequency value Pl:1, a step of;
correction constant E:0.8;
obtaining according to a calculation formula of a network flow coefficient Llxs:
Llxs=(|1200/300|+|50/300|+|150/300|)/(3+1)+0.8=2；
setting a network flow threshold L to be 5, comparing with a network flow coefficient Llxs, and obtaining: the network flow coefficient Llxs is smaller than the network flow threshold L, and the network flow transmission is not abnormal;
correction constant F:0.7;
Fwxs=(|500/150|+|300/150|)/(3+1)=2；
setting an address access threshold Z to be 5, and comparing the address access threshold Z with a network flow coefficient Llxs to obtain: the address access coefficient Fwxs is smaller than the address access threshold Z, and the access address is not abnormal;
scaling factor a:0.47, scaling factor B:0.48, correction constant C:0.1;
obtaining according to a calculation formula of the abnormality index Yczs:
Yczs=[(0.47*2)+(0.48*2)]+0.1=2；
setting the anomaly alert threshold value Y to 5, comparing with the anomaly index Ycz, obtaining: the anomaly index Yczs < anomaly alert threshold Y, and the traffic packets and source IP address are not anomalous.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (10)
1. A data processing system for network security operation based on big data, characterized in that: the system comprises a flow acquisition module, an IP acquisition module, a data analysis module, a detection module, an evaluation module and a response module;
the flow acquisition module monitors network flow information to acquire real-time network flow data and form a flow data set;
the IP acquisition module monitors network communication content to acquire real-time IP address related information to form an IP address data set;
the data analysis module preprocesses the flow data set and the IP address data set to obtain flow related information, forms a first data set, obtains IP address related information, forms a second data set, matches the flow related information with the IP address related information, obtains one-to-many or one-to-one relation matching quantity of the IP addresses, and records the relation matching quantity to form a third data set;
the monitoring module performs normalization processing on the first data set, the second data set and the third data set, performs calculation, and obtains: abnormality index Yczs;
the abnormality index Yczs is obtained by the following formula:
；
wherein Llxs represents a network flow coefficient, fwxs represents an address access coefficient, A and B represent proportionality coefficients of the network flow coefficient Llxs and the address access coefficient Fwxs respectively, and C represents a first correction constant;
the network flow coefficient Llxs is obtained through calculation of a first data set and a third data set, and is compared with a preset network flow threshold L to obtain a network flow evaluation scheme;
the address access coefficient Fwxs is obtained through calculation of the second data set and the third data set, and is compared with a preset address access threshold Z to obtain an address access evaluation scheme;
the evaluation module is matched with the preset abnormal alert threshold Y and the abnormal index Yczs to acquire an abnormal level strategy scheme;
and the response module performs specific execution through the content of the abnormal level strategy scheme.
2. A data processing system for secure operation of a big data based network according to claim 1, wherein: the flow acquisition module comprises a monitoring unit and an integration unit;
the monitoring unit is used for monitoring data packets in network communication in real time, extracting relevant information in the traffic packets, including source addresses, target addresses, data packet lengths, time stamps, the number of the data packets, loads, service types and port numbers, and marking the acquired information;
the integration unit integrates the marked flow packet information to form a structured data form so as to form a flow data set.
3. A data processing system for secure operation of a big data based network according to claim 1, wherein: the IP acquisition module comprises a classification unit and an extraction unit;
the classifying unit classifies the protocol field and the data packet type in the protocol network communication content to obtain the IP address related information, wherein the protocol field comprises: TCP, UDP, and ICMP, packet types include: HTTP, FTP, and DNS;
the extracting unit extracts the classified IP address related information to obtain: the source IP address, the target IP address, the number of active times, the time stamp information, the transmission rate, the data packet quantity value and the connection duration form an IP address data group.
4. A data processing system for secure operation of a big data based network according to claim 1, wherein: the data analysis module comprises a flow processing unit, an IP processing unit and a matching association unit;
the flow processing unit performs checksum preprocessing on the flow data set to obtain flow packet related information, and the flow processing unit forms a first data set, and comprises: packet length Sjb, traffic transmission duration Llsc, load Sjfz, and traffic packet number Llb;
the IP processing unit performs checksum preprocessing on the IP address data set to acquire IP address related information, and forms a second data set, and the method comprises the following steps: address packet Dzb, connection duration Ljsc, and active number Hycs;
the source address and the target address in the flow data set and the IP address data set of the matching association unit are matched, and an association relation between the flow and the IP address is established and marked, wherein the association relation comprises the following steps: one-to-many and one-to-one, recording the number of incidence relations of the marks, and integrating the number of incidence relations and the frequency value of occurrence of the same IP address to form a third data set, wherein the third data set comprises: the IP traffic relation quantity Gxsl and the IP traffic frequency value Plz.
5. A data processing system for secure operation of a big data based network according to claim 1, wherein: the monitoring module comprises a normalization unit and a calculation unit;
the normalization unit performs normalization processing on the first data set, the second data set and the third data set to enable the first data set, the second data set and the third data set to be in the same dimension;
the computing unit performs first computation on the normalized first data set, the normalized second data set and the normalized third data set to obtain: and (3) carrying out second calculation on the network flow coefficient Llxs and the address access coefficient Fwxs to obtain: abnormality index Yczs.
6. A data processing system for secure operation of a big data based network as claimed in claim 4, wherein: the network flow coefficient lxs is obtained by the following formula:
；
in the formula, the absolute ratio value of the data packet length Sjb to the flow transmission duration Llsc, the absolute ratio value of the load Sjfz to the flow transmission duration Llsc, the absolute ratio value of the flow data packet quantity Llb to the flow transmission duration Llsc are calculated, and then the calculated value is compared with the calculated result of the IP flow relation quantity Gxsl and the IP flow frequency value Plz to obtain a network flow coefficient Llxs, wherein E represents a second correction constant;
and, the network flow coefficient Llxs is compared with a preset network flow threshold L, and a network flow evaluation scheme is obtained:
the network flow coefficient Llxs is smaller than the network flow threshold L, and the network flow transmission is not abnormal;
the network flow coefficient Llxs is more than or equal to the network flow threshold L, the network flow transmission is abnormal, the full checking and killing are carried out, and the abnormal flow packet is deleted and marked.
7. A data processing system for secure operation of a big data based network as claimed in claim 4, wherein: the address access coefficient Fwxs is obtained by:
；
in the formula, the absolute ratio value of the address data packet Dzb to the connection duration Ljsc is used for calculating the active times Hycs to the connection duration Ljsc, and then comparing the calculated result with the calculated result of the IP flow relation quantity Gxsl and the IP flow frequency value Plz to obtain an address access coefficient Fwxs, wherein F represents a third correction constant;
and, address access coefficient Fwxs is compared with a preset address access threshold Z, and an address access evaluation scheme is obtained:
the address access coefficient Fwxs is smaller than the address access threshold Z, and the access address is not abnormal;
the address access coefficient Fwxs is more than or equal to the address access threshold Z, the access address is abnormal, the abnormal address is marked and a blacklist is added, and meanwhile, the address of the IP section is set to be the upper limit of the flow.
8. A data processing system for secure operation of a big data based network according to claim 1, wherein: the evaluation module comprises a threshold storage unit and a matching unit;
the threshold storage unit is used for storing an abnormal alert threshold Y, a network flow threshold L, an address access threshold Z, an abnormal level policy scheme, a network flow evaluation scheme, an address access evaluation scheme and the contact modes of related notification personnel;
the matching unit compares the abnormality index Yczs with an abnormality alert threshold Y to obtain an abnormality level policy scheme:
the abnormality index Yczs is less than the abnormality alert threshold Y, and the traffic packet and the source IP address are not abnormal;
the abnormality index Yczs is more than or equal to an abnormality warning threshold Y, the flow packet and the source IP address are abnormal, virus searching and killing are carried out on the flow packet, a blacklist is added to the source IP address contained in the flow packet, the upper flow limit is increased by the IP section, and meanwhile workers are informed of tracing or searching and killing the flow packet.
9. A data processing system for secure operation of a big data based network according to claim 1, wherein: the response module comprises an execution unit and a recording unit;
the execution unit executes corresponding predefined operations and informs related personnel according to specific protective measures in the abnormal level strategy scheme content, wherein the predefined operations comprise: blocking the source IP of attack, increasing the defending level, triggering alarm notification and limiting the upper and lower limits of traffic, wherein the notification modes comprise: broadcasting, short messages, presetting call recording and internal application communication;
the recording unit records log information generated in the execution process and is used for post audit and analysis, and the recorded information comprises: secure operation, execution time, execution result, and notification personnel.
10. A data processing method for network security operation based on big data, comprising the data processing system for network security operation based on big data according to any one of claims 1 to 9, characterized in that: the method comprises the following steps:
step one: acquiring real-time network flow data through a flow acquisition module to form a flow data set;
step two: acquiring real-time IP address related information through an IP acquisition module to form an IP address data set;
step three: preprocessing a flow data set and an IP address data set through a data analysis module to obtain a first data set, a second data set and a third data set;
step four: and carrying out normalization processing on the first data set, the second data set and the third data set through a monitoring module, and calculating to obtain: abnormality index Yczs;
step five: matching the abnormality index Yczs with a preset abnormality warning threshold Y through an evaluation module to acquire an abnormality grade strategy scheme;
step six: and the response module is used for executing the specific execution of the abnormal level strategy scheme content.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410103156.6A CN117640257B (en) | 2024-01-25 | 2024-01-25 | Data processing method and system for network security operation based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410103156.6A CN117640257B (en) | 2024-01-25 | 2024-01-25 | Data processing method and system for network security operation based on big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117640257A true CN117640257A (en) | 2024-03-01 |
| CN117640257B CN117640257B (en) | 2024-04-16 |
Family
ID=90021933
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410103156.6A Active CN117640257B (en) | 2024-01-25 | 2024-01-25 | Data processing method and system for network security operation based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117640257B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117857222A (en) * | 2024-03-07 | 2024-04-09 | 国网江西省电力有限公司电力科学研究院 | Dynamic IP-based network dynamic defense system and method for new energy centralized control station |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1999049717A2 (en) * | 1998-03-27 | 1999-10-07 | Hughes Electronics Corporation | System and method for multicasting multimedia content |
| CN102572799A (en) * | 2011-12-29 | 2012-07-11 | 华为终端有限公司 | Method for acquiring network connection information of client terminal accessing Wi-Fi spot and terminal |
| CN106656966A (en) * | 2016-09-30 | 2017-05-10 | 广州华多网络科技有限公司 | Method and device for intercepting service processing request |
| US9736147B1 (en) * | 2013-04-08 | 2017-08-15 | Titanium Crypt, Inc. | Artificial intelligence encryption model (AIEM) with device authorization and attack detection (DAAAD) |
| CN108111487A (en) * | 2017-12-05 | 2018-06-01 | 全球能源互联网研究院有限公司 | A kind of safety monitoring method and system |
| CN114338120A (en) * | 2021-12-23 | 2022-04-12 | 绿盟科技集团股份有限公司 | Segment scanning attack detection method, device, medium and electronic equipment |
-
2024
- 2024-01-25 CN CN202410103156.6A patent/CN117640257B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1999049717A2 (en) * | 1998-03-27 | 1999-10-07 | Hughes Electronics Corporation | System and method for multicasting multimedia content |
| CN102572799A (en) * | 2011-12-29 | 2012-07-11 | 华为终端有限公司 | Method for acquiring network connection information of client terminal accessing Wi-Fi spot and terminal |
| US9736147B1 (en) * | 2013-04-08 | 2017-08-15 | Titanium Crypt, Inc. | Artificial intelligence encryption model (AIEM) with device authorization and attack detection (DAAAD) |
| CN106656966A (en) * | 2016-09-30 | 2017-05-10 | 广州华多网络科技有限公司 | Method and device for intercepting service processing request |
| CN108111487A (en) * | 2017-12-05 | 2018-06-01 | 全球能源互联网研究院有限公司 | A kind of safety monitoring method and system |
| CN114338120A (en) * | 2021-12-23 | 2022-04-12 | 绿盟科技集团股份有限公司 | Segment scanning attack detection method, device, medium and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| 姚欣;王劲松;: "基于Apache Storm的大规模网络流量实时监控系统研究", 天津理工大学学报, no. 06, 15 December 2016 (2016-12-15) * |
| 李江涛, 姜永玲: "P2P流量识别与管理技术", 电信科学, no. 03, 15 March 2005 (2005-03-15) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117857222A (en) * | 2024-03-07 | 2024-04-09 | 国网江西省电力有限公司电力科学研究院 | Dynamic IP-based network dynamic defense system and method for new energy centralized control station |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117640257B (en) | 2024-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112651006B (en) | Power grid security situation sensing system | |
| CN110460594B (en) | Threat information data acquisition processing method, device and storage medium | |
| CN108933791B (en) | Intelligent optimization method and device based on power information network safety protection strategy | |
| US20040250169A1 (en) | IDS log analysis support apparatus, IDS log analysis support method and IDS log analysis support program | |
| CN107360118B (en) | Advanced persistent threat attack protection method and device | |
| CN114143064B (en) | Multi-source network security alarm event tracing and automatic disposal method and device | |
| CN104135474B (en) | Intrusion Detection based on host goes out the Network anomalous behaviors detection method of in-degree | |
| CN109361673B (en) | Network anomaly detection method based on flow data sample statistics and balance information entropy estimation | |
| CN112995161B (en) | Network security situation prediction system based on artificial intelligence | |
| CN113810362B (en) | Safety risk detection and treatment method | |
| CN112491805A (en) | Network security equipment management system applied to cloud platform | |
| CN110191004B (en) | Port detection method and system | |
| CN113596028A (en) | Method and device for handling network abnormal behaviors | |
| CN117376031B (en) | Print control instrument network transmission supervision early warning system based on data analysis | |
| CN108600166A (en) | A kind of network security detection method and system | |
| CN114640548A (en) | Network security sensing and early warning method and system based on big data | |
| CN116094817A (en) | Network security detection system and method | |
| CN112671801B (en) | Network security detection method and system | |
| CN112217777A (en) | Attack backtracking method and equipment | |
| CN117640257B (en) | Data processing method and system for network security operation based on big data | |
| CN113489703A (en) | Safety protection system | |
| CN111526109A (en) | Method and device for automatically detecting running state of web threat recognition defense system | |
| CN116633685A (en) | Analysis method based on IPv6 development situation monitoring | |
| CN113923021B (en) | Sandbox-based encrypted traffic processing method, system, equipment and medium | |
| CN110753053B (en) | Flow abnormity prejudging method based on big data analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |