CN112925805B - Big data intelligent analysis application method based on network security - Google Patents
Big data intelligent analysis application method based on network securityInfo
- Publication number
- CN112925805B CN112925805B CN202110401931.2A CN202110401931A CN112925805B CN 112925805 B CN112925805 B CN 112925805B CN 202110401931 A CN202110401931 A CN 202110401931A CN 112925805 B CN112925805 B CN 112925805B
- Authority
- CN
- China
- Prior art keywords
- data
- analysis
- network
- security
- establishing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 60
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000013178 mathematical model Methods 0.000 claims abstract description 17
- 238000007726 management method Methods 0.000 claims abstract description 15
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 14
- 238000003860 storage Methods 0.000 claims abstract description 14
- 238000005516 engineering process Methods 0.000 claims abstract description 13
- 238000007781 pre-processing Methods 0.000 claims abstract description 12
- 238000007405 data analysis Methods 0.000 claims abstract description 6
- 238000007418 data mining Methods 0.000 claims abstract description 5
- 230000002776 aggregation Effects 0.000 claims description 10
- 238000004220 aggregation Methods 0.000 claims description 10
- 238000005065 mining Methods 0.000 claims description 9
- 238000012502 risk assessment Methods 0.000 claims description 9
- 238000012098 association analyses Methods 0.000 claims description 7
- 230000000903 blocking effect Effects 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 230000008439 repair process Effects 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 3
- 230000008521 reorganization Effects 0.000 claims description 2
- 230000002457 bidirectional effect Effects 0.000 claims 1
- 238000012544 monitoring process Methods 0.000 abstract description 6
- 230000000007 visual effect Effects 0.000 abstract description 4
- 206010000117 Abnormal behaviour Diseases 0.000 abstract description 3
- 230000007123 defense Effects 0.000 abstract description 3
- 230000007246 mechanism Effects 0.000 abstract description 3
- 230000004044 response Effects 0.000 abstract description 3
- 238000004590 computer program Methods 0.000 description 9
- 230000006399 behavior Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013481 data capture Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
- 239000002023 wood Substances 0.000 description 1
Abstract
The invention provides a big data intelligent analysis application method based on network security, which comprises the following steps: acquiring and preprocessing network data, storing the network data and forming a distributed storage management system; establishing a mathematical model library according to the network data to complete the definition of a data structure; providing an entry for data mining analysis, establishing a corresponding model according to algorithm parameters, and generating an analysis result. The invention can be based on security big data, and all-weather all-round network security situation sensing is realized; by utilizing the existing real-time monitoring technology, abnormal behaviors are found through long-time data analysis; establishing visual analysis capability through data modeling, rapidly studying and judging the influence range, attack path, purpose and means of threat, and making effective security decision and response; and establishing a risk notification and threat early warning mechanism, comprehensively grasping information such as the purpose, technical tactics, attack tools and the like of an attacker, and perfecting a defense system.
Description
Technical Field
The invention relates to the technical field of network security, in particular to a big data intelligent analysis application method based on network security.
Background
Currently, network security technologies generally detect persistent threats of all suspicious network activities through big data analysis and detection of known network threats and attack behaviors, namely through technologies of analyzing network traffic, threat data, behaviors and the like, so as to protect.
However, although the technology can synchronously perform network detection and file detection, the technology lacks the capabilities of data acquisition, analysis and prediction, early warning of research and judgment, solution and the like of unknown threats.
Disclosure of Invention
The invention aims to provide a big data intelligent analysis application method based on network security, which can effectively monitor various network threats.
In order to achieve the above purpose, the present invention provides a big data intelligent analysis application method based on network security, which is characterized in that the application method comprises: acquiring and preprocessing network data, storing the network data and forming a distributed storage management system; establishing a mathematical model library according to the network data to complete the definition of a data structure; providing an entry for data mining analysis, establishing a corresponding model according to algorithm parameters, and generating an analysis result.
Preferably, the step of "acquiring and preprocessing network data, storing the network data and forming a distributed storage management system" includes: and capturing network data through the threat information feature library, carrying out data aggregation, and carrying out unified preprocessing on the network data according to the big data distributed computing characteristics and the algorithm characteristics to form a unified distributed storage management system.
Preferably, the "uniformly preprocessing the network data according to the big data distributed computing characteristic and the algorithm characteristic" includes: the method comprises the steps of carrying out hierarchical aggregation, reorganization, cleaning, extraction, conversion, management, slicing and other pretreatment operations on network data by utilizing data slicing, data classification and data aggregation, and utilizing a data index marking technology.
Preferably, the step of "establishing a mathematical model library according to the network data, and completing the definition of the data structure" includes: and carrying out intelligent analysis, mining and exploration on the network data, establishing a mathematical model library, realizing data structure definition of a big data format, and carrying out unified management on algorithm parameters, the mathematical model library, a model evaluation system and mining analysis results.
Preferably, the step of "establishing a mathematical model library according to the network data, and completing the definition of the data structure" includes: an event understanding engine is established, the integrated security log is subjected to association analysis based on event understanding rules, the security log is understood as a security event, and the alarm accuracy is improved; abstract analysis model, and implement the analysis model in Spark-streaming.
Preferably, the step of "establishing a mathematical model library according to the network data, and completing the definition of the data structure" includes: and establishing an attack chain analysis model, generating network security events by analyzing the security logs and the flow security logs collected by each network security device, and carrying out forward and backward reasoning, forward reasoning and early warning potential threat and backward reasoning and restoring attack scenes.
Preferably, the method further comprises: the attack chain mining program aggregates all security events according to the dimension of the destination asset on the basis of network security events and corresponds to all stages of an attack chain so as to discover vulnerable hosts in the current network.
Preferably, the step of "establishing a mathematical model library according to the network data, and completing the definition of the data structure" includes: establishing an information association model, and realizing potential threat alarm through association analysis of cloud information and local events; wherein the potential threats include malicious IP, malicious URL.
Preferably, the step of "establishing a mathematical model library according to the network data, and completing the definition of the data structure" includes: when the security log passes through the analysis engine, the security log is matched with a knowledge base, the security log is labeled to generate a security event, and the event is uploaded to cloud information for information verification, so that analysis accuracy is improved.
Preferably, the step of "establishing a mathematical model library according to the network data, and completing the definition of the data structure" includes: establishing a risk assessment model, carrying out risk assessment on external threats and asset vulnerabilities in combination with asset values, obtaining risk scores through the risk assessment, generating a decision of a treatment mode based on the risk scores, and carrying out vulnerability repair and threat blocking.
Compared with the prior art, the intelligent analysis application method for the big data based on the network security can be used for sensing the network security situation in all weather and all directions based on the security big data; by utilizing the existing real-time monitoring technology, abnormal behaviors are found through long-time data analysis; establishing visual analysis capability through data modeling, rapidly studying and judging the influence range, attack path, purpose and means of threat, and making effective security decision and response; and establishing a risk notification and threat early warning mechanism, comprehensively grasping information such as the purpose, technical tactics, attack tools and the like of an attacker, and perfecting a defense system.
Drawings
Fig. 1 is a flow chart of a big data intelligent analysis application method based on network security according to an embodiment of the present invention;
Fig. 2 is a schematic structural diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
For a further understanding of the objects, construction, features and functions of the invention, reference should be made to the following detailed description of the preferred embodiments.
Certain terms are used throughout the description and claims to refer to particular components. It will be appreciated by those of ordinary skill in the art that manufacturers may refer to a component by different names. The description and claims do not take the form of an element differentiated by name, but rather by functional differences. In the following description and in the claims, the terms "include" and "comprise" are used in an open-ended fashion, and thus should be interpreted to mean "include, but not limited to.
Referring to fig. 1, fig. 1 is a flow chart of a network security-based intelligent analysis application method according to an embodiment of the present invention.
As shown in FIG. 1, the big data intelligent analysis application method provided by the application comprises the following steps:
step S100, acquiring and preprocessing network data, storing the network data and forming a distributed storage management system;
Step S200, a mathematical model library is established according to the network data, and data structure definition is completed;
Step S300, providing an entry for data mining analysis, establishing a corresponding model according to algorithm parameters, and generating an analysis result.
In the specific implementation, the application is based on the comprehensive application of data analysis, information research and judgment and monitoring and early warning of the network security situation, takes a security comparison module as a core, and comprises a data grabbing aggregation unit, an active analysis module, a storage unit module, a display unit module, a real-time monitoring module and a behavior auxiliary module;
In step S100, the threat information feature library is utilized to perform data capture and data aggregation, and the data is uniformly preprocessed according to the large data distributed computing characteristics and the algorithm characteristics to form a uniform distributed storage management system. And (3) carrying out rapid calculation and mining analysis on the data by using a distributed computing architecture, and constructing a corresponding service model and visual analysis based on the collected big data so as to discover and reveal hidden elements and associations. The data acquisition mode is mainly implemented by a syslog and flow technology, for a large number of multi-source heterogeneous data sources, a front probe is adopted, the data are collected in a centralized mode, normalized and the like, the data are integrated and then are uniformly transmitted to a big data application system, and the application system carries out association analysis according to the correlation among security events to obtain more accurate monitoring information and discover attack sources.
The large data preprocessing utilizes data slicing, data classification, data aggregation, data index marking technology to perform hierarchical preprocessing operations such as aggregation, recombination, cleaning, extraction, conversion, management, cutting and dividing on original data, unifies standard interfaces and unified data standards, and realizes safe, reliable, quick and effective unified storage management on multiple types and multiple formats of data on the basis of meeting consistency requirements through a distributed storage management technology.
In step S200, the active analysis module performs intelligent analysis, mining and exploration on the big data. And establishing a mathematical model library to realize data structure definition of a big data format, uniformly managing algorithm parameters, the mathematical model library, a model evaluation system and results of mining analysis, providing an entry for data mining analysis, and automatically calling an algorithm used for mining analysis and a corresponding model thereof according to the input algorithm parameters.
In step S300, an event understanding engine is established, and the log after merging is subjected to association analysis based on a certain event understanding rule, so that the log is understood as a security event, and the alarm accuracy is improved; abstract analysis model, and implement the analysis model in Spark-streaming, thereby completing the intelligent analysis scheme from the past security service analysis to the present big data.
Further, an attack chain analysis model is established, network security events are generated by analyzing security logs and flow logs collected by each network security device, forward and backward reasoning is carried out, potential threats are early-warned by forward reasoning, and attack scenes are restored by backward reasoning. The attack chain mining program aggregates all security events according to the dimension of the destination asset on the basis of network security events and corresponds to all stages of an attack chain so as to discover vulnerable hosts in the current network. The current security situation of the whole network is convenient to know, and fragile assets are reinforced.
Further, an information association model is established, and potential threat alarming is achieved through association analysis of cloud information and local events. When the security log passes through the analysis engine, the security log is matched with a knowledge base, a security event is generated by marking the log, the event is uploaded to cloud information for information verification, analysis accuracy is improved, in addition, malicious IP information in the cloud information is queried, other attack behaviors of the malicious IP are analyzed, and feedback is carried out, so that an early warning function is realized.
Further, a risk assessment model is established, external threats and asset vulnerabilities are combined with asset values to carry out risk assessment, risk scores are obtained through the risk assessment, decisions of treatment modes are generated based on the risk scores, vulnerability repair is carried out, and threat blocking is carried out.
The risk of the whole system is evaluated by the system, and for each asset, some security events initiated outside the system, such as intrusion events, abnormal flow events, stiff wood vermicular events and the like, are scored as threats; some vulnerabilities of the inside itself, such as system vulnerabilities, website security, etc., are scored. Finally, comprehensive analysis is carried out by combining the asset value, and the security score based on the asset group, the business domain and even the whole system is obtained.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a preferred embodiment of a terminal device according to the present invention. The terminal device comprises a processor 301, a memory 302 and a computer program stored in the memory 302 and configured to be executed by the processor 301, wherein the processor 301 implements the network security based big data intelligent analysis application method according to any of the above embodiments when executing the computer program.
Preferably, the computer program may be divided into one or more modules/units (e.g. computer program 1, computer program 2, … …) stored in the memory 302 and executed by the processor 301 to complete the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing the specified functions, which instruction segments are used for describing the execution of the computer program in the terminal device.
The Processor 301 may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc., or the Processor 301 may be a microprocessor, or the Processor 301 may be any conventional Processor, the Processor 301 being a control center of the terminal device, connecting the various parts of the terminal device using various interfaces and lines.
The memory 302 mainly includes a program storage area, which may store an operating system, application programs required for at least one function, and the like, and a data storage area, which may store related data and the like. In addition, the memory 302 may be a high-speed random access memory, a nonvolatile memory such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), etc., or the memory 302 may be other volatile solid-state memory devices.
It should be noted that the above-mentioned terminal device may include, but is not limited to, a processor, a memory, and those skilled in the art will understand that the schematic structural diagram of fig. 2 is merely an example of the above-mentioned terminal device, and does not limit the above-mentioned terminal device, and may include more or fewer components than those shown, or may combine some components or different components.
The embodiment of the invention also provides a computer readable storage medium, which comprises a stored computer program, wherein when the computer program runs, the equipment where the computer readable storage medium is located is controlled to execute the big data intelligent analysis application method based on network security.
The embodiment of the invention provides a big data intelligent analysis application method based on network security, which can carry out clue mining according to the resources of various open platforms, integrate the resources through technologies such as data packet capturing, reverse analysis and the like, and search deep data through manual deep analysis.
It should be noted that the system embodiments described above are merely illustrative, and that the units described as separate units may or may not be physically separate, and that units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the system embodiment of the present invention, the connection relationship between the modules represents that there is a communication connection between them, and may be specifically implemented as one or more communication buses or signal lines.
In conclusion, the application can sense the network security situation in all weather and all directions based on the security big data; by utilizing the existing real-time monitoring technology, abnormal behaviors are found through long-time data analysis; establishing visual analysis capability through data modeling, rapidly studying and judging the influence range, attack path, purpose and means of threat, and making effective security decision and response; and establishing a risk notification and threat early warning mechanism, comprehensively grasping information such as the purpose, technical tactics, attack tools and the like of an attacker, and perfecting a defense system.
The invention has been described with respect to the above-described embodiments, however, the above-described embodiments are merely examples of practicing the invention. It should be noted that the disclosed embodiments do not limit the scope of the invention. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.
Claims (4)
1. The intelligent big data analysis application method based on network security is characterized by comprising the following steps:
Acquiring and preprocessing network data, storing the network data and forming a distributed storage management system;
Establishing a mathematical model library according to the network data to complete the definition of a data structure;
Intelligent analysis, mining and exploration are carried out on the network data, a mathematical model library is established, data structure definition of a big data format is realized, and algorithm parameters, the mathematical model library, a model evaluation system and mining analysis results are uniformly managed;
An event understanding engine is established, the integrated security log is subjected to association analysis based on event understanding rules, the security log is understood as a security event, and the alarm accuracy is improved; abstracting an analysis model, and implementing codes of the analysis model in Spark-streaming;
Establishing an attack chain analysis model, generating network security events by analyzing the security logs and the flow security logs collected by each network security device, and carrying out forward and backward bidirectional reasoning, forward reasoning and early warning potential threat, and backward reasoning and restoring attack scenes;
The attack chain mining program aggregates all security events according to the dimension of the destination asset on the basis of network security events and corresponds to all stages of an attack chain, so that a fragile host in the current network is found;
Establishing an information association model, and realizing potential threat alarm through association analysis of cloud information and local events;
Wherein the potential threats include malicious IP, malicious URL;
When the safety log passes through the analysis engine, the safety log is matched with a knowledge base, the safety log is labeled to generate a safety event, and the event is uploaded to cloud information for information verification, so that analysis accuracy is improved;
providing an entry for data mining analysis, establishing a corresponding model according to algorithm parameters, and generating an analysis result.
2. The intelligent analysis application method for big data based on network security according to claim 1, wherein the step of acquiring and preprocessing network data, storing the network data and forming a distributed storage management system comprises:
and capturing network data through the threat information feature library, carrying out data aggregation, and carrying out unified preprocessing on the network data according to the big data distributed computing characteristics and the algorithm characteristics to form a unified distributed storage management system.
3. The intelligent analysis application method of big data based on network security according to claim 2, wherein the step of uniformly preprocessing the network data according to the big data distributed computing characteristic and the algorithm characteristic comprises the following steps:
the method comprises the steps of carrying out hierarchical aggregation, reorganization, cleaning, extraction, conversion, management, slicing and other pretreatment operations on network data by utilizing data slicing, data classification and data aggregation, and utilizing a data index marking technology.
4. The intelligent analysis application method of big data based on network security according to claim 1, wherein the step of establishing a mathematical model library based on the network data to complete the definition of the data structure comprises:
establishing a risk assessment model, carrying out risk assessment on external threats and asset vulnerabilities in combination with asset values, obtaining risk scores through the risk assessment, generating a decision of a treatment mode based on the risk scores, and carrying out vulnerability repair and threat blocking.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110401931.2A CN112925805B (en) | 2021-04-14 | Big data intelligent analysis application method based on network security |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110401931.2A CN112925805B (en) | 2021-04-14 | Big data intelligent analysis application method based on network security |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112925805A CN112925805A (en) | 2021-06-08 |
| CN112925805B true CN112925805B (en) | 2024-07-09 |
Family
ID=
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107196910A (en) * | 2017-04-18 | 2017-09-22 | 国网山东省电力公司电力科学研究院 | Threat early warning monitoring system, method and the deployment framework analyzed based on big data |
| CN110334155A (en) * | 2019-07-09 | 2019-10-15 | 佛山市伏宸区块链科技有限公司 | A kind of block chain threat intelligence analysis method and system based on big data integration |
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107196910A (en) * | 2017-04-18 | 2017-09-22 | 国网山东省电力公司电力科学研究院 | Threat early warning monitoring system, method and the deployment framework analyzed based on big data |
| CN110334155A (en) * | 2019-07-09 | 2019-10-15 | 佛山市伏宸区块链科技有限公司 | A kind of block chain threat intelligence analysis method and system based on big data integration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9369484B1 (en) | Dynamic security hardening of security critical functions | |
| US9191398B2 (en) | Method and system for alert classification in a computer network | |
| CN110213226B (en) | Network attack scene reconstruction method and system based on risk full-factor identification association | |
| US10505986B1 (en) | Sensor based rules for responding to malicious activity | |
| EP3623983A1 (en) | Method and device for identifying security threats, storage medium, processor and terminal | |
| US20230087309A1 (en) | Cyberattack identification in a network environment | |
| CN114531283B (en) | Method, system, storage medium and terminal for measuring robustness of intrusion detection model | |
| Colbert et al. | A process-oriented intrusion detection method for industrial control systems | |
| CN117220961B (en) | Intrusion detection method, device and storage medium based on association rule patterns | |
| CN117478433A (en) | Network and information security dynamic early warning system | |
| CN112925805B (en) | Big data intelligent analysis application method based on network security | |
| CN116668054A (en) | Security event collaborative monitoring and early warning method, system, equipment and medium | |
| CN114006719B (en) | AI verification method, device and system based on situation awareness | |
| CN115987544A (en) | Network security threat prediction method and system based on threat intelligence | |
| CN115277472A (en) | Network security risk early warning system and method for multidimensional industrial control system | |
| Protic et al. | WK-FNN design for detection of anomalies in the computer network traffic | |
| CN110881016A (en) | Network security threat assessment method and device | |
| CN112925805A (en) | Big data intelligent analysis application method based on network security | |
| Song | Public cloud network intrusion and internet legal supervision based on abnormal feature detection | |
| Yu et al. | Mining anomaly communication patterns for industrial control systems | |
| Polozhentsev et al. | Novel Cyber Incident Management System for 5G-based Critical Infrastructures | |
| CN113032774A (en) | Training method, device and equipment of anomaly detection model and computer storage medium | |
| CN115085965B (en) | Power system information network attack risk assessment method, device and equipment | |
| CN117609990B (en) | Self-adaptive safety protection method and device based on scene association analysis engine | |
| CN112637142B (en) | Security threat tracing method and system based on power network environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |