CN116865994A - Network data security prediction method based on big data - Google Patents
Network data security prediction method based on big data Download PDFInfo
- Publication number
- CN116865994A CN116865994A CN202310617495.1A CN202310617495A CN116865994A CN 116865994 A CN116865994 A CN 116865994A CN 202310617495 A CN202310617495 A CN 202310617495A CN 116865994 A CN116865994 A CN 116865994A
- Authority
- CN
- China
- Prior art keywords
- data
- classification
- samples
- administrator
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 26
- 238000007637 random forest analysis Methods 0.000 claims abstract description 17
- 238000007635 classification algorithm Methods 0.000 claims abstract description 14
- 238000007621 cluster analysis Methods 0.000 claims abstract description 9
- 238000005516 engineering process Methods 0.000 claims abstract description 7
- 238000002372 labelling Methods 0.000 claims abstract description 7
- 230000006399 behavior Effects 0.000 claims abstract description 5
- 238000011282 treatment Methods 0.000 claims abstract description 5
- 238000003066 decision tree Methods 0.000 claims description 33
- 238000004458 analytical method Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 14
- 238000012549 training Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 6
- 238000013145 classification model Methods 0.000 claims description 5
- 238000004140 cleaning Methods 0.000 claims description 5
- 238000003860 storage Methods 0.000 claims description 5
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 238000005094 computer simulation Methods 0.000 claims description 2
- 238000007405 data analysis Methods 0.000 claims description 2
- 238000013480 data collection Methods 0.000 claims description 2
- 238000013500 data storage Methods 0.000 claims description 2
- 230000002194 synthesizing effect Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 10
- 238000010801 machine learning Methods 0.000 abstract description 3
- 230000000694 effects Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000019771 cognition Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000012502 risk assessment Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06F18/24155—Bayesian classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1433—Vulnerability analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/20—Network architectures or network communication protocols for network security for managing network security; network security policies in general
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Security & Cryptography (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses a network data security prediction method based on big data, which comprises the following steps: collecting log data and pushing the log data to Kafka; preliminary classifying the useful features extracted from the Kafka through a random forest algorithm; performing cluster analysis on the data by using a K-means algorithm; labeling according to the classification samples, and modeling by using a Bayesian classification algorithm; and carrying out different treatments on the classified data labels. According to the network data security prediction method based on big data, the rapid prediction and recognition of network security risks are realized by collecting, processing and analyzing massive network data and combining a machine learning algorithm and a cluster analysis technology. The network attack behavior is classified, modeled and predicted by adopting a plurality of methods such as a random forest algorithm, a K-means algorithm, a Bayesian classification algorithm and the like, and unknown security threats can be efficiently detected and responded, so that the network security of enterprises is effectively protected, and the method has the advantages of instantaneity, accuracy, expandability and the like.
Description
Technical Field
The application relates to the technical field of network data security, in particular to a network data security prediction method based on big data.
Background
In recent years, the internet has rapidly developed, and the technical field has been diversified. With the rising of emerging technologies such as cloud computing, internet of things, big data, 5G and the like, network information safety boundaries are weakened continuously, safety protection contents are increased continuously, great challenges are provided for data safety and information safety, policies in the aspects of network safety and data management in China are discharged continuously, legislation systems are perfected continuously, and supervision of network safety is improved continuously.
The key technology of the network security situation awareness system and the awareness process thereof are combined, main activities for network situation awareness can be summarized, including measurement, processing of original network data, extraction of feature data, detection, risk assessment of various network activities through calculation, capturing of abnormal network activity data, finally cognition of factors which possibly influence normal operation of the network system is enhanced through the series of processes, network security trend is predicted based on accurate cognition, and support is provided for making effective defense measures through attack tracing and visual display.
Because of randomness and uncertainty of network attack, security situation change based on the random network attack is a complex nonlinear process, the traditional prediction model method (manual matching and rule-based matching) cannot meet the requirements gradually, and more researches are developing towards intelligent prediction methods, such as artificial intelligent methods like machine learning algorithms and the like. The method has the advantages of self-learning capability, high medium-short term prediction accuracy and less human participation.
Disclosure of Invention
This section is intended to outline some aspects of embodiments of the application and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section as well as in the description of the application and in the title of the application, which may not be used to limit the scope of the application.
The present application has been made in view of the above-described problems.
Therefore, the technical problems solved by the application are as follows: because of randomness and uncertainty of network attack, the security situation change based on the randomness and uncertainty is a complex nonlinear process, and the requirement of the traditional prediction model method cannot be met.
In order to solve the technical problems, the application provides the following technical scheme: a network data security prediction method based on big data comprises the following steps:
collecting log data and pushing the log data to Kafka;
preliminary classifying the useful features extracted from the Kafka through a random forest algorithm;
performing cluster analysis on the data by using a K-means algorithm;
labeling according to the classification samples, and modeling by using a Bayesian classification algorithm;
and carrying out different treatments on the classified data labels.
As a preferable scheme of the big data based network data security prediction method of the present application, the method comprises: the process of the preliminary classification includes,
when useful features are extracted from Kafka, the system performs preliminary classification on the useful features according to the existing classification samples and the classifier added with the random forest algorithm, and feeds back analysis results to an administrator;
when classifying network attack behaviors, firstly collecting known attack samples, then constructing a plurality of decision trees, secondly comparing new data features collected from Kafka with the samples, and determining whether the new data features belong to a certain known attack type by integrating prediction results of the plurality of decision trees in a voting method;
when the new data features are primarily classified into a certain attack type, the system feeds back an analysis result to an administrator, informs the administrator of possible risks of the new data features, and provides detailed analysis reports including attack types, attack targets and attack modes;
when the new data features cannot be initially classified into a certain known attack type, the system feeds back an analysis result to an administrator, and informs the administrator that the new data features are not of the known attack type, but potential risks still exist; the system may suggest an administrator to add relevant classification samples, and the administrator may perform corresponding operations based on the suggestions, add new classification samples, and retrain the model.
As a preferable scheme of the big data based network data security prediction method of the present application, the method comprises: the step of classifying the network attack by the random forest algorithm includes,
collecting and preparing data sets, collecting known attack samples, extracting useful features from the attack samples, preprocessing the data, and ensuring the quality and the integrity of the data sets;
selecting features, namely selecting a random sample from the cleaned data set to establish a first decision tree, randomly selecting k features from a feature list for each node, and calculating an optimal dividing point by using the features;
training a decision tree, randomly selecting k features, selecting an optimal dividing point, dividing data into two subsets, and recursively repeating the feature selection and the decision tree training until a stop condition is met;
constructing a random forest, repeatedly selecting characteristics and training decision trees to generate a plurality of decision trees, wherein each decision tree is independent and random, evaluating the performance of the model by using cross verification and other technologies, and selecting optimal parameters and characteristics;
predicting a new sample, comparing the new sample with the existing attack sample when the new data characteristic is collected from Kafka, and determining whether the new data characteristic belongs to a known attack type or not by integrating the prediction results of a plurality of decision trees in a voting method; and transmitting the samples to be predicted to each decision tree to obtain a prediction result of each tree, and synthesizing the results of all the decision trees in a majority voting or average value mode to obtain a final classification result.
As a preferable scheme of the big data based network data security prediction method of the present application, the method comprises: the step of cluster analysis includes the steps of,
for each cluster, calculating the mass center of the cluster, and distributing the data points closest to the mass center to the corresponding cluster;
when the distances between one data point and a plurality of centroids are equal, randomly selecting one centroid for allocation;
after the allocation is completed, the data points belonging to the same cluster are marked as the same class.
As a preferable scheme of the big data based network data security prediction method of the present application, the method comprises: the labeling process based on the classified samples includes,
for each data point, calculating the similarity between the data point and each classified sample;
according to the similarity, assigning the data points to the categories to which the most similar classification samples belong;
after the distribution is completed, the data points are labeled correspondingly, and the label types are classified into safe events and unsafe events.
As a preferable scheme of the big data based network data security prediction method of the present application, the method comprises: the bayesian classification algorithm is represented as,
wherein, P (Y=k|X=x) is posterior probability, the variable Y has k results, X is given observation data, pi k A priori probability that variable Y belongs to the kth class, f k (x) As likelihood function, on the premise that the classification result is k, the probability of the observation data being x is f k (x)，π 1 A priori probability that the variable Y belongs to class 1, f l (x) Is a likelihood function;
the Bayes presumes that the characteristics are independent, and on the premise that the classification result is k, the characteristic variables are mutually independent and expressed as:
P(x 1 ，x 2 ，...，x n |y＝k)＝P(x 1 |y＝k)*P(x 1 |y＝k)*…*P(x n |y＝k)
wherein x is 1 ～x n Is the acquired data.
As a preferable scheme of the big data based network data security prediction method of the present application, the method comprises: the Bayesian classification algorithm generates a Bayesian classification model according to the existing classification sample under a system modeling module, the model is adjusted in the way that,
a new classification sample is added, and when an administrator finds that the new data feature is not covered by the current classification sample, it is added to the classification sample library and the model is retrained.
Unnecessary classification samples are deleted, and when an administrator finds that some classification samples are no longer needed or out of date, they are deleted from the classification sample library and the model is retrained.
The weights of the classified samples are adjusted, and when an administrator finds that the influence of some classified samples on the model is too large or too small, the weights of the classified samples are adjusted, and the model is retrained.
When the administrator adjusts the model, the system retrains the model and sends a notification to the administrator after training is completed to inform the update of the model.
In order to solve the technical problems, the application also provides the following technical scheme: a big data based network data security prediction system comprising:
a data collection module for collecting network data security-related information from various channels;
the data cleaning and preprocessing module is used for cleaning, removing weight and standardizing the collected data;
the data storage module is used for storing the preprocessed data into a database or a data warehouse for subsequent analysis and modeling;
the data analysis and modeling module is used for analyzing and modeling the data stored in the data warehouse, finding out potential network security threats and vulnerabilities and predicting the potential network security threats and vulnerabilities;
and the prediction result display module is used for displaying the prediction result to a user, and the user views the prediction result through the visual interface.
A computer device, comprising: a memory and a processor; the memory stores a computer program characterized in that: the processor, when executing the computer program, implements the steps of the method of any of the present application.
A computer-readable storage medium having stored thereon a computer program, characterized by: which when executed by a processor, carries out the steps of the method described in the application.
The application has the beneficial effects that: according to the network data security prediction method based on big data, the rapid prediction and recognition of network security risks are realized by collecting, processing and analyzing massive network data and combining a machine learning algorithm and a cluster analysis technology. The network attack behaviors are classified, modeled and predicted by adopting a plurality of methods such as a random forest algorithm, a K-means algorithm, a Bayesian classification algorithm and the like, and unknown security threats can be efficiently detected and responded, so that the network security of enterprises is effectively protected. In addition, the method has the advantages of instantaneity, accuracy, expandability and the like, and can be flexibly adjusted and optimized according to different requirements, so that the method shows more excellent technical effects in practice.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
fig. 1 is an overall flowchart of a network data security prediction method based on big data according to a first embodiment of the present application;
fig. 2 is a schematic diagram of a scheme architecture of a network data security prediction method based on big data according to a first embodiment of the present application;
fig. 3 is a schematic application scenario diagram of a network data security prediction method based on big data according to a first embodiment of the present application;
Detailed Description
So that the manner in which the above recited objects, features and advantages of the present application can be understood in detail, a more particular description of the application, briefly summarized above, may be had by reference to the embodiments, some of which are illustrated in the appended drawings. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, but the present application may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present application is not limited to the specific embodiments disclosed below.
Further, reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic can be included in at least one implementation of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
While the embodiments of the present application have been illustrated and described in detail in the drawings, the cross-sectional view of the device structure is not to scale in the general sense for ease of illustration, and the drawings are merely exemplary and should not be construed as limiting the scope of the application. In addition, the three-dimensional dimensions of length, width and depth should be included in actual fabrication.
Also in the description of the present application, it should be noted that the orientation or positional relationship indicated by the terms "upper, lower, inner and outer", etc. are based on the orientation or positional relationship shown in the drawings, are merely for convenience of describing the present application and simplifying the description, and do not indicate or imply that the apparatus or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present application. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
The terms "mounted, connected, and coupled" should be construed broadly in this disclosure unless otherwise specifically indicated and defined, such as: can be fixed connection, detachable connection or integral connection; it may also be a mechanical connection, an electrical connection, or a direct connection, or may be indirectly connected through an intermediate medium, or may be a communication between two elements. The specific meaning of the above terms in the present application will be understood in specific cases by those of ordinary skill in the art.
Example 1
Referring to fig. 1-3, for one embodiment of the present application, there is provided a network data security prediction method based on big data, including:
s1: collecting log data and pushing the log data to Kafka;
further, when log data such as a new data security event is generated, the jump tool collects and pushes it to Kafka (a distributed stream processing platform). At the same time the system will send a notification to the administrator reminding him to pay attention to the event.
S2: preliminary classifying the useful features extracted from the Kafka through a random forest algorithm;
further, when extracting useful features from Kafka, the system will perform preliminary classification according to the existing classification samples and the classifier added with the random forest algorithm, push a piece of real-time data to another topic of Kafka, and simultaneously save an off-line data to the HBASE database; and feeding back the analysis result to an administrator, which can further process according to the analysis result.
Furthermore, when classifying network attack behaviors, firstly collecting known attack samples, then constructing a plurality of decision trees, secondly comparing new data features collected from Kafka with the samples, and determining whether the new data features belong to a certain known attack type by integrating the prediction results of the plurality of decision trees in a voting method mode.
Further, when the new data feature is initially classified as a certain attack type, the system will feed back the analysis result to the administrator, inform the administrator of the possible risk of the new data feature, and provide detailed analysis report including information of attack type, attack target and attack mode. And the administrator further processes according to the information, so as to strengthen the safety protection measures and timely check the system loopholes.
Further, when the new data feature cannot be initially classified as a known attack type, the system will also feed back the analysis result to the administrator, informing the administrator that the new data feature is not of the known attack type, but may still have a potential risk. At the same time, the system may suggest that the administrator add relevant classification samples; the administrator may perform the corresponding operations based on the suggestions, add new classification samples, and retrain the model.
Further, the step of classifying the network attack by the random forest algorithm includes,
data sets are collected and prepared, known attack samples are collected, and useful features are extracted therefrom, including attack type, attacker IP address, etc. The data are then subjected to basic processing such as cleaning, missing value processing and the like, and the quality and the integrity of the data set are ensured.
Selecting a feature, selecting a random sample from the feature based on the cleaned data set, and establishing a first decision tree. For each node, k features are randomly chosen from the feature list, where k < < m (m is the total feature number), and then these features are used to calculate the best scoring point.
Training the decision tree, randomly selecting k features, selecting the best dividing point, dividing the data into two subsets, and recursively repeating selecting features and training the decision tree until a stopping condition is met, e.g., the number of leaf nodes has reached a maximum or the depth has reached a maximum.
The method comprises the steps of constructing a random forest, repeatedly selecting characteristics and training decision trees to generate a plurality of decision trees, wherein each decision tree is independent and random, evaluating the performance of the model by using cross verification and other technologies, and selecting optimal parameters and characteristics to improve the accuracy and stability of the random forest.
Predicting a new sample, comparing the new sample with the existing attack sample when the new data characteristic is collected from Kafka, and determining whether the new data characteristic belongs to a known attack type or not by integrating the prediction results of a plurality of decision trees in a voting method; and transmitting the samples to be predicted to each decision tree to obtain a prediction result of each tree. And then the results of all decision trees are integrated in a majority voting or average value mode and the like, so that a final classification result is obtained.
S3: performing cluster analysis on the data by using a K-means algorithm;
further, when using the K-means (a clustering algorithm) algorithm for clustering, data is taken from the HBASE database, the system generates a classification sample according to the clustering result and stores it in the database, the clustering step includes,
for each cluster, calculating the centroid of the cluster, and then distributing the data points closest to the centroid to the cluster;
when the distances between one data point and a plurality of centroids are equal, randomly selecting one centroid for allocation;
after the allocation is completed, the data points belonging to the same cluster are marked as the same class.
S4: labeling according to the classification samples, and modeling by using a Bayesian classification algorithm;
furthermore, when the characteristic data of the real-time data stream is labeled by using the Bayesian classification algorithm, the system classifies the characteristic data according to the existing classification sample and feeds the classification result back to the administrator. The step of labelling comprises the steps of,
for each data point, calculating the similarity between the data point and each classified sample;
according to the similarity, assigning the data points to the categories to which the most similar classification samples belong;
after the distribution is completed, the data points are labeled correspondingly, and the label types are classified into safe events and unsafe events.
Further, modeling is performed by using a Bayesian classification algorithm to generate a Bayesian classification model, the Bayesian classification model is operated periodically, the Bayesian classification algorithm is expressed as,
where P (y=k|x=x) is the posterior probability, the possible values of the variable Y are k results (in the present application k is 1 and 2, the security event is classified as 1, the unsafe event is classified as 2), X is given observed data, pi k A priori probability that variable Y belongs to the kth class, f k (x) As likelihood function, on the premise that the classification result is k, the probability of the observation data being x is f k (x)，π l A priori probability that the variable Y belongs to class I, f l (x) Is a likelihood function;
it should be noted that, on the premise that the bayesian hypothesis features are independent, that is, the classification result is k, feature variables are mutually independent, and are expressed as:
P(x 1 ，x 2 ，...，x n |y＝k)＝P(x 1 |y＝k)*P(x 1 |y＝k)+…*P(x n |y＝k)
wherein x is 1 ～x n Represented as acquired data.
Further, when the Bayesian classification model is run, the system generates the model based on the existing classification samples and updates the model periodically. The administrator can check the state of the model at any time and adjust the model according to the requirement, and the model state comprises:
the number of classification samples currently in use;
accuracy and recall of the model;
training progress and time of the model;
the time of the last model update.
Further, depending on the model state, the administrator may decide whether the model needs to be adjusted. The mode of adjusting the model is as follows:
a new classification sample is added, and when an administrator finds that the new data feature is not covered by the current classification sample, it is added to the classification sample library and the model is retrained.
Unnecessary classification samples are deleted, and when an administrator finds that some classification samples are no longer needed or out of date, they are deleted from the classification sample library and the model is retrained.
The weights of the classified samples are adjusted, and when an administrator finds that the influence of some classified samples on the model is too large or too small, the weights of the classified samples are adjusted, and the model is retrained.
Furthermore, when the administrator adjusts the model, the system retrains the model, and sends a notification to the administrator after the training is completed to inform the administrator of the update condition of the model, and the administrator can check the state of the model at any time and adjust the model according to the requirement.
S5: and carrying out different treatments on the classified data labels.
Furthermore, when the classified data tag is obtained, the system performs different processes according to the type of the tag, and the specific processing scheme is as follows:
when the tag belongs to an unsafe event, pushing the tag data to Kafka in real time for a service system platform to call; at the same time, the system will send a notification to the administrator, reminding him to pay attention to the event.
When the tag belongs to a security event, the tag is stored in a database;
furthermore, through the data labels in the database, real-time monitoring and quick response are realized, the network security protection level is effectively improved, unknown attacks are predicted and analyzed by using the existing classification samples and algorithm models, security risks are found timely, corresponding measures are taken for protection and repair, and the probability and loss of the network being attacked are reduced.
The present embodiment also provides a computing device comprising, a memory and a processor; the memory is used for storing computer executable instructions, and the processor is used for executing the computer executable instructions to realize the network data security prediction method based on big data as proposed by the above embodiment.
The present embodiment also provides a storage medium having stored thereon a computer program which, when executed by a processor, implements a network data security prediction method based on big data as proposed in the above embodiments.
The storage medium according to the present embodiment belongs to the same inventive concept as the network data security prediction method based on big data according to the above embodiment, and technical details not described in detail in the present embodiment can be seen in the above embodiment, and the present embodiment has the same beneficial effects as the above embodiment.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile memory may include read only memory, magnetic tape, floppy disk, flash memory, optical memory, high density embedded nonvolatile memory, resistive memory, magnetic memory, ferroelectric memory, phase change memory, graphene memory, and the like. Volatile memory can include random access memory, external cache memory, or the like. By way of illustration, and not limitation, RAM can take many forms, such as static random access memory or dynamic random access memory. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
Example 2
In order to verify the beneficial effects of the present application, the following is a scientific demonstration through simulation experiments.
The implementation adopts the traditional technical scheme and the my technical scheme to process the same weblog data set;
TABLE 1 analog network packet
| Time stamp | Source IP | Target IP | Data packet size | Label type |
| 1621963185 | 192.168.1.10 | 10.0.0.2 | 500bytes | Security event |
| 1621963186 | 10.0.0.3 | 192.168.1.12 | 1000bytes | Unsafe event |
| 1621963187 | 192.168.1.14 | 10.0.0.5 | 2000bytes | Security event |
| 1621963188 | 10.0.0.7 | 192.168.1.16 | 1500bytes | Unsafe event |
| 1621963189 | 192.168.1.18 | 10.0.0.9 | 800bytes | Security event |
This dataset containing 5 pieces of weblog data was processed and classified using conventional techniques, with 2 each of the security events and the non-security events taking a total of 1 hour. The final classification accuracy is 80%, the false alarm rate is 20%, and the missing report rate is 0%.
We then process and classify the same set of weblog data using the above solution. Data were first pushed into Kafka, with 10 minutes for the initial classification of random forest algorithm, 5 minutes for the cluster analysis of K-means algorithm, and 30 minutes for modeling of bayesian classification algorithm. At the time of classification, we do different treatments according to classification tags, such as for security events, pass directly; for unsafe events, relevant information is recorded and a corresponding alarm notification is triggered, etc. The total time of the whole process is 45 minutes, the classification accuracy reaches 100%, the false alarm rate is 0%, and the missing report rate is 0%.
The technical effect of the application is obvious under the condition of a small number of samples, the method has obvious advantages in accuracy and processing time, and the advantages of the application compared with the traditional scheme are more obvious under the condition of a large number of samples in a large data network.
It should be noted that the above embodiments are only for illustrating the technical solution of the present application and not for limiting the same, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present application may be modified or substituted without departing from the spirit and scope of the technical solution of the present application, which is intended to be covered in the scope of the claims of the present application.
Claims (10)
1. The network data security prediction method based on big data is characterized by comprising the following steps:
collecting log data and pushing the log data to Kafka;
preliminary classifying the useful features extracted from the Kafka through a random forest algorithm;
performing cluster analysis on the data by using a K-means algorithm;
labeling according to the classification samples, and modeling by using a Bayesian classification algorithm;
and carrying out different treatments on the classified data labels.
2. The big data based network data security prediction method of claim 1, wherein: the process of the preliminary classification includes,
when useful features are extracted from Kafka, the system performs preliminary classification on the useful features according to the existing classification samples and the classifier added with the random forest algorithm, and feeds back analysis results to an administrator;
when classifying network attack behaviors, firstly collecting known attack samples, then constructing a plurality of decision trees, secondly comparing new data features collected from Kafka with the samples, and determining whether the new data features belong to a certain known attack type by integrating prediction results of the plurality of decision trees in a voting method;
when the new data features are primarily classified into a certain attack type, the system feeds back an analysis result to an administrator, informs the administrator of possible risks of the new data features, and provides detailed analysis reports including attack types, attack targets and attack modes;
when the new data features cannot be initially classified into a certain known attack type, the system feeds back an analysis result to an administrator, and informs the administrator that the new data features are not of the known attack type, but potential risks still exist; the system may suggest an administrator to add relevant classification samples, and the administrator may perform corresponding operations based on the suggestions, add new classification samples, and retrain the model.
3. The big data based network data security prediction method of claim 2, wherein: the step of classifying the network attack by the random forest algorithm includes,
collecting and preparing data sets, collecting known attack samples, extracting useful features from the attack samples, preprocessing the data, and ensuring the quality and the integrity of the data sets;
selecting features, namely selecting a random sample from the cleaned data set to establish a first decision tree, randomly selecting k features from a feature list for each node, and calculating an optimal dividing point by using the features;
training a decision tree, randomly selecting k features, selecting an optimal dividing point, dividing data into two subsets, and recursively repeating the feature selection and the decision tree training until a stop condition is met;
constructing a random forest, repeatedly selecting characteristics and training decision trees to generate a plurality of decision trees, wherein each decision tree is independent and random, evaluating the performance of the model by using a cross verification technology, and selecting optimal parameters and characteristics;
predicting a new sample, comparing the new sample with the existing attack sample when the new data characteristic is collected from Kafka, and determining whether the new data characteristic belongs to a known attack type or not by integrating the prediction results of a plurality of decision trees in a voting method; and transmitting the samples to be predicted to each decision tree to obtain a prediction result of each tree, and synthesizing the results of all the decision trees in a majority voting or average value mode to obtain a final classification result.
4. The big data based network data security prediction method of claim 1, wherein: the step of cluster analysis includes the steps of,
for each cluster, calculating the mass center of the cluster, and distributing the data points closest to the mass center to the corresponding cluster;
when the distances between one data point and a plurality of centroids are equal, randomly selecting one centroid for allocation;
after the allocation is completed, the data points belonging to the same cluster are marked as the same class.
5. The big data based network data security prediction method of claim 1, wherein: the labeling process based on the classified samples includes,
for each data point, calculating the similarity between the data point and each classified sample;
according to the similarity, assigning the data points to the categories to which the most similar classification samples belong;
after the distribution is completed, the data points are labeled correspondingly, and the label types are classified into safe events and unsafe events.
6. The big data based network data security prediction method of claim 1, wherein: the bayesian classification algorithm is represented as,
wherein, P (Y=k|X=x) is posterior probability, the variable Y has k results, X is given observation data, pi k A priori probability that variable Y belongs to the kth class, f k (x) As likelihood function, on the premise that the classification result is k, the probability of the observation data being x is f k (x)，π l A priori probability that the variable Y belongs to class I, f l (x) Is a likelihood function;
the Bayes presumes that the characteristics are independent, and on the premise that the classification result is k, the characteristic variables are mutually independent and expressed as:
P(x 1 ，x 2 ，...，x n |y＝k)P(x 1 |y＝k)*P(x 1 |y＝k)*…*P(x n |y＝k)
wherein x is 1 ～x n Is the acquired data.
7. The big data based network data security prediction method of claim 6, wherein: the Bayesian classification algorithm generates a Bayesian classification model according to the existing classification sample under a system modeling module, the model is adjusted in the way that,
adding a new classification sample, and when an administrator finds that the new data features are not covered by the current classification sample, adding the new classification sample into a classification sample library and retraining a model;
deleting unnecessary classification samples, deleting some classification samples from the classification sample library when an administrator finds that the classification samples are no longer needed or outdated, and retraining the model;
adjusting the weights of the classified samples, and when an administrator finds that the influence of some classified samples on the model is too large or too small, adjusting the weights of the classified samples and retraining the model;
when the administrator adjusts the model, the system retrains the model and sends a notification to the administrator after training is completed to inform the update of the model.
8. A big data based network data security prediction system, comprising:
a data collection module for collecting network data security-related information from various channels;
the data cleaning and preprocessing module is used for cleaning, removing weight and standardizing the collected data;
the data storage module is used for storing the preprocessed data into a database or a data warehouse for subsequent analysis and modeling;
the data analysis and modeling module is used for analyzing and modeling the data stored in the data warehouse, finding out potential network security threats and vulnerabilities and predicting the potential network security threats and vulnerabilities;
and the prediction result display module is used for displaying the prediction result to a user, and the user views the prediction result through the visual interface.
9. A computer device, comprising: a memory and a processor; the memory stores a computer program characterized in that: the processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program implementing the steps of the method of any of claims 1 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310617495.1A CN116865994A (en) | 2023-05-29 | 2023-05-29 | Network data security prediction method based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310617495.1A CN116865994A (en) | 2023-05-29 | 2023-05-29 | Network data security prediction method based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116865994A true CN116865994A (en) | 2023-10-10 |
Family
ID=88234737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310617495.1A Pending CN116865994A (en) | 2023-05-29 | 2023-05-29 | Network data security prediction method based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116865994A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117633665A (en) * | 2024-01-26 | 2024-03-01 | 深圳市互盟科技股份有限公司 | Network data monitoring method and system |
| CN117828688A (en) * | 2024-01-29 | 2024-04-05 | 北京亚鸿世纪科技发展有限公司 | Data security processing method and system |
-
2023
- 2023-05-29 CN CN202310617495.1A patent/CN116865994A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117633665A (en) * | 2024-01-26 | 2024-03-01 | 深圳市互盟科技股份有限公司 | Network data monitoring method and system |
| CN117633665B (en) * | 2024-01-26 | 2024-05-28 | 深圳市互盟科技股份有限公司 | Network data monitoring method and system |
| CN117828688A (en) * | 2024-01-29 | 2024-04-05 | 北京亚鸿世纪科技发展有限公司 | Data security processing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108881194B (en) | Method and device for detecting abnormal behaviors of users in enterprise | |
| CN111475804B (en) | Alarm prediction method and system | |
| CN108737406B (en) | Method and system for detecting abnormal flow data | |
| CN106790256B (en) | Active machine learning system for dangerous host supervision | |
| Ektefa et al. | Intrusion detection using data mining techniques | |
| CN110163647B (en) | Data processing method and device | |
| CN116865994A (en) | Network data security prediction method based on big data | |
| CN109525595A (en) | A kind of black production account recognition methods and equipment based on time flow feature | |
| CN111126820B (en) | Method and system for preventing electricity stealing | |
| CN111612041A (en) | Abnormal user identification method and device, storage medium and electronic equipment | |
| CN105574544A (en) | Data processing method and device | |
| CN114138968B (en) | Network hotspot mining method, device, equipment and storage medium | |
| CN111368926B (en) | Image screening method, device and computer readable storage medium | |
| CN111259985A (en) | Classification model training method and device based on business safety and storage medium | |
| CN107169106A (en) | Video retrieval method, device, storage medium and processor | |
| US20190164245A1 (en) | Method for automatically linking associated incidents related to criminal activity | |
| Agarwal et al. | Comparison of machine learning approaches in the prediction of terrorist attacks | |
| US20170300582A1 (en) | Event identification through analysis of social-media postings | |
| CN111641608A (en) | Abnormal user identification method and device, electronic equipment and storage medium | |
| CN113918367A (en) | Large-scale system log anomaly detection method based on attention mechanism | |
| CN110909195A (en) | Picture labeling method and device based on block chain, storage medium and server | |
| CN117272204A (en) | Abnormal data detection method, device, storage medium and electronic equipment | |
| CN108121912B (en) | Malicious cloud tenant identification method and device based on neural network | |
| CN115115369A (en) | Data processing method, device, equipment and storage medium | |
| CN114912109B (en) | Abnormal behavior sequence identification method and system based on graph embedding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |