CN116366303A - Network anomaly detection method, device, equipment and medium based on deep learning - Google Patents

Network anomaly detection method, device, equipment and medium based on deep learning Download PDF

Info

Publication number
CN116366303A
CN116366303A CN202310207251.6A CN202310207251A CN116366303A CN 116366303 A CN116366303 A CN 116366303A CN 202310207251 A CN202310207251 A CN 202310207251A CN 116366303 A CN116366303 A CN 116366303A
Authority
CN
China
Prior art keywords
template
sequence
preset
target
generate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310207251.6A
Other languages
Chinese (zh)
Inventor
李贝贝
杜卿芸
李晓慧
常玉洁
刘翱
黄翰媛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan University
Original Assignee
Sichuan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan University filed Critical Sichuan University
Priority to CN202310207251.6A priority Critical patent/CN116366303A/en
Publication of CN116366303A publication Critical patent/CN116366303A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1425—Traffic logging, e.g. anomaly detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/194—Calculation of difference between files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Probability & Statistics with Applications (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a network anomaly detection method, device, equipment and medium based on deep learning, wherein the method comprises the following steps: responding to the received initial log data, carrying out feature extraction on the initial log data according to a preset constant event template to generate a template symbol sequence, dividing the template symbol sequence according to a preset sliding window by setting a sliding rule to generate a target log sequence and a label template symbol, detecting the target log sequence according to a preset abnormality detection model to generate a target template symbol corresponding to the target log sequence, and determining network behavior abnormality corresponding to the initial log data under the condition that the target template symbol is not matched with the label template symbol. By analyzing the log data and extracting the characteristics, the log data abnormality is determined according to the abnormality detection model, so that the network abnormality behavior caused by the network attack is detected, and the network abnormality detection efficiency is improved.

Description

Network anomaly detection method, device, equipment and medium based on deep learning
Technical Field
The invention relates to the technical field of data detection, in particular to a network anomaly detection method, device, equipment and medium based on deep learning.
Background
In the related art, the detection of the network behavior abnormality is to analyze the log data, and then determine the abnormality of the network behavior according to the abnormality of the log data. Network behavior anomaly detection based on run-time logs is one of several key building blocks to ensure system security and post-event investigation. However, because the types of network behaviors of users in the network system are more, the richness of the log data is higher, the data needed for anomaly detection is rich, and the detection efficiency of the network behavior anomaly detection based on the log data is lower on the existing network system.
Disclosure of Invention
Aiming at the technical problem of low network anomaly detection efficiency based on deep learning in the prior art, the invention provides a network anomaly detection method, device, equipment and medium based on deep learning.
In order to achieve the above purpose, the invention is realized by the following technical scheme:
in a first aspect of the embodiment of the present invention, a network anomaly detection method based on deep learning is provided, where the method includes:
responding to the received initial log data, and extracting rules for the initial log data according to a preset constant event template to generate a template symbol sequence;
dividing the template sequence according to a preset sliding window by setting a sliding rule to generate a target log sequence and a label template, wherein the label template is the next template after the target log sequence in the template sequence;
detecting the target log sequence according to a preset abnormality detection model to generate a target template symbol corresponding to the target log sequence;
and under the condition that the target template symbol is not matched with the label template symbol, determining network behavior abnormality corresponding to the initial log data.
Optionally, the preset anomaly detection model is generated by the following steps:
acquiring a template training sequence of the preset abnormality detection model, wherein the template training sequence comprises a plurality of templates;
dividing the template training sequence according to the preset sliding window through the preset sliding rule to generate a training log sequence and training tag templates, wherein the training tag templates are the next templates after the training log sequence in the template training sequence;
extracting features of the training log sequence to generate a sequence vector and a counting vector corresponding to the training log sequence;
and carrying out model training on the preset abnormality detection model according to the sequence vector, the counting vector and the training tag template symbol so as to generate the trained preset abnormality detection model.
Optionally, the training the preset anomaly detection model according to the sequence vector, the count vector and the training tag template symbol to generate a trained preset anomaly detection model includes:
generating a first target vector according to the sequence vector, and generating a second target vector according to the count vector;
inputting the first target vector and the training tag template symbol to a time convolution neural network module of the preset anomaly detection model to generate a first hidden vector;
inputting the second target vector and the training tag template symbol to a circular convolution neural network module of the preset anomaly detection model to generate a second hidden vector;
and carrying out model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector so as to generate the trained preset anomaly detection model.
Optionally, the training the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate a trained preset anomaly detection model includes:
inputting the first hidden vector and the second hidden vector to a fully-connected neural network module of the preset anomaly detection model to generate a neural network output result;
and transmitting the neural network output result to an output layer of the preset abnormality detection model to generate the trained preset abnormality detection model.
Optionally, the responding to the received initial log data, extracting rules for the initial log data according to a preset constant event template, so as to generate a template symbol sequence, including:
according to the preset constant event template, extracting rules are carried out on the initial log data, and an event database corresponding to the initial log data is generated;
and generating the template sequence according to the sequence of the event database in the initial log data.
Optionally, the dividing the template sequence according to a preset sliding window by setting a sliding rule to generate a target log sequence and a tag template includes:
acquiring the window length and the sliding step length of the preset sliding window;
and dividing the template sequence according to the window length and the sliding step length to generate the target log sequence and the label template.
Optionally, the detecting the target log sequence according to a preset anomaly detection model to generate a target template corresponding to the target log sequence includes:
detecting the target log sequence according to the preset abnormality detection model, and determining a plurality of initial candidate templates;
and determining a plurality of candidate templates with highest probability from the plurality of initial candidate templates as the target templates.
In a second aspect of the embodiment of the present invention, there is provided a network anomaly detection device based on deep learning, the device including:
the determining module is used for responding to the received initial log data, extracting rules from the initial log data according to a preset constant event template to generate a template symbol sequence;
the first generation module is used for dividing the template sequence according to a preset sliding window by setting a sliding rule so as to generate a target log sequence and a label template, wherein the label template is the next template after the target log sequence in the template sequence;
the second generation module is used for detecting the target log sequence according to a preset abnormality detection model so as to generate a target template symbol corresponding to the target log sequence;
and the execution module is used for determining network behavior abnormality corresponding to the initial log data under the condition that the target template symbol is not matched with the label template symbol.
In a third aspect of the embodiment of the present invention, there is provided an electronic device, including:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the deep learning based network anomaly detection method of any one of the first aspect of the present disclosure.
In a fourth aspect of the embodiments of the present invention, there is provided a computer readable storage medium having stored thereon a terminal application control program, which when executed by a processor, implements the steps of the deep learning-based network anomaly detection method according to any one of the first aspects of the present disclosure.
The invention provides a network anomaly detection method and device based on deep learning, electronic equipment and a storage medium. Compared with the prior art, the method has the following beneficial effects:
according to the method, the received initial log data is responded, the extraction rule is carried out on the initial log data according to the preset constant event template, so that a template symbol sequence is generated, the template symbol sequence is segmented according to the preset sliding window by setting the sliding rule, so that a target log sequence and a tag template symbol are generated, wherein the tag template symbol is the next template symbol after the target log sequence in the template symbol sequence, the target log sequence is detected according to the preset abnormality detection model, so that a target template symbol corresponding to the target log sequence is generated, and network behavior abnormality corresponding to the initial log data is determined under the condition that the target template symbol is not matched with the tag template symbol. By analyzing the log data and extracting the characteristics, the log data abnormality is determined according to the abnormality detection model, so that the network abnormality behavior caused by the network attack is detected, and the network abnormality detection efficiency is improved.
Drawings
Fig. 1 is a flowchart of a network anomaly detection method based on deep learning according to the present invention.
Fig. 2 is a flowchart of another network anomaly detection method based on deep learning according to the present invention.
Fig. 3 is a block diagram of a network anomaly detection device based on deep learning according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a flowchart of a network anomaly detection method based on deep learning according to the present invention, please refer to fig. 1, and the present invention provides a network anomaly detection method based on deep learning, which includes the following steps.
In step S11, in response to the received initial log data, extraction rules are performed on the initial log data according to a preset constant event template, so as to generate a sequence of templates.
The embodiment is applied to terminal equipment, and initial log data is obtained by collecting log data generated by user equipment. The log data is unstructured text and comprises two parts, namely a constant part named as an event template and a change part named as a log parameter, wherein the constant part represents the unchanged amount in the recorded log data, and the log parameter part represents the data difference of each day in the log data, and the changed amount in the recorded log data is generated. By way of example, the log message "081109 203615 148INFO dfs data node packet responder: the packet responder 1 terminates for block blk_38865049064139660 "event recorded" the packet responder 1 terminates for block blk_3886509906413966 ", wherein" packet response procedure "," for "," block "," blk_ ", and" terminate "are constant event templates; "1" and "38865049064139660" are log parameters. The terminal device compares the initial log data, so that an event template in the initial log data can be determined.
In this embodiment, a preset constant event template extraction rule is set in the terminal device, where the rule includes a mapping relationship between a plurality of event templates and a plurality of templates, and the templates in the event templates are extracted according to the mapping relationship, so as to determine the templates corresponding to the event templates, and generate a sequence of templates corresponding to the event templates according to the arrangement sequence of the templates in the event templates. It should be noted that, in this embodiment, when extracting the template, feature extraction is sequentially performed from the event template according to the sequence of the event template in the initial log data, so as to generate a sequence of the template, where the sequence of the template may include a plurality of templates, and the sequence of each template is consistent with the sequence of the event template in the initial log data. By converting unstructured log data into structured data in the above manner, for example, an event template database may be generated according to an event template, and a mapping relationship between the event template and a template symbol may be established in the template database.
Optionally, the step S11 includes:
according to the preset constant event template, extracting rules are carried out on the initial log data, and an event database corresponding to the initial log data is generated;
and generating the template sequence according to the sequence of the event database in the initial log data.
For example, in this embodiment, the objective of log parsing is to extract an event template and a log parameter from each log data, determine a plurality of event templates in the initial log data according to a preset constant event template, and generate an event database. And generating a template symbol sequence corresponding to the initial log data according to the sequence of each event template in the initial log data.
In step S12, the template sequence is divided according to a preset sliding window by setting a sliding rule, so as to generate a target log sequence and a tag template.
For example, in this embodiment, the template sequence is segmented into a preset sliding window, and the template sequence is divided according to the preset sliding window to generate the target log sequence and the tag template. Wherein the tag template is the order in the sequence of the tag templateThe next template after the log sequence is marked. The preset sliding window is a movable sliding window, the window comprises a preset length and a unit step length of each sliding, after the template symbol sequence is segmented into the preset sliding window, the template symbol sequence is divided according to the preset length and the unit step length of the window, and a plurality of target log sequences and a plurality of tag template symbols which correspond to each other one by one can be generated based on the sliding sequence of the preset sliding window. It should be noted that, in this embodiment, when the template sequence is divided by the preset sliding window, according to the length of the template sequence, the window length and the unit sliding step length, the template may be divided into a plurality of log sequences, and when the number of log information corresponding to the template sequence is 6, the log information is respectively: e, e 1 ，e 2 ，e 3 ，e 4 ，e 2 ，e 3 The method comprises the steps of carrying out a first treatment on the surface of the When the window length corresponding to the preset sliding window is 2 and the unit sliding step length is 1, dividing the template symbol sequence according to the preset sliding window to generate 5 target log sequences, wherein the target log sequences are respectively as follows: e, e 1 e 2 ，e 2 e 3 ，e 3 e 4 ，e 4 e 2 ，e 2 e 3 And determining that the tag templates corresponding to the 5 target log sequences are respectively: e, e 2 ，e 3 ，e 4 ，e 2 ，e 3 . The window length and the unit sliding step length of the preset sliding window can be set according to actual needs, and the embodiment is not limited.
Optionally, the step S12 includes:
acquiring the window length and the sliding step length of the preset sliding window set;
and dividing the template sequence according to the window length and the sliding step length to generate the target log sequence and the label template.
For example, in this embodiment, the window length and the sliding step length of the preset sliding window are determined, the template sequence is divided according to the window length and the sliding step length until the preset sliding window slides onto the last template of the template sequence, so that the template sequence is divided into a plurality of target log sequences and a plurality of label templates, and it is understood that when the template sequence is divided according to the preset sliding window, the template sequence needs to be divided into a plurality of target log sequences according to the window length and the sliding step length of the preset sliding window, and the next template of the target log sequence in the template sequence is recorded as the label template corresponding to the target log sequence. The number of the target log sequences is related to the window length of the preset sliding window, the sliding step length and the length of the template symbol sequence.
In step S13, the target log sequence is detected according to a preset anomaly detection model, so as to generate a target template corresponding to the target log sequence.
In this embodiment, after the target log sequence is generated through the steps, a next template symbol of the target log sequence in the template symbol sequence is used as a tag template symbol, and the target log sequence is detected through a preset anomaly detection model, so as to generate a target template symbol corresponding to the target log sequence. The target template symbol is the next template symbol after the target log sequence in the template symbol sequence, which is predicted after the target log sequence is detected by a preset abnormality detection model. When the number of the target log sequences is multiple, the preset abnormality detection model predicts the next template symbol of the multiple target log sequences, so as to generate target template symbols corresponding to the multiple target log sequences one by one.
Alternatively, in one embodiment, the preset anomaly detection model may be generated by the steps comprising:
acquiring a template training sequence of the preset abnormality detection model, wherein the template training sequence comprises a plurality of templates;
dividing the template training sequence according to the preset sliding window through the preset sliding rule to generate a training log sequence and training tag templates, wherein the training tag templates are the next templates after the training log sequence in the template training sequence;
extracting features of the training log sequence to generate a sequence vector and a counting vector corresponding to the training log sequence;
and carrying out model training on the preset abnormality detection model according to the sequence vector, the counting vector and the training tag template symbol so as to generate the trained preset abnormality detection model.
It should be noted that, after the template training sequence of the preset anomaly detection model is obtained in the embodiment, model training is performed on the preset anomaly detection model according to the template training sequence, where the template training sequence is generated after processing training log data, and the processing manner of the training log data may refer to the processing manner of the initial log data in the above scheme, which is not described in detail in this embodiment.
The method for dividing the template sequence can refer to the method for dividing the template sequence in the above scheme, and will not be described in detail in this embodiment. After a template character training sequence is divided into a plurality of training log sequences and a plurality of training label template characters corresponding to each other one by one through a preset sliding window, feature extraction is carried out on the training log sequences, a sequence vector and a counting vector corresponding to the training log sequences are calculated and generated, wherein the sequence vector i identifies the sequence of each template character in the training log sequences, the counting vector identifies the occurrence times of each different type of template character in the training log sequences, the sequence vector, the counting vector and the training label template characters are input into a preset abnormality detection model, the preset abnormality detection model is trained, the corresponding relation between the training log sequences and the training label template characters is established through a deep learning method, a trained preset abnormality detection model is generated, and network abnormality detection is carried out on initial log data according to the trained preset abnormality detection model.
Optionally, in another embodiment, the foregoing step performs model training on the preset anomaly detection model according to the sequence vector, the count vector and the training tag template symbol to generate the trained preset anomaly detection model, including:
generating a first target vector according to the sequence vector, and generating a second target vector according to the count vector;
inputting the first target vector and the training tag template symbol to a time convolution neural network module of the preset anomaly detection model to generate a first hidden vector;
inputting the second target vector and the training tag template symbol to a circular convolution neural network module of the preset anomaly detection model to generate a second hidden vector;
and carrying out model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector so as to generate the trained preset anomaly detection model.
For example, in this embodiment, when a training log sequence is generated by swiping a preset sliding window, the next template symbol of the training log sequence in the template symbol training sequence is recorded and used as a label to train a preset anomaly detection model, that is, the preset anomaly detection model is trained through the training log sequence and the training label template symbol, and after corresponding sequence vectors and count vectors are generated according to the training log sequence through the steps, the sequence vectors are converted into first target vectors according to the training log sequence:
s t ＝[m t-h ，m t-h+1 ，...，m t-1 ]
wherein m is i The value of the template symbol at the time stamp i is represented, and h is the window length of a preset sliding window;
a set of n different templates for each training log sequence is identified by i ∈E n )：
E n ＝{e 1 ，e 2 ，...，e n }
According to the log sequence of each trainingEach template character e in the column n Generating a count vector corresponding to the training log sequence according to the occurrence times in the training log sequence, wherein the count vector can be expressed as:
p t ＝[count(e 1 )，count(e 2 )，...，count(e n )]
wherein count (e i ) Representing template e in training log sequence i The number of occurrences, for example, given a training log sequence of: [1,2,3]The corresponding relation of the template symbols is set by comparison, and the sequence vector of the training log sequence can be determined as s according to the training log sequence t ＝[e 1 ，e 2 ，e 1 ]According to each template e in the sequence vector i The number of occurrences, determining the corresponding count vector as: p is p t ＝[2，1,0,0,0]Where n=5.
Will sequence vector s t And count vector p t Respectively inputting the output results of the TCN module and the ACNN module into an MLP module of a preset abnormality detection model, wherein the MLP module consists of two TCN modules and ACNN modules which are completely connected, and the TCN module and the ACNN module are exemplified:
h 1 ＝TCN(s t )
h 2 ＝ACNN(p t )
τ＝MLP(h 1 ；h 2 )
wherein h is 1 For the first hidden vector sum h 2 For the second hidden vector, τ represents the output of the MLP module, and then τ is forwarded to the output layer of the preset anomaly detection model, and model training is performed on the preset anomaly detection model based on τ of the output layer, so as to generate a trained preset anomaly detection model, which needs to be described that, in this embodiment, the trained preset anomaly detection model is used for detecting, according to the log sequence, the tag template symbol after the log sequence in the template symbol sequence, and comparing the generated target template symbol with the tag template symbol after the log sequence in the template symbol sequence, when the target isWhen the template symbol is matched with the label template symbol, determining that no network abnormal behavior exists in the initial log data, and when the target template symbol is not matched with the label template symbol, determining that the network abnormal behavior exists in the initial log data.
Optionally, in another embodiment, the foregoing step performs model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model, including:
inputting the first hidden vector and the second hidden vector to a fully-connected neural network module of the preset anomaly detection model to generate a neural network output result;
transmitting the neural network output result to an output layer of the preset anomaly detection model, and training the preset anomaly detection model based on the output layer to generate the trained preset anomaly detection model.
By way of example, in the present embodiment, the first hidden vector h is generated through the above steps 1 And a second hidden vector h 2 Then, through the fully connected neural network module of the preset abnormality detection model, namely the MLP module pair h 1 And h 2 Fusing to generate a neural network output result tau, forwarding the tau to an output layer of a preset anomaly detection model, performing model training on the preset anomaly detection model based on the tau of the output layer, establishing a corresponding relation between a training log sequence and a training label template, and generating a trained preset anomaly detection model.
Optionally, the step S13 includes:
detecting the target log sequence according to the preset abnormality detection model, and determining a plurality of initial candidate templates;
and determining a plurality of candidate templates with highest probability from the plurality of initial candidate templates as the target templates.
In this embodiment, the target log sequence is detected by a preset anomaly detection model to determine a plurality of initial candidate templates corresponding to the target log sequence, where the plurality of initial candidate templates are tag templates that may occur after the target log sequence in the template sequence. And sorting the plurality of initial candidate templates according to the occurrence times of the plurality of initial candidate templates, wherein the more the occurrence times of the templates are, the higher the corresponding probability is, the more the corresponding sorting is forward, and the plurality of candidate templates with the largest occurrence times in the sequence are used as target templates. For example, the number of candidate templates extracted is determined to be 3, and after the probability sequence of the candidate templates is obtained according to the ranking, the first 3 candidate templates with highest occurrence probability are extracted as target templates.
In step S14, if the target template and the tag template do not match, determining that the network behavior corresponding to the initial log data is abnormal.
In this embodiment, the target template generated by the preset anomaly detection model is compared with a next template (tag template) corresponding to the target log sequence in the template sequence, and when it is determined that the tag template and the target template are not matched, that is, the target template generated by the preset anomaly detection model is not the next template corresponding to the target log sequence in the template sequence, the network behavior anomaly corresponding to the initial log data is determined.
According to the scheme, log analysis is conducted on the initial log data in response to the received initial log data, so that an event template in the initial log data is determined, feature extraction is conducted on the event template according to a preset constant event template extraction rule to generate a template symbol sequence, the template symbol sequence is divided according to a preset sliding window by setting a sliding rule to generate a target log sequence and a tag template symbol, wherein the tag template symbol is the next template symbol after the target log sequence in the template symbol sequence, detection is conducted on the target log sequence according to a preset abnormality detection model to generate a target template symbol corresponding to the target log sequence, and network behavior abnormality corresponding to the initial log data is determined under the condition that the target template symbol is not matched with the tag template symbol. By analyzing the log data and extracting the characteristics, the log data abnormality is determined according to the abnormality detection model, so that the network abnormality behavior caused by the network attack is detected, and the network abnormality detection efficiency is improved.
Fig. 2 is a flowchart of another network anomaly detection method based on deep learning according to the present invention, as shown in fig. 2, the network anomaly detection method based on deep learning includes:
(1) Journal analysis: the log message is an unstructured text comprising two parts, a constant part named event template and a variable part named log parameter. The goal of log parsing is to extract event templates and log parameters from each log message and create a dictionary that maps each event template to a unique template. In this way, unstructured log messages will be converted into structured data. FIG. 2 shows an example of parsing an HDFS log. For example, log message "081109 203615 148INFO dfs data node packet responder: the packet transponder 1 terminates for block blk 38865049064139660 "event recorded" packet transponder 1 terminates for block blk 3886509906413966 ", wherein the words" packet response procedure "," for "," block "," blk_ ", terminate" are constant event templates, extracted as "packet response procedure </x > terminates for block blk_ >. Note that, where </indicates log parameters, i.e., "1" and "38865049064139660".
(2) Feature extraction: the purpose of feature extraction is to convert structured log data into a numerical vector. The log data is first partitioned into a set of sliding windows. In the set of sliding windows, log messages located on the same window form a log sequence. As shown in fig. 2, if the log message series is [1,2,3,4,5], the window size is set to 3, and the step size is set to 1, we can get three log sequences, [1,2,3], [,3,4] and [,4,5]. Further, two vectors are calculated from each derived log sequence, the sequence vector representing the sequence of templates in the log sequence, and the count vector representing the number of times each template occurs in the log sequence. In addition, the next template for each window is recorded as a tag to train the anomaly detection model. Specifically, given a log sequence, the sequence vector is expressed as:
s t ＝[m t-h ，m t-h+1 ，...，m t-1 ]
wherein m is i The value of the template symbol at the time stamp i is represented, and h is the window size of a preset sliding window set;
a set of n different templates in the log data set is identified by (where m i ∈E n )：
E n ＝{e 1 ，e 2 ，...，e n }
The count vector is expressed as:
p t ＝[count(e 1 )，count(e 2 )，...，count(e n )]
wherein count (e i ) Representing the number of occurrences of each template in a log sequence, e.g. a given log sequence [1,2,3]]From the log sequence s can be determined t ＝[e 1 ，e 2 ，e 3 ]And p t ＝[2，1,0,0,0]Where n=5.
Will sequence vector S t And count vector p t Respectively inputting the output results of the TCN module and the ACNN module into an MLP module of a preset abnormality detection model, wherein the MLP module consists of two TCN modules and ACNN modules which are completely connected, and the TCN module and the ACNN module are exemplified:
h 1 ＝TCN(s t )
h 2 ＝ACNN(p t )
τ＝MLP(h 1 ；h 2 )
wherein h is 1 And h 2 For hiding the vector, τ represents the output of the MLP module, and then τ is forwarded to the output layer of the preset anomaly detection model, given the past window W, m is obtained t And by:
P(m t ＝e i |w)＝softmax(τ)
during training, P (m t ＝e i I w) cross entropy loss and the label obtained by the feature extraction section. And uses the SGD to train parameters of the TCN-ACNN model. At the time of reasoning, the method is applied to P (m t ＝e i I w) ordering and selecting e i The top c templates with highest probability are used as a plurality of candidate templates.
If the template of the incoming log message is not one of these candidates, then
Figure BDA0004111338990000141
The log sequence containing the log message is classified as abnormal.
(3) Abnormality detection: anomaly detection focuses on identifying data patterns that differ significantly from normal observations. The purpose of log-based anomaly detection is to reveal collective anomalies, i.e., anomaly log events that occur in a sequence. Log messages are predicted relative to a historical log sequence. Specifically, if the template of an incoming log message is different from the predicted template, the log sequence containing the log message will be considered abnormal
Fig. 3 is a block diagram of a network anomaly detection device based on deep learning according to the present invention, and as shown in fig. 3, the device 100 includes: a determination module 110, a first generation module 120, a second generation module 130, and an execution module 140.
The determining module 110 is configured to respond to the received initial log data, and perform extraction rules on the initial log data according to a preset constant event template, so as to generate a template symbol sequence;
the first generating module 120 is configured to segment the sequence of templates according to a preset sliding window by setting a sliding rule, so as to generate a target log sequence and a tag template, where the tag template is a next template after the target log sequence in the sequence of templates;
the second generating module 130 is configured to detect the target log sequence according to a preset anomaly detection model, so as to generate a target template corresponding to the target log sequence;
and the execution module 140 is configured to determine that the network behavior corresponding to the initial log data is abnormal if the target template symbol does not match the tag template symbol.
Optionally, the preset anomaly detection model is generated by the following steps:
acquiring a template training sequence of the preset abnormality detection model, wherein the template training sequence comprises a plurality of templates;
dividing the template training sequence according to the preset sliding window through the preset sliding rule to generate a training log sequence and training tag templates, wherein the training tag templates are the next templates after the training log sequence in the template training sequence;
extracting features of the training log sequence to generate a sequence vector and a counting vector corresponding to the training log sequence;
and carrying out model training on the preset abnormality detection model according to the sequence vector, the counting vector and the training tag template symbol so as to generate the trained preset abnormality detection model.
Optionally, the training the preset anomaly detection model according to the sequence vector, the count vector and the training tag template symbol to generate a trained preset anomaly detection model includes:
generating a first target vector according to the sequence vector, and generating a second target vector according to the count vector;
inputting the first target vector and the training tag template symbol to a time convolution neural network module of the preset anomaly detection model to generate a first hidden vector;
inputting the second target vector and the training tag template symbol to a circular convolution neural network module of the preset anomaly detection model to generate a second hidden vector;
and carrying out model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector so as to generate the trained preset anomaly detection model.
Optionally, the training the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate a trained preset anomaly detection model includes:
inputting the first hidden vector and the second hidden vector to a fully-connected neural network module of the preset anomaly detection model to generate a neural network output result;
transmitting the neural network output result to an output layer of the preset anomaly detection model, and training the preset anomaly detection model based on the output layer to generate the trained preset anomaly detection model.
Optionally, the determining module 110 is configured to:
generating an event database according to the event template;
and extracting the template symbols in the event database according to the preset constant event template extraction rule to generate the template symbol sequence.
Optionally, the first generating module 120 is configured to:
acquiring the window length and the sliding step length of the preset sliding window;
and dividing the template sequence according to the window length and the sliding step length to generate the target log sequence and the label template.
Optionally, the second generating module 130 is configured to:
detecting the target log sequence according to the preset abnormality detection model, and determining a plurality of initial candidate templates;
and determining a plurality of candidate templates with highest probability from the plurality of initial candidate templates as the target templates.
According to the method, the received initial log data is responded, the extraction rule is carried out on the initial log data according to the preset constant event template, so that a template symbol sequence is generated, the template symbol sequence is segmented according to the preset sliding window by setting the sliding rule, so that a target log sequence and a tag template symbol are generated, wherein the tag template symbol is the next template symbol after the target log sequence in the template symbol sequence, the target log sequence is detected according to the preset abnormality detection model, so that a target template symbol corresponding to the target log sequence is generated, and network behavior abnormality corresponding to the initial log data is determined under the condition that the target template symbol is not matched with the tag template symbol. Therefore, the efficiency of detecting the data abnormality is improved by analyzing and extracting the characteristics of the log data in the terminal equipment and determining the abnormality of the log data according to the abnormality detection model.
In another exemplary embodiment, a computer program product is also provided, comprising a computer program executable by a programmable apparatus, the computer program having code portions for performing the above-described deep learning based network anomaly detection method when executed by the programmable apparatus.
In another exemplary embodiment, there is also provided an electronic device including:
a memory having a computer program stored thereon;
and the processor is used for executing the computer program in the memory to perform the steps of the network anomaly detection method based on deep learning.
With the above-described preferred embodiments according to the present application as a teaching, the related workers can make various changes and modifications without departing from the scope of the technical idea of the present application. The technical scope of the present application is not limited to the contents of the specification, and must be determined according to the scope of claims.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (10)

1. A network anomaly detection method based on deep learning, the method comprising:
responding to the received initial log data, and extracting rules for the initial log data according to a preset constant event template to generate a template symbol sequence;
dividing the template sequence according to a preset sliding window by setting a sliding rule to generate a target log sequence and a label template, wherein the label template is the next template after the target log sequence in the template sequence;
detecting the target log sequence according to a preset abnormality detection model to generate a target template symbol corresponding to the target log sequence;
and under the condition that the target template symbol is not matched with the label template symbol, determining network behavior abnormality corresponding to the initial log data.
2. The method of claim 1, wherein the predetermined anomaly detection model is generated by:
acquiring a template training sequence of the preset abnormality detection model, wherein the template training sequence comprises a plurality of templates;
dividing the template training sequence according to the preset sliding window through the preset sliding rule to generate a training log sequence and training tag templates, wherein the training tag templates are the next templates after the training log sequence in the template training sequence;
extracting features of the training log sequence to generate a sequence vector and a counting vector corresponding to the training log sequence;
and carrying out model training on the preset abnormality detection model according to the sequence vector, the counting vector and the training tag template symbol so as to generate the trained preset abnormality detection model.
3. The method of claim 2, wherein the model training the preset anomaly detection model based on the sequence vector, the count vector, and the training tag template to generate the trained preset anomaly detection model comprises:
generating a first target vector according to the sequence vector, and generating a second target vector according to the count vector;
inputting the first target vector and the training tag template symbol to a time convolution neural network module of the preset anomaly detection model to generate a first hidden vector;
inputting the second target vector and the training tag template symbol to a circular convolution neural network module of the preset anomaly detection model to generate a second hidden vector;
and carrying out model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector so as to generate the trained preset anomaly detection model.
4. A method according to claim 3, wherein the model training the preset anomaly detection model based on the first and second concealment vectors to generate the trained preset anomaly detection model comprises:
inputting the first hidden vector and the second hidden vector to a fully-connected neural network module of the preset anomaly detection model to generate a neural network output result;
and transmitting the neural network output result to an output layer of the preset abnormality detection model to generate the trained preset abnormality detection model.
5. The method of claim 1, wherein the extracting the initial log data according to a predetermined constant event template in response to the received initial log data to generate a sequence of templates comprises:
according to the preset constant event template, extracting rules are carried out on the initial log data, and an event database corresponding to the initial log data is generated;
and generating the template sequence according to the sequence of the event database in the initial log data.
6. The method according to claim 1, wherein the dividing the sequence of templates by setting a sliding rule according to a preset sliding window to generate a target log sequence and a tag template includes:
acquiring the window length and the sliding step length of the preset sliding window;
and dividing the template sequence according to the window length and the sliding step length to generate the target log sequence and the label template.
7. The method of claim 1, wherein the detecting the target log sequence according to a preset anomaly detection model to generate a target template corresponding to the target log sequence comprises:
detecting the target log sequence according to the preset abnormality detection model, and determining a plurality of initial candidate templates;
and determining a plurality of candidate templates with highest probability from the plurality of initial candidate templates as the target templates.
8. A network anomaly detection device based on deep learning, the device comprising:
the determining module is used for responding to the received initial log data, extracting rules from the initial log data according to a preset constant event template to generate a template symbol sequence;
the first generation module is used for dividing the template sequence according to a preset sliding window by setting a sliding rule so as to generate a target log sequence and a label template, wherein the label template is the next template after the target log sequence in the template sequence;
the second generation module is used for detecting the target log sequence according to a preset abnormality detection model so as to generate a target template symbol corresponding to the target log sequence;
and the execution module is used for determining network behavior abnormality corresponding to the initial log data under the condition that the target template symbol is not matched with the label template symbol.
9. An electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the deep learning based network anomaly detection method of any one of claims 1-7.
10. A computer-readable storage medium, wherein a terminal application control program is stored on the computer-readable storage medium, and the terminal application control program, when executed by a processor, implements the steps of the deep learning-based network anomaly detection method of any one of claims 1 to 7.
CN202310207251.6A 2023-03-06 2023-03-06 Network anomaly detection method, device, equipment and medium based on deep learning Pending CN116366303A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310207251.6A CN116366303A (en) 2023-03-06 2023-03-06 Network anomaly detection method, device, equipment and medium based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310207251.6A CN116366303A (en) 2023-03-06 2023-03-06 Network anomaly detection method, device, equipment and medium based on deep learning

Publications (1)

Publication Number Publication Date
CN116366303A true CN116366303A (en) 2023-06-30

Family

ID=86911565

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310207251.6A Pending CN116366303A (en) 2023-03-06 2023-03-06 Network anomaly detection method, device, equipment and medium based on deep learning

Country Status (1)

Country Link
CN (1) CN116366303A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117395071A (en) * 2023-11-16 2024-01-12 南方电网数字电网集团信息通信科技有限公司 Abnormality detection method, abnormality detection device, abnormality detection equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117395071A (en) * 2023-11-16 2024-01-12 南方电网数字电网集团信息通信科技有限公司 Abnormality detection method, abnormality detection device, abnormality detection equipment and storage medium
CN117395071B (en) * 2023-11-16 2024-05-14 南方电网数字电网集团信息通信科技有限公司 Abnormality detection method, abnormality detection device, abnormality detection equipment and storage medium

Similar Documents

Publication Publication Date Title
US20240028571A1 (en) Automatic entity resolution with rules detection and generation system
CN108737406B (en) Method and system for detecting abnormal flow data
US20170026390A1 (en) Identifying Malware Communications with DGA Generated Domains by Discriminative Learning
US20170063893A1 (en) Learning detector of malicious network traffic from weak labels
CN111600919B (en) Method and device for constructing intelligent network application protection system model
CN109831460B (en) Web attack detection method based on collaborative training
CN112966088B (en) Unknown intention recognition method, device, equipment and storage medium
CN112199670A (en) Log monitoring method for improving IFOREST (entry face detection sequence) to conduct abnormity detection based on deep learning
CN116366303A (en) Network anomaly detection method, device, equipment and medium based on deep learning
CN116318830A (en) Log intrusion detection system based on generation of countermeasure network
CN112052453A (en) Webshell detection method and device based on Relief algorithm
Sujana et al. Temporal based network packet anomaly detection using machine learning
CN111291078A (en) Domain name matching detection method and device
CN116232708A (en) Attack chain construction and attack tracing method and system based on text threat information
CN113360899B (en) Machine behavior recognition method and system
CN115859191A (en) Fault diagnosis method and device, computer readable storage medium and computer equipment
CN112883703B (en) Method, device, electronic equipment and storage medium for identifying associated text
Tsai et al. Toward more generalized malicious url detection models
CN114329453A (en) Anomaly detection method based on system log
Nandakumar et al. A Novel Approach to User Agent String Parsing for Vulnerability Analysis Using Multi-Headed Attention
Khatun et al. An Approach to Detect Phishing Websites with Features Selection Method and Ensemble Learning
JP2021022264A (en) Text data analysis system, text data analysis method, and fault response recommend system
Schwenk et al. Classification of structured validation data using stateless and stateful features
CN113347021B (en) Model generation method, collision library detection method, device, electronic equipment and computer readable storage medium
CN114666391B (en) Method, device, equipment and storage medium for determining access track

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination