CN112887304B - WEB application intrusion detection method and system based on character-level neural network - Google Patents

WEB application intrusion detection method and system based on character-level neural network Download PDF

Info

Publication number
CN112887304B
CN112887304B CN202110096602.1A CN202110096602A CN112887304B CN 112887304 B CN112887304 B CN 112887304B CN 202110096602 A CN202110096602 A CN 202110096602A CN 112887304 B CN112887304 B CN 112887304B
Authority
CN
China
Prior art keywords
data
attack
neural network
network
network traffic
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110096602.1A
Other languages
Chinese (zh)
Other versions
CN112887304A (en
Inventor
吴晓明
李阳
王睿思
汪付强
李昌盛
朱庆晨
王京守
张鹏
刘祥志
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Computer Science Center National Super Computing Center in Jinan
Original Assignee
Shandong Computer Science Center National Super Computing Center in Jinan
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Computer Science Center National Super Computing Center in Jinan filed Critical Shandong Computer Science Center National Super Computing Center in Jinan
Priority to CN202110096602.1A priority Critical patent/CN112887304B/en
Publication of CN112887304A publication Critical patent/CN112887304A/en
Application granted granted Critical
Publication of CN112887304B publication Critical patent/CN112887304B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/14Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1416Event detection, e.g. attack signature detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/02Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
    • H04L63/0227Filtering policies
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/02Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
    • H04L63/0227Filtering policies
    • H04L63/0263Rule management
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/02Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]

Landscapes

  • Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Security & Cryptography (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The application discloses a WEB application intrusion detection method and a system based on a character-level neural network, which receive network traffic data; normalizing the network flow data; carrying out blacklist pre-detection on the normalized network traffic data; if the network traffic data is matched with any one of the blacklists, an intrusion warning is output; otherwise, entering the next step; carrying out rule filtering detection on the network traffic data detected by the blacklist; if the rule filtering detection is passed, entering the next step; otherwise, outputting an intrusion warning; inputting the network flow data detected by rule filtering into a trained character-level-based cyclic neural network, and outputting an intrusion detection result by the neural network; recording the intrusion detection result into a log, forwarding the network traffic data with the detection result of safety to a WEB server, intercepting the network traffic data with the detection result of unsafe, and displaying an intrusion warning.

Description

WEB application intrusion detection method and system based on character-level neural network
Technical Field
The application relates to the technical field of network security and machine learning, in particular to a WEB application intrusion detection method and system based on a character-level neural network.
Background
The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.
With the development of internet industrial technology, WEB application has become one of the most influential network application technologies in the internet era, and the traditional C/S architecture is gradually replaced by a brand-new B/S architecture mode, i.e. a browser and server architecture mode, in the B/S architecture, a working interface of a user and a small part of functions and business logic are unified by the browser, and main business and business logic are handed over to the server. The B/S architecture unifies clients on a user level, the clients can quickly access a large number of WEB applications with powerful functions only by one browser, core functions of the applications and hardware resources required by calculation are concentrated on a server, the cost and workload of system maintenance and upgrading are reduced, and the overall cost of users is reduced. Today, a large number of powerful WEB applications run on the internet.
With the rapid development of WEB applications. A large number of attack means and intrusion methods for WEB applications also appear, and in OWASPTop10, 10 attack threats for WEB applications are defined, which are: injection (Injection), failed Authentication (Broken Authentication), sensitive information leakage (Sensitive Data Exposure), XML External Entities (XML External Entities), failed Access Control (Broken Access Control), security configuration error (Broken Access Control), cross-Site Scripting (Cross-Site XSS), insecure Deserialization (Insecure Deserialization), use of Components with Known Vulnerabilities (Using Components with Known Vulnerabilities), and inadequate Logging and Monitoring (instrumentation Logging & Monitoring).
Nowadays, attack means and attack methods aiming at WEB application are also rapidly iterated, and a large number of attack means and attack modes resisting a traditional WEB intrusion detection system appear. The attack modes can avoid a part of WEB intrusion detection systems and cause serious threat to WEB application safety by modifying attack load, detecting bugs in rules by using a traditional WEB intrusion detection method, replacing spaces by special codes and annotators, mixing the upper case and the lower case, labeling malformations, double writing and using escape characters and other methods.
However, a large number of methods for detecting intrusion at the WEB application level still remain in the stage based on the local policy and the filtering rule. Although intrusion detection methods based on rule filtering can defend most common attack modes, the deadly rule filtering modes are easily bypassed by attackers in various modes. Therefore, a new detection method capable of adapting to the current attack mode and attack means which change rapidly is urgently needed by the intrusion detection technology of the WEB application level.
Disclosure of Invention
In order to overcome the defects of the prior art, the application provides a WEB application intrusion detection method and a system based on a character-level neural network; the method can detect and intercept the attack load aiming at the WEB application, and has higher detection rate particularly for the attack load of text type, such as attack forms of remote command execution, injection, cross-site scripting, deserialization and the like.
In a first aspect, the application provides a WEB application intrusion detection method based on a character-level neural network;
a WEB application intrusion detection method based on a character-level neural network comprises the following steps:
receiving network flow data; normalizing the network flow data;
carrying out blacklist pre-detection on the normalized network traffic data; if the network traffic data is matched with any one of the blacklists, an intrusion warning is output; otherwise, entering the next step;
carrying out rule filtering detection on the network traffic data detected by the blacklist; if the rule filtering detection is passed, entering the next step; otherwise, outputting an intrusion warning;
inputting the network flow data detected by rule filtering into a trained character-level-based cyclic neural network, and outputting an intrusion detection result by the neural network;
recording the intrusion detection result into a log, forwarding the network traffic data with the detection result of safety to a WEB server, intercepting the network traffic data with the detection result of unsafe, and displaying an intrusion warning.
In a second aspect, the present application provides a WEB application intrusion detection system based on a character-level neural network;
WEB application intrusion detection system based on character level neural network includes:
a normalization processing module configured to: receiving network traffic data; normalizing the network flow data;
a preamble detection module configured to: carrying out blacklist pre-detection on the normalized network traffic data; if the network traffic data is matched with any one of the blacklists, an intrusion warning is output; otherwise, entering a filtering detection module;
a filter detection module configured to: carrying out rule filtering detection on the network traffic data detected by the blacklist; if the rule filtering detection is passed, entering an intrusion detection module; otherwise, outputting an intrusion warning;
an intrusion detection module configured to: inputting the network flow data detected by rule filtering into a trained character-level-based cyclic neural network, and outputting an intrusion detection result by the neural network;
an output module configured to: and recording the intrusion detection result into a log, forwarding the network traffic data with the detection result of safety to a WEB server, intercepting the network traffic data with the detection result of unsafe, and displaying an intrusion warning.
In a third aspect, the present application further provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein a processor is connected to the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so as to make the electronic device execute the method according to the first aspect.
In a fourth aspect, the present application also provides a computer-readable storage medium for storing computer instructions which, when executed by a processor, perform the method of the first aspect.
In a fifth aspect, the present application also provides a computer program (product) comprising a computer program for implementing the method of any of the preceding first aspects when run on one or more processors.
Compared with the prior art, the beneficial effects of this application are:
1. the intrusion detection method adopts a machine learning technology as a core technology, uses a classifier based on a character-level recurrent neural network as a core part of the intrusion detection method, has great difference from the traditional rule-based intrusion detection method principle, and has the advantages that the method can be used for detecting unknown network attack behaviors, namely, the method has the advantages of blocking the attack means adopting the bypass technology structure and attacking by 0 day.
2. The method can adopt a customized deployment mode, namely customized deployment is carried out according to the WEB application site requirements, a training set is generated on the WEB application site in a customized mode through an attack data generator and a normal data generator, and a customized neural network weight file is obtained through character-level cyclic neural network training so as to improve the intrusion detection rate and reduce the false alarm rate and the false missing rate.
3. The intrusion detection method is realized by adopting a two-classifier based on the character-level recurrent neural network, has the characteristic of predicting unknown attack behaviors by the known attack behaviors, and does not need maintenance and upgrading operation for a long time. During maintenance and upgrading, maintenance can be completed only by updating the weight file of the neural network, and compared with the traditional intrusion detection method, the method is extremely low in maintenance cost.
4. The intrusion detection method has the characteristics of light weight, high speed, easiness in use and strong compatibility, the core implementation method is realized based on a Pythrch dynamic neural network, various computing environments are supported, and cloud deployment and cloud access modes can be adopted.
5. Compared with the traditional WEB intrusion detection method based on rule filtering and local strategy, the method obtains inspiration from the field of natural language processing, constructs a two-classifier through a character-level recurrent neural network, and can be used for intercepting special attack loads generated by methods of special codes, annotation symbols replacing spaces, case and case mixing, malformed tags, double writing, escape characters and the like. Or the method is used for intercepting unexpected attack behaviors aiming at the text class of the WEB application, namely WEB 0day attack based on text load, and can adapt to the current attack mode and attack means which change rapidly.
Advantages of additional aspects of the application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application.
FIG. 1 is a block diagram of the present application;
FIG. 2 is a flowchart of the present application as a whole;
FIG. 3 is a flow chart of a training data generation phase of the present application;
FIG. 4 is a block diagram of a character level recurrent neural network as described herein;
FIG. 5 is a flow chart of the classifier training phase of the present application;
FIG. 6 is a process flow diagram of the deployment phase of the present application;
Detailed Description
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise, and it should be understood that the terms "comprises" and "comprising", and any variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The embodiments and features of the embodiments in the present application may be combined with each other without conflict.
Example one
The embodiment provides a WEB application intrusion detection method based on a character-level neural network;
a WEB application intrusion detection method based on a character-level neural network comprises the following steps:
s101: receiving network traffic data; normalizing the network flow data;
s102: carrying out blacklist pre-detection on the normalized network traffic data; if the network traffic data is matched with any one of the blacklists, an intrusion warning is output; otherwise, entering the next step;
s103: carrying out rule filtering detection on the network traffic data detected by the blacklist; if the rule filtering detection is passed, entering the next step; otherwise, outputting an intrusion warning;
s104: inputting the network flow data detected by rule filtering into a trained character-level-based cyclic neural network, and outputting an intrusion detection result by the neural network;
s105: recording the intrusion detection result into a log, forwarding the network traffic data with the detection result of safety to a WEB server, intercepting the network traffic data with the detection result of unsafe, and displaying an intrusion warning.
As one or more embodiments, the normalizing the network traffic data of S101 is performed; the method specifically comprises the following steps:
unpacking the network traffic data into readable data, storing the readable data in a json format in a key-value pair mode, wherein the formatted data describing the single network traffic comprises the following steps: source address, destination address, source port number, destination port number, network protocol employed, original payload, encoded payload, and flag bits.
Further, the payload is processed by a one-hot coding method and stored in the coded payload. After the above processing, the normalized network traffic data includes eight parts, i.e., the source address, the destination address, the source port number, the destination port number, the adopted network protocol, the original load, the encoded load, and the flag bit, and is sent to the blacklist for pre-detection.
As one or more embodiments, the S102: carrying out blacklist pre-detection on the normalized network traffic data; if the network flow data is matched with any item in the blacklist, an intrusion warning is output; otherwise, entering the next step; the blacklist comprises two parts, wherein one part is a blacklist built in the detection system, is generated by the system and is stored in a source ip address list mode. The other part is a user-defined blacklist which is generated by a user and can store characteristic items describing network traffic data packets: source address, source port, destination address, destination port, network protocol, and payload keywords.
As one or more embodiments, the S103: carrying out rule filtering detection on the network traffic data detected by the blacklist; the method specifically comprises the following steps:
firstly, matching a blacklist generated by a system, and directly judging attack traffic when a source ip address of network traffic data is found to be in the blacklist generated by the system;
and the second step is matched with a blacklist defined by a user, wherein the source address, the source port, the destination address, the destination port and the network protocol feature item adopt a strict matching mode. And matching the load keyword characteristic items in a regular mode. And if the matching is successful, judging the flow as the attack flow.
As one or more embodiments, the S104: inputting the network flow data detected by rule filtering into a trained character-level-based cyclic neural network, and outputting an intrusion detection result by the neural network; the specific training steps of the trained character-level-based recurrent neural network comprise:
constructing a training set; the training set includes: positive sample data and negative sample data;
inputting the training set into a cyclic neural network based on character level, training the network, and adjusting network parameters;
after each iteration is set for times, the loss function value of the network and the set threshold value are judged;
when the loss function value of the neural network is larger than a set threshold value, continuing iterative training;
when the loss function value of the neural network is smaller than a set threshold value, stopping training and outputting neural network parameters;
and deploying the neural network parameters to the character-level-based cyclic neural network, testing the neural network, obtaining the trained character-level-based cyclic neural network if the test is passed, and continuing training if the test is not passed.
Character-level recurrent neural networks are known in the art, and are based on recurrent neural networks known in the art, in which the text used for iteration is processed as a sequence of characters, i.e. words in the language model at the word level are replaced by characters, and the input sequence of the model is changed from a sequence of words to a sequence of characters.
The character-level cyclic neural network-based characteristics are as follows: the length of the input vector is the length of a self-defined character expansion table 95, the characteristic quantity of a hidden layer is 128, the output layer outputs a vector with the length of 2, and the vector is the probability that the input predicted by the classifier belongs to attack and normal flow respectively.
The two classifiers receive X sets of One-Hot encoded attack data and normal data, iterate n times at a learning rate of lr, where three parameters of lr, X, and n need to be adjusted according to the evaluation method described in S122, and generally initialize values n =1000000, lr =0.0000001, and X =100000.
The evaluation method comprises the following steps: the evaluation method is used for evaluating and detecting the quality of the model obtained by the training of the two classifiers. The evaluation method comprises evaluation during training and evaluation after training.
The evaluation during training is initialized when the character-level recurrent neural network starts training, and the following variables are defined during initialization:
Δ lr = lr 0.01 (1% of lr)
Δ n = n 0.1 (10% of n)
Checkpoint =100 (loss rate compression ratio, which is an integer divisible by n, smaller constant)
Δ loss = loss per Checkpoint from the current iteration
Upperadjust =0.2 (upper bound adjustment threshold, typically 0.2)
LowerAdjust =0.0001 (lower bound adjustment threshold, typically 0.0001)
FinalLoss =0.5 (final loss rate threshold, used to reflect the final accuracy of the model, is typically 0.5.)
And (2) detecting the value of the delta loss by the evaluation function every iteration Checkpoint along with the training of the neural network, wherein the value of the delta loss can reflect the descending speed of the loss rate in the training process of the neural network to a certain extent, if the value of the delta loss is greater than the initial InitAdjust value initialized during the training, before the next iteration starts, lr = lr- Δ lr, namely the learning rate is reduced by 1%, and then the next iteration is continued. If the value of Δ loss is smaller than the LowerAdjust value initialized during training, lr = lr + Δ lr before the start of the next iteration, that is, the learning rate is improved by 1%.
After the neural network model is iterated for n times, the evaluation method checks the final loss rate of the model iteration (the loss rate of the last round of training), if the loss rate is greater than FinalLoss, the model is iterated for delta n times continuously, the model precision is continuously improved until the final loss rate is less than or equal to FinalLoss, and the training is finished.
After training, entering a post-training evaluation method, wherein the post-training evaluation method uses a training data generation phase method to generate 2X (X is a positive integer, generally 5000), wherein X pieces of attack and normal data are sent to a trained neural network for prediction, and finally a floating point number Cfinal is output and can be used for reflecting the quality degree of a model obtained by training. Where Cfinal = number of predicted correct data/2X.
Because a large amount of attack data for WEB application is text-based, words of the attack text generally have short-term front-back relevance but do not have long-term relevance, such as attack load 'or 1=1' in SQL injection attack and < script > statement in XSS attack, the attack load is considered to be processed by adopting a character-level recurrent neural network, and a good effect can be obtained.
Further, the method for constructing the training set comprises the following specific steps:
acquiring known attack data, performing data deduplication processing and noise elimination processing on the known attack data, and performing One-Hot coding (One-Hot) on the processed known attack data to obtain negative sample data;
and acquiring known normal data, performing data deduplication processing and noise elimination processing on the known normal data, and performing One-Hot coding (One-Hot) on the processed known normal data to obtain positive sample data.
Further, the known attack data includes: general attack data and characteristic attack data;
the general attack data comprises: SQL injection attacks, XSS attacks, or remote command execution;
the characteristic attack data is attack data aiming at a target framework and target WEB application, and comprises the following steps: attack data aiming at WEB service frameworks and plug-ins such as FCKeditor and ThinkPHP.
Further, the known attack data is generated by an attack data generator;
the attack data generator can specify whether a customized deployment mode is adopted or not when working, if the customized deployment mode is adopted, a WEB site map file or an access log in set time is provided for the attack data generator, and the attack data generator generates possible attack load output aiming at the site according to the provided site map or the access log and the access log.
The attack data generator consists of a general attack load database and a characteristic attack database, wherein the general attack load database stores attack loads common in a large number of web applications, and the characteristic attack database stores the known attack loads of a specific framework or a web component.
When a customized deployment mode is adopted, the work of the attack data generator is divided into two steps:
firstly, an attack data generator reads and analyzes a WEB site map file or a part of access log, extracts access characteristics in the WEB site map file or the part of access log, and comprises the following steps: site fingerprints, site directory structures, url parameters, file uploading points and sensitive directory paths, and storing the information in a temporary database.
Secondly, randomly selecting an attack load from a built-in general attack load database by an attack data generator, and combining the attack load with site information stored in a temporary database to construct attack loads which may appear, wherein the attack loads of the constructions are not necessarily effective, and the attack data generator can also be simply understood as a part of a fuzzy test method in web security, only has offline data collection and fuzzy test attack load construction capabilities, and cannot influence web services;
if a general deployment mode is adopted, the attack data generator intercepts the attack load in the stored characteristic attack data and combines the attack load with the locally stored general attack data to generate attack training data.
Further, the known normal data is generated by a normal data generator;
the normal data generator is used for outputting the same amount of normal service data, access logs in a set time of a site are required to be provided, whether a customized deployment mode is adopted or not needs to be appointed when the normal data generator works, and if the customized deployment mode is adopted, a parameter name and a complete access path appearing in a WEB application access log are reserved for final output by the normal data generator. If the customized deployment mode is not adopted, the normal data generator only reserves the parameter content and part of the access path for output.
The normal data generator consists of a feature extractor, a random data generator and a feature database, and the normal data generator also comprises two steps:
firstly, reading and analyzing a log and a site map file by a feature extractor, extracting site access parameters and corresponding parameter values thereof from the log and the site map file, and storing the information in a feature database by the feature extractor;
secondly, a random data generator reads the characteristic information in the characteristic database, and randomly replaces letters and numbers in original access data in a random replacement mode, wherein the target of letter replacement is letters, for example, the letter 'a' in the access load can be replaced by other letters but cannot be replaced by numbers, conversely, the numbers can only be replaced by numbers but cannot be replaced by letters, and special characters are not operated, so that the length and meaning of part of data with specific meanings are not changed (for example, the telephone number is still a telephone number with legal length after being processed), and finally a large amount of false access data are generated;
further, the noise elimination process removes unrecognized characters, repeated items and incorrectly processed data in a regular matching manner.
Further, in the One-Hot encoding, an adopted encoding table is a self-defined character expansion table and is a matrix with the shape of 1x95, so as to be used for processing WEB access data.
As shown in fig. 1, the forwarder is deployed at the front end of the Web server as a container of the two classifiers, performs load balancing on the back-end Web server in a reverse proxy manner, and is also responsible for loading the rule engine and the neural network model, and when receiving a traffic request from the outside, the forwarder performs http/http resolution on the traffic, performs security evaluation on the traffic through the rule engine and the two classifiers, and finally filters and forwards the traffic. In order to improve the overall performance, a memory pool and a thread pool are adopted to carry out high concurrent processing on the flow.
The log system collects malicious traffic, provides further iterative upgrade for the neural network, and provides security analysis for maintenance personnel.
The blacklist system maintains a dynamic blacklist which is provided for the rule engine to carry out pre-filtering, an attacker ip is recorded in the blacklist, and time limit sealing is carried out on the attacker ip.
As shown in fig. 2, a module structure diagram of the present application, a specific embodiment of the present application includes three parts as described above: a training data generation phase, a classifier training phase and a deployment phase.
As shown in fig. 3, in the classifier training phase, an attack data generator and a normal data generator are used to generate original training data, and then the original training data is formatted into generated training data through a data de-noising and encoder. And ending the training data generation phase, and entering a classifier training phase.
As shown in fig. 5, which is a flow chart of a classifier training phase, data generated in a training data generation phase is sent to a classifier training based on a character-level recurrent neural network, the neural network structure is shown in fig. 4, each parameter during training is adjusted by an evaluation method in the training process, the model accuracy rate is improved, and an optimal weight file is finally output.
As shown in fig. 6, as a processing flow chart of the deployment phase, after obtaining an optimal weight file through a classifier training phase, the deployment phase enters the deployment phase, where the deployment phase includes a repeater, the repeater is deployed at the front end of the Web server as a container of a second classifier, performs load balancing on the back-end Web server in a reverse proxy manner, and is responsible for loading of a rule engine and a neural network model, when receiving a traffic request from the outside, the repeater first parses and formats traffic, sends the traffic to a front black list detection system for detection, after passing the detection, the traffic is forwarded to a layer of a conventional rule filtering intrusion detection system for detection, if passing the detection again, the traffic is processed by the neural network second classifier, after all the three layers of detection pass, the traffic is confirmed to be normal access traffic, and is sent to a log module for recording, and forwarded to the Web server.
The log system collects malicious flow, provides further iterative upgrade for the neural network, and provides security analysis for maintenance personnel. The blacklist system maintains a dynamic blacklist which is provided for the rule engine to carry out pre-filtering, an attacker ip is recorded in the blacklist, and time limit sealing is carried out on the attacker ip.
The application discloses a machine learning intrusion detection method based on a character-level recurrent neural network, which comprises the following steps: and a training data generation stage, which consists of an attack data generator and a normal data generator, is used for carrying out data deduplication and noise elimination on the attack data and the normal data, and carrying out One-Hot coding (One-Hot) on the data for training a character-level recurrent neural network. A classifier training stage: and sending the coded normal and attack data obtained in the training data generation stage into a classifier based on a character-level cyclic neural network, adjusting parameters and iteration times, and dynamically adjusting by the evaluation method to obtain a qualified neural network model. And a deployment stage, namely deploying the neural network weight file obtained in the classifier training stage into a two-classifier realized based on a character-level cyclic neural network, and completing deployment of the intrusion detection method in a WEB proxy mode.
The method obtains inspiration from the field of natural language processing, and a two-classifier is constructed through the character-level recurrent neural network and can be used for intercepting special attack loads generated by special codes, annotation symbols replacing spaces, case and case mixing, malformed labels, double writing and the like by utilizing escape characters. Or the method is used for intercepting unexpected attack behaviors aiming at the text class of the WEB application, namely WEB 0day attack based on text load, and can adapt to the current attack mode and attack means which change rapidly. Compared with the traditional intrusion detection method, the method has extremely low maintenance cost. The method does not need maintenance and upgrading operation for a long time. During maintenance and upgrading, the maintenance can be completed only by updating the weight file of the neural network, and more accurate, simple, efficient and easy-to-use safety protection service is provided for WEB application.
Example two
The embodiment provides a WEB application intrusion detection system based on a character-level neural network;
WEB application intrusion detection system based on character level neural network includes:
a normalization processing module configured to: receiving network flow data; normalizing the network flow data;
a pre-detection module configured to: carrying out blacklist pre-detection on the normalized network traffic data; if the network traffic data is matched with any one of the blacklists, an intrusion warning is output; otherwise, entering a filtering detection module;
a filter detection module configured to: carrying out rule filtering detection on the network traffic data detected by the blacklist; if the rule filtering detection is passed, entering an intrusion detection module; otherwise, outputting an intrusion warning;
an intrusion detection module configured to: inputting the network flow data detected by rule filtering into a trained character-level-based cyclic neural network, and outputting an intrusion detection result by the neural network;
an output module configured to: and recording the intrusion detection result into a log, forwarding the network traffic data with the detection result of safety to a WEB server, intercepting the network traffic data with the detection result of unsafe, and displaying an intrusion warning.
It should be noted here that the normalization processing module, the pre-detection module, the filtering detection module, the intrusion detection module and the output module correspond to steps S101 to S105 in the first embodiment, and the modules are the same as the corresponding steps in the implementation example and application scenarios, but are not limited to the disclosure in the first embodiment. It should be noted that the modules described above as part of a system may be implemented in a computer system such as a set of computer executable instructions.
In the foregoing embodiments, the descriptions of the embodiments have different emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The proposed system can be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the above-described modules is merely a logical functional division, and in actual implementation, there may be another division, for example, a plurality of modules may be combined or may be integrated into another system, or some features may be omitted, or not executed.
EXAMPLE III
The present embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, a processor is connected with the memory, the one or more computer programs are stored in the memory, and when the electronic device runs, the processor executes the one or more computer programs stored in the memory, so as to make the electronic device execute the method according to the first embodiment.
It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general purpose processors, digital signal processors DSP, application specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may include both read-only memory and random access memory, and may provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.
In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software.
The method in the first embodiment may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor. The software modules may be located in ram, flash, rom, prom, or eprom, registers, etc. as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor. To avoid repetition, it is not described in detail here.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
Example four
The present embodiments also provide a computer-readable storage medium for storing computer instructions, which when executed by a processor, perform the method of the first embodiment.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims (9)

1. A WEB application intrusion detection method based on a character-level neural network is characterized by comprising the following steps:
receiving network flow data; normalizing the network flow data;
carrying out blacklist pre-detection on the normalized network traffic data; if the network traffic data is matched with any one of the blacklists, an intrusion warning is output; otherwise, entering the next step;
carrying out rule filtering detection on the network traffic data detected by the blacklist; if the rule filtering detection is passed, entering the next step; otherwise, outputting an intrusion warning; carrying out rule filtering detection on the network traffic data detected by the blacklist; the method specifically comprises the following steps:
firstly, matching a blacklist generated by a system, and directly judging as an attack traffic when a source ip address of network traffic data is found to be in the blacklist generated by the system;
secondly, matching with a blacklist defined by a user, wherein a strict matching mode is adopted for a source address, a source port, a destination address, a destination port and a network protocol characteristic item; matching the load keyword characteristic items in a regular mode; if the matching is successful, judging the flow rate to be attack flow rate;
inputting the network flow data detected by rule filtering into a trained character-level-based cyclic neural network, and outputting an intrusion detection result by the neural network;
recording the intrusion detection result into a log, forwarding the network traffic data with the detection result of safety to a WEB server, intercepting the network traffic data with the detection result of unsafe, and displaying an intrusion warning;
the method comprises the steps of performing customized deployment aiming at WEB application site requirements, performing customized generation on WEB application sites through an attack data generator and a normal data generator to generate a training set, and performing character-level cyclic neural network training to obtain a customized neural network weight file.
2. The method for detecting the intrusion of the WEB application based on the character-level neural network as claimed in claim 1, wherein the network traffic data is normalized; the method specifically comprises the following steps:
unpacking the network traffic data into readable data, storing the readable data in a json format in a key-value pair mode, wherein the formatted data describing the single network traffic comprises the following steps: source address, destination address, source port number, destination port number, network protocol employed, original payload, encoded payload, and flag bits.
3. The method for detecting the intrusion of the WEB application based on the character-level neural network as claimed in claim 2, wherein the load is processed by a one-hot coding mode and is stored in the coded load; after the processing, the normalized network traffic data includes eight parts, namely the source address, the destination address, the source port number, the destination port number, the adopted network protocol, the original load, the coding load and the flag bit, and is sent to a blacklist for pre-detection.
4. The method according to claim 1, wherein the normalized network traffic data is pre-detected by a blacklist; if the network traffic data is matched with any item in the blacklist, an intrusion warning is output; otherwise, entering the next step; the blacklist comprises two parts, wherein one part is a blacklist built in the detection system, is generated by the system and is stored in a source ip address list mode; the other part is a user-defined blacklist which is generated by a user and can store characteristic items describing network traffic data packets: source address, source port, destination address, destination port, network protocol, and payload keywords.
5. The method for detecting the intrusion on the WEB application based on the character-level neural network as claimed in claim 1, wherein the network traffic data detected by the rule filtering is input into the trained character-level-based recurrent neural network, and the neural network outputs the intrusion detection result; the specific training steps of the trained character-level-based recurrent neural network comprise:
constructing a training set; the training set includes: positive sample data and negative sample data;
inputting the training set into a cyclic neural network based on character level, training the network, and adjusting network parameters;
after each iteration is set for times, the loss function value of the network and the set threshold value are judged;
when the loss function value of the neural network is larger than a set threshold value, continuing iterative training;
when the loss function value of the neural network is smaller than a set threshold value, stopping training and outputting neural network parameters;
and deploying the neural network parameters to the character-level-based cyclic neural network, testing the neural network, obtaining the trained character-level-based cyclic neural network if the test is passed, and continuing training if the test is not passed.
6. The method according to claim 5, wherein the step of constructing the training set comprises the following steps:
obtaining known attack data, carrying out data deduplication processing and noise elimination processing on the known attack data, and carrying out one-hot coding on the processed known attack data to obtain negative sample data;
obtaining known normal data, carrying out data deduplication processing and noise elimination processing on the known normal data, and carrying out single hot coding on the processed known normal data to obtain positive sample data;
or,
the known attack data comprises: general attack data and characteristic attack data;
the general attack data comprises: SQL injection attacks, XSS attacks, or remote command execution;
the characteristic attack data is attack data aiming at a target framework and target WEB application, and comprises the following steps: attack data for WEB services frameworks and plug-ins;
or,
the known attack data is generated by an attack data generator;
the attack data generator can specify whether a customized deployment mode is adopted or not when working, if the customized deployment mode is adopted, a WEB site map file or an access log in set time is provided for the attack data generator, and the attack data generator generates possible attack load output aiming at the site according to the provided site map or the access log and the access log;
the attack data generator consists of a general attack load database and a characteristic attack database, the general attack load database stores a large amount of attack loads common in web application, and the characteristic attack database stores the known attack loads of a specific framework or a web component;
or,
when a customized deployment mode is adopted, the work of the attack data generator is divided into two steps:
firstly, an attack data generator reads and analyzes a WEB site map file or a part of access log, extracts access characteristics in the WEB site map file or the part of access log, and comprises the following steps: site fingerprints, site directory structures, url parameters, file uploading points and sensitive directory paths, and storing the information into a temporary database;
secondly, randomly selecting an attack load from a built-in general attack load database by an attack data generator, and splicing the attack load with site information stored in a temporary database to construct attack loads which may appear;
if a universal deployment mode is adopted, the attack data generator intercepts the attack load in the stored characteristic attack data and combines the attack load with the locally stored universal attack data to generate attack training data;
or,
the known normal data is generated by a normal data generator;
the normal data generator is used for outputting the same amount of normal service data, an access log in a set time of a site is required to be provided, whether a customized deployment mode is adopted or not is required to be appointed when the normal data generator works, and if the customized deployment mode is adopted, a parameter name and a complete access path appearing in the access log of the WEB application are reserved for final output by the normal data generator; if a customized deployment mode is not adopted, only the parameter content and part of access paths are reserved for output by the normal data generator;
or,
the normal data generator consists of a feature extractor, a random data generator and a feature database, and the normal data generator also comprises two steps:
firstly, reading and analyzing a log and a site map file by a feature extractor, extracting site access parameters and corresponding parameter values thereof from the log and the site map file, and storing the information in a feature database by the feature extractor;
secondly, the random data generator reads the feature information in the feature database, and randomly replaces letters and numbers in the original access data in a random replacement mode, wherein the target of letter replacement is also letters, for example, the letter 'a' in the access load can be replaced by other letters but cannot be replaced by numbers, conversely, the numbers can only be replaced by numbers but cannot be replaced by letters, and no operation is performed on special characters, so that the length and meaning of part of data with specific meanings are ensured to be unchanged, and a large amount of false access data are finally generated;
or,
the noise elimination processing is to remove unrecognizable characters, repeated items and wrongly processed data in a regular matching mode;
or,
the One-Hot coding adopts a coding table which is a self-defined character expansion table and a matrix with the shape of 1x95 so as to be used for processing WEB access data.
7. A WEB application intrusion detection system based on a character-level neural network is characterized by comprising:
a normalization processing module configured to: receiving network traffic data; normalizing the network flow data;
a pre-detection module configured to: carrying out blacklist pre-detection on the normalized network traffic data; if the network traffic data is matched with any one of the blacklists, an intrusion warning is output; otherwise, entering a filtering detection module;
a filter detection module configured to: carrying out rule filtering detection on the network traffic data detected by the blacklist; if the rule filtering detection is passed, entering an intrusion detection module; otherwise, outputting an intrusion warning; carrying out rule filtering detection on the network traffic data detected by the blacklist; the method specifically comprises the following steps:
firstly, matching a blacklist generated by a system, and directly judging attack traffic when a source ip address of network traffic data is found to be in the blacklist generated by the system;
secondly, matching with a blacklist defined by a user, wherein a strict matching mode is adopted for a source address, a source port, a destination address, a destination port and a network protocol feature item; matching the load keyword characteristic items in a regular mode; if the matching is successful, judging the flow rate to be attack flow rate;
an intrusion detection module configured to: inputting the network flow data which is subjected to rule filtering detection into a trained character-level-based cyclic neural network, and outputting an intrusion detection result by the neural network;
an output module configured to: recording the intrusion detection result into a log, forwarding the network traffic data with the detection result of safety to a WEB server, intercepting the network traffic data with the detection result of unsafe, and displaying an intrusion warning;
the method comprises the steps of performing customized deployment aiming at WEB application site requirements, performing customized generation on WEB application sites through an attack data generator and a normal data generator to generate a training set, and performing character-level cyclic neural network training to obtain a customized neural network weight file.
8. An electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein a processor is connected to the memory, the one or more computer programs being stored in the memory, the processor executing the one or more computer programs stored in the memory when the electronic device is running, to cause the electronic device to perform the method of any of the preceding claims 1-6.
9. A computer-readable storage medium storing computer instructions which, when executed by a processor, perform the method of any one of claims 1 to 6.
CN202110096602.1A 2021-01-25 2021-01-25 WEB application intrusion detection method and system based on character-level neural network Active CN112887304B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110096602.1A CN112887304B (en) 2021-01-25 2021-01-25 WEB application intrusion detection method and system based on character-level neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110096602.1A CN112887304B (en) 2021-01-25 2021-01-25 WEB application intrusion detection method and system based on character-level neural network

Publications (2)

Publication Number Publication Date
CN112887304A CN112887304A (en) 2021-06-01
CN112887304B true CN112887304B (en) 2022-12-30

Family

ID=76050947

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110096602.1A Active CN112887304B (en) 2021-01-25 2021-01-25 WEB application intrusion detection method and system based on character-level neural network

Country Status (1)

Country Link
CN (1) CN112887304B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113904862A (en) * 2021-10-22 2022-01-07 中车株洲电力机车有限公司 Distributed train control network intrusion detection method, system and storage medium
CN114449523B (en) * 2022-04-07 2022-06-24 北京航天驭星科技有限公司 Flow filtering method, device, equipment and medium for satellite measurement and control system
CN114826753B (en) * 2022-04-28 2024-05-03 武汉思普崚技术有限公司 Full-flow intrusion detection method, device, equipment and medium based on rule characteristics
CN115473672B (en) * 2022-08-03 2024-03-29 广西电网有限责任公司电力科学研究院 Leak-proof detection method based on online interactive WEB dynamic defense
CN117278322B (en) * 2023-11-13 2024-02-20 国家工业信息安全发展研究中心 Web intrusion detection method, device, terminal equipment and storage medium

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106656981A (en) * 2016-10-21 2017-05-10 东软集团股份有限公司 Network intrusion detection method and device
KR20180070247A (en) * 2016-12-16 2018-06-26 주식회사 페타바이코리아 An efficient method and device for generating network intrusion detection rules
CN109308494A (en) * 2018-09-27 2019-02-05 厦门服云信息科技有限公司 LSTM Recognition with Recurrent Neural Network model and network attack identification method based on this model
CN109992782A (en) * 2019-04-02 2019-07-09 深圳市华云中盛科技有限公司 Legal documents name entity recognition method, device and computer equipment
CN110213287A (en) * 2019-06-12 2019-09-06 北京理工大学 A kind of double mode invasion detecting device based on ensemble machine learning algorithm
CN110830435A (en) * 2019-08-27 2020-02-21 国家电网有限公司信息通信分公司 Method and device for extracting network flow space-time characteristics and detecting abnormity
CN111371806A (en) * 2020-03-18 2020-07-03 北京邮电大学 Web attack detection method and device
CN111586071A (en) * 2020-05-19 2020-08-25 上海飞旗网络技术股份有限公司 Encryption attack detection method and device based on recurrent neural network model
CN112202722A (en) * 2020-09-08 2021-01-08 华东师范大学 Intrusion detection method

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106656981A (en) * 2016-10-21 2017-05-10 东软集团股份有限公司 Network intrusion detection method and device
KR20180070247A (en) * 2016-12-16 2018-06-26 주식회사 페타바이코리아 An efficient method and device for generating network intrusion detection rules
CN109308494A (en) * 2018-09-27 2019-02-05 厦门服云信息科技有限公司 LSTM Recognition with Recurrent Neural Network model and network attack identification method based on this model
CN109992782A (en) * 2019-04-02 2019-07-09 深圳市华云中盛科技有限公司 Legal documents name entity recognition method, device and computer equipment
CN110213287A (en) * 2019-06-12 2019-09-06 北京理工大学 A kind of double mode invasion detecting device based on ensemble machine learning algorithm
CN110830435A (en) * 2019-08-27 2020-02-21 国家电网有限公司信息通信分公司 Method and device for extracting network flow space-time characteristics and detecting abnormity
CN111371806A (en) * 2020-03-18 2020-07-03 北京邮电大学 Web attack detection method and device
CN111586071A (en) * 2020-05-19 2020-08-25 上海飞旗网络技术股份有限公司 Encryption attack detection method and device based on recurrent neural network model
CN112202722A (en) * 2020-09-08 2021-01-08 华东师范大学 Intrusion detection method

Also Published As

Publication number Publication date
CN112887304A (en) 2021-06-01

Similar Documents

Publication Publication Date Title
CN112887304B (en) WEB application intrusion detection method and system based on character-level neural network
US11899786B2 (en) Detecting security-violation-associated event data
Gupta et al. Enhancing the browser-side context-aware sanitization of suspicious HTML5 code for halting the DOM-based XSS vulnerabilities in cloud
Shibahara et al. Efficient dynamic malware analysis based on network behavior using deep learning
US20190132334A1 (en) System and method for analyzing binary code for malware classification using artificial neural network techniques
US8220048B2 (en) Network intrusion detector with combined protocol analyses, normalization and matching
US9392004B2 (en) Method and system for dynamic protocol decoding and analysis
CN108183916B (en) Network attack detection method and device based on log analysis
US7596809B2 (en) System security approaches using multiple processing units
CN112468520B (en) Data detection method, device and equipment and readable storage medium
US20040205411A1 (en) Method of detecting malicious scripts using code insertion technique
US11212297B2 (en) Access classification device, access classification method, and recording medium
EA037617B1 (en) Method and system for detecting an intrusion in data traffic on a data communication network
CN111049819A (en) Threat information discovery method based on threat modeling and computer equipment
Dong et al. Towards interpreting recurrent neural networks through probabilistic abstraction
CN112685738B (en) Malicious confusion script static detection method based on multi-stage voting mechanism
WO2019190403A1 (en) An industrial control system firewall module
US9979697B2 (en) Packet filtering apparatus and packet filtering method
CN114024761B (en) Network threat data detection method and device, storage medium and electronic equipment
CN111770097B (en) Content lock firewall method and system based on white list
CN110647749A (en) Second-order SQL injection attack defense method
Hubballi et al. XSSmitigate: Deep packet inspection based XSS attack quarantine in software defined networks
JP6984760B2 (en) Converter and conversion program
Antunes et al. Automatically complementing protocol specifications from network traces
KR101893029B1 (en) Method and Apparatus for Classifying Vulnerability Information Based on Machine Learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
EE01 Entry into force of recordation of patent licensing contract

Application publication date: 20210601

Assignee: Shandong Manfen Information Technology Co.,Ltd.

Assignor: SHANDONG COMPUTER SCIENCE CENTER(NATIONAL SUPERCOMPUTER CENTER IN JINAN)

Contract record no.: X2024980001095

Denomination of invention: WEB Application Intrusion Detection Method and System Based on Character Level Neural Network

Granted publication date: 20221230

License type: Common License

Record date: 20240119

EE01 Entry into force of recordation of patent licensing contract