CN114417345A - Web attack detection method based on NLP - Google Patents

Abstract

The invention discloses a Web attack detection method based on NLP, which comprises the steps of adopting a scanning command to collect basic information of a target system, simulating human behavior, recording each click interaction of HTML codes of the current whole page, transmitting HTML requests to an AI module for classification and identification, classifying through a fastText rapid text classification algorithm, carrying out batch labeling through rules, carrying out numerical processing on texts, realizing text vectorization, carrying out word segmentation cleaning and vectorization on a part of special characters, modeling results, obtaining classification output of one-hot codes through a deep learning model, mapping the one-hot code labels to corresponding two-classification labels, and carrying out POC plug-in selection and verification through a verification module according to contained information; an automatic penetration testing system designed by an NLP artificial intelligence algorithm replaces the traditional manpower to carry out attack inspection, and helps to solve the problems of insufficient personnel and uneven operation level.

Description

Web attack detection method based on NLP

Technical Field

The invention belongs to the technical field of wide area networks, and particularly relates to a Web attack detection method based on NLP.

Background

Web attack detection is a basic function in penetration testing. The penetration test refers to a process that a penetration test engineer simulates information detection, vulnerability assessment and attack means used by a hacker, deeply detects the security of a target and finds a system fragile surface. Penetration tests are currently not well defined standards, and their theory and technology is in constant development. The penetration test requires that testers have comprehensive capability, most of used penetration test tools are auxiliary tools, and no very mature comprehensive penetration platform exists. The business scale based on the internet is continuously enlarged and is more and more complex, the cost of the manual penetration test is more and more high, and the difficulty degree is obviously improved. The penetration testing tool converts part of heavy, repetitive work, vulnerability scanning and other basic work in the work into software or equipment, so that the continuous, automatic and intelligent penetration testing requirement is increasingly urgent, but due to the huge factors of a person in the penetration process, an automatic penetration platform cannot break through in the past, but with the rapid development of artificial intelligence, the automatic penetration testing becomes possible gradually.

At present, the number of domestic vulnerability scanning tools is large, the functions are concentrated on vulnerability scanning of a target, work of whether vulnerabilities are available is still carried out by analyzing and comparing scanning results manually by professional security personnel, workload is heavy, and vulnerability analysis and false alarm verification are carried out with large labor input. The automatic penetration testing tool in the current market is an implementation mode based on the combination of a semi-automatic scanning tool and a POC tool, and cannot well meet the requirement of automatic penetration testing. The foreign network security industry is developed earlier, a large number of mature automatic penetration testing tools such as Canva, MetaSpirot, Core-Impact and the like also belong to semi-automation, and as a penetration testing tool with an auxiliary effect, the penetration testing tool can only play a role in a certain direction generally, and the relevance and logic vulnerability consideration of the whole target system are lacked.

Disclosure of Invention

The present invention aims to provide a method for detecting a Web attack based on NLP to solve the problems set forth in the background art.

In order to achieve the purpose, the invention adopts the following technical scheme: a Web attack detection method based on NLP comprises the following steps:

A. adopting a scanning command to carry out basic information collection on a target system, transmitting scanning result information to an AI module to carry out vulnerability judgment decision, judging whether the Web service is available or not by sending a request according to a scanning result such as a scanned port, and sending an instruction to a crawler module to carry out page content grabbing and request capturing after the Web service is judged;

B. simulating mouse and click recording behaviors of a human browsing page by using a web crawler technology, recording each click interaction of an HTML code of the current whole page, and transmitting an HTML request to an AI module for classification and identification;

the AI module receives HTTP request data transmitted by a crawler, carries out data processing based on NLP, carries out classification through a fastText rapid text classification algorithm, carries out batch labeling through rules, carries out numerical processing on the text, realizes text vectorization, carries out word segmentation cleaning and vectorization on a part of special characters, and then carries out modeling learning on the data to start deep learning;

D. the obtained information is output according to the classification of the one-hot codes through a deep learning model, and the corresponding one-hot coded labels are mapped to the corresponding two classification labels;

E. for the data transmitted by the AI module, the loophole check module can select the POC plug-in according to the contained information, the AI module can output the corresponding POC file name, dynamically execute the corresponding POC file code through the import lib library of python, after the execution is successful, the loophole is confirmed, the module can send the operation result of the POC code to the signal module, and the signal module can confirm the existence of the loophole and store the information into the corresponding database. Further, after the above steps, the AI model processing results are stored, and implemented using the signal processing mechanism of Django, interacting with the database through signals, and the signal system allows one or more senders to send notifications or signals to a group of receivers. The signal accepts two parameters, task ID and result content. The signal receives the feedback that the POC code detection is finished, and once the signal is received, the logic code corresponding to the signal is executed. Further, before the step a, the user inputs the detection target information at the client, the user input is sent to the server through an HTTP request, and the server stores the scanning information in the database after checking the data and sends the scanning information to the scanning engine through the message queue to complete scanning.

Further, in step B, specifically, the crawler module will take all visible (a) tags of the target after the target is requested for the first time, analyzing the href attribute, archiving the href attribute in a queue to wait for next crawling, searching the form of the current page after the extraction is finished, the form is simulated and submitted, the required type and content in the form are automatically identified, the simulated submission action is carried out after the form is filled, the corresponding form is submitted to a remote server, the corresponding HTTP request is captured and sent to an AI module, after all the forms are operated, all the buttons in the page are identified and clicked, the HTTP request triggered by the clicked button is captured and sent to the AI module after the clicked button is clicked, the second request will take the url previously parsed from the queue and again make the form submission and button click until the data in the queue is empty.

Further, in the step C, batch labeling is performed, and the labeling rule is as follows: the request contains an ID and is a character labeled 1; there are parameter values and there are Chinese characters labeled 1; the shorter or random and numbered parameters are labeled 0.

Furthermore, during the numerical processing, a series text operation is carried out by using a Keras Tokenizer class, including establishing a bag-of-words model TF-IDF, and the analysis of the request parameters of the hacking can find that the dimension and the semantic are not factors which closely influence the hacking success, and the text vectorization can also be carried out by using word2 vec.

Furthermore, modeling processing data including three models, namely an ANN model, a TEXTCNN model and a DQN model, and various models are adopted to enhance the detection accuracy of the attack detection behavior.

Further, in the ANN model, positive integers are converted into dense vectors with fixed sizes through word Embedding layer Embedding, the length of the dense vectors with the whole fixed sizes is shortened through global pooling, the average value is obtained, the dimensionality of a next layer of network is reduced, and the deviation phenomenon of the estimated average value caused by parameter errors is restrained to be output through modeling through the ANN model.

Further, the same operation as the ANN model was employed in the TEXTCNN model.

Furthermore, in the DQN model, vectorization is carried out on state information obtained by interacting with the environment each time, an action is obtained according to the state information, and the action and the state are transmitted to the neural network to obtain a Q value.

Compared with the prior art, the invention has the beneficial effects that:

the Web attack automatic detection is realized through an artificial intelligence algorithm, the Web attack automatic detection method can be used for an automatic penetration testing system, replaces the traditional manpower to carry out Web safety testing, helps to solve the problems of insufficient safety technicians and uneven operation level, and simultaneously reduces the data pollution of the traditional detection method to the target system test. The method and the device have the advantages that the searching of the application vulnerabilities such as SQL injection points is focused, the complete penetration test platform is realized, the accuracy of Web attack detection depends on multiple factors, the completeness of crawler data can be guaranteed by comprehensively and deeply searching the target site, and the completeness of the crawler data influences the quality of modeling data. The accuracy of AI module classification and filtering, as well as the stringency of POC codes, are also critical to the accuracy of attack detection. In the research, the page request parameters and the page subjects are classified through the NLP algorithm, and the detection result output is completed through modeling, so that the problems that the scanning time is too long and the attack test of invalid parameters is more in the traditional detection method are solved, and meanwhile, the output of flow and the possibility of network storm are reduced. The accuracy of the whole model is whether the training set is accurate or not and the magnitude of the order of magnitude, and subsequently, automatic modeling is carried out on the collection and labeling of data by the aid of the focus so as to form a high-quality training set by fast iteration requirements, and accordingly more accurate influence is generated on output results.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.

FIG. 1 is a system framework diagram of the present invention;

FIG. 2 is a crawler work flow diagram of the present invention;

FIG. 3 is a Loss graph of an embodiment of the present invention;

FIG. 4 is a graph of an Epochs plot according to an embodiment of the present invention;

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.

Referring to fig. 1 to 4, the present invention provides a technical solution: a Web attack detection method based on NLP comprises the following steps:

s1, the user inputs target information at the client, the user input is sent to the server through HTTP request, the server stores the scanning information into the database after checking the data, and sends the scanning information to the scanning engine through the message queue to complete scanning;

s2, using the NMAP to collect basic information of a target system, wherein the basic information comprises all open ports, port services, an operating system and version, MAC addresses and open Web service ports, outputting scanning result information into an XML format to be transmitted into an AI module to carry out vulnerability judgment decision, judging whether the scanned port is the Web service or not by sending a request, and sending the Web service to a crawler module to carry out page content capture and request capture after judging that the Web service is the Web service;

s3, the crawler module uses Webkit technology, imitates the behavior of browsing page, records each click interaction of current whole page HTML code, and transmits HTML request to AI module for classification and identification, when the Web service is judged, the HTML request is transmitted to the crawler module for page content capture and request capture, concretely, the crawler module takes all visible < a > tags of the target after first request of the target, analyzes href attribute in the visible < a > tags and stores the href into a queue for next crawl, searches the form of the current page after extraction, automatically identifies the type and content needed in the form by analog submission of the form, submits the corresponding form to a remote server by analog submission action after filling the form, captures corresponding HTTP request and transmits the HTTP request to AI module, identifies and clicks all buttons in the page after operating all forms, after the button is clicked, capturing an HTTP request triggered by the clicked button and sending the HTTP request to an AI module, taking the url analyzed before from the queue for the second time, and submitting the form and clicking the button again until the data in the queue is empty;

s4, receiving HTTP request data transmitted by a crawler, performing NLP-based processing, classifying through a fastText rapid text classification algorithm, labeling in batches through rules, performing numerical processing on a text to realize text vectorization, modeling data obtained after word segmentation cleaning and vectorization are performed on a part of special characters to complete deep learning, performing serialized text operation by using a Keras-like Tokenizer during numerical processing, including establishing a word bag model TF-IDF, and performing text vectorization by using word2 vec;

and (4) classification processing, namely, adopting a fastText fast text classification algorithm, not needing pre-trained word vectors, and accelerating training speed and testing speed under the condition of keeping high precision. For hacker attack, a general pure static website or system cannot be invaded, and a characteristic of a dynamic website is that a database interaction behavior exists in the access process, so that classification is carried out according to whether the database interaction behavior exists or not, two classifications are adopted, wherein the database interaction behavior possibly exists and is 1, and the database interaction behavior does not exist and is 0;

and marking labels, and carrying out batch labeling on the classified data through rules. The rules here include that the request contains an ID and is a character, denoted 1; there are parameter values and there are Chinese characters labeled 1; the shorter or random parameter with numbers is marked as 0, and the marking can be carried out manually, so that the access and the quality of data output can be better controlled;

modeling and learning the quantified data, and establishing three models, namely an ANN model, a TEXTCNN model and a DQN model, by the system; in the ANN model, positive integers are first converted into dense vectors of fixed size by the word Embedding layer Embedding, for example: [ [4], [20] ] - > [ [0.25, 0.1], [0.6, -0.2] ], then through GlobalatagePooling 1D, the overall pooling shortens the length of the whole dense vector with fixed size, averages, reduces the dimensionality of the next layer of network, inhibits the deviation phenomenon of the estimated mean value caused by parameter errors, carries out modeling output through ANN, adopts the same operation as the ANN model in the TEXTCNN model, carries out vectorization on state information obtained by interaction with the environment each time in the DQN model, obtains an action according to the state information, and then transmits the action and the state to the neural network to obtain a Q value;

s5, obtaining a classification output of one-hot codes through the deep learning model, and mapping the corresponding one-hot coded labels to the corresponding two classification labels, for example: [ [0, 1] ] represents that there may be database interaction behavior; [ [1, 0] ] represents the absence of behavior interacting with the database; in the actual test, the common attack types in vulnerability detection are used for prediction, including SQL injection, file inclusion, cross-site scripting and the like, and the method comprises the following steps:

[ 'SQL _ INJECT' ', FILE _ INCLUDE' ', SQL _ INJECT, XSS _ INJECT' ', PHPINFO' ], constructing a matrix by one-hot encoding.

S6, the loophole check module selects the POC plug-in according to the contained information, the AI module outputs the corresponding POC file name, dynamically executes the corresponding POC file code through the import lib library of python, confirms that the POC file code contains the loophole after the execution is successful, and the module sends the operation result of the POC code to the signal module.

S7, a signal module for storing the result, the signal module interacting with the database by a signal, the signal system allowing one or more senders to send notifications or signals to a group of recipients.

The data set is divided into three parts, a training dataset (training dataset), a validation dataset (validation dataset) and a test dataset (test dataset). Then, when the model is trained, whether the model is normally trained is diagnosed according to the Loss of the training set and the Loss of the verification set, so that algorithm parameters can be optimized through a visual Loss curve, and a better model, namely a network model with strong generalization ability (generation), is finally trained. In this figure we can see that the Loss curves begin to converge after approximately 3 epochs, resulting in a good-fit model.

And the accuracy curve is in a scalar form whether the true value is consistent with the model predicted value, and the performance of the algorithm at the current stage is displayed by outputting the accuracy through each test on the verification set. The performance grade climbing degree of the algorithm is described through the accuracy, whether the neural network advances in the correct direction for learning can be observed, and training can be conveniently stopped when the neural network is suitable. In this figure, by observing the convergence curves of the objective function values and the accuracy of the validation set, the algorithm obtains an algorithm with sufficient performance after 3 epochs passes.

The key of automation of Web attack detection is vulnerability discovery and verification, and the existing discovery tools are all based on characteristics such as scanning feedback, and the like, so that the target attention is easily attracted, and meanwhile, the influence of polluted data and network traffic is also generated. The complete penetration test platform containing the Web attack detection is realized, whether the used crawler data is completely determined by whether the target site is completely retrieved or not is judged, and the accuracy of the system vulnerability detection is the accuracy of AI module classification and filtration and the rigor of POC codes. The number of the detection platform vulnerability verifications depends on the number of POC codes, and new vulnerability attack detection can be supported by adding new POC codes. The request parameters and the page subjects are classified through the NLP algorithm, so that the problems of too long scanning time and more attack tests of invalid parameters are solved, the output of flow and the possibility of network storms are reduced, and attack detection is realized in a mode with the minimum influence on a target system.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention should be equivalent or changed within the scope of the present invention.

Claims (10)

1. A Web attack detection method based on NLP is characterized by comprising the following steps:

A. adopting a scanning command to carry out basic information collection on a target system, judging whether the target system is Web service or not by sending a request according to a scanning result such as a scanned port, and sending an instruction to a crawler module to carry out page content grabbing and request capturing after judging that the target system is the Web service;

B. simulating mouse click record behaviors of a human browsed page by using a web crawler technology, recording each click interaction of an HTML code of the current whole page, and transmitting an HTML request to an AI module for classification and identification;

an AI module receives HTTP request data transmitted by a crawler, carries out data processing based on NLP, carries out classification through a fastText fast text classification algorithm, carries out batch labeling through rules, carries out numerical processing on a text, realizes text vectorization, carries out word segmentation cleaning and vectorization on a part of special characters, and then carries out modeling on the data to start deep learning;

D. classifying and outputting the obtained information according to one-hot coding through a deep learning model, and mapping the corresponding one-hot coding label to the corresponding two classification labels;

E. and for the data transmitted by the AI module, the vulnerability checking module selects the POC plug-in according to the contained information, dynamically executes the corresponding POC file code through an importplib library of python, confirms that the POC file code contains the vulnerability after the execution is successful, the module sends the operation result of the POC code to the signal module, and the signal module confirms that the vulnerability attack exists and stores the information into the corresponding database.

2. The NLP-based Web attack detection method according to claim 1, wherein: after the above steps, the results of the detection and verification are stored and interacted with the database by a signal, which allows one or more senders to send notifications or signals to a group of recipients.

3. The NLP-based Web attack detection method according to claim 1, wherein: before the step A, a user inputs target information at a client, the input of the user is sent to a server through an HTTP request, the server stores the information into a database after checking data, and the information is sent to a scanning engine through a message queue to complete scanning.

4. The NLP-based Web attack detection method according to claim 1, wherein: in step B, specifically, the crawler module will get all visible (a) tags of the target after requesting the target for the first time, analyzing the href attribute, storing in a queue, waiting for the next crawling, searching the form of the current page after the extraction is finished, the type and content required in the form can be automatically identified by performing simulated submission on the form, the corresponding form is submitted to a remote server by performing a simulated submission action after the form is filled, a corresponding HTTP request is captured and sent to an AI module, after all the forms are operated, all the buttons in the page are identified and clicked, the HTTP request triggered by the clicked button is captured and sent to the AI module after the clicked button is clicked, the second request will take the url previously parsed from the queue and again make the form submission and button click until the data in the queue is empty.

5. The NLP-based Web attack detection method according to claim 1, wherein: in the step C, the rule for batch marking is that the request contains ID and the mark of the character is 1; there are parameter values and there are Chinese characters labeled 1; the shorter or random and numbered parameters are labeled 0.

6. The NLP-based Web attack detection method according to claim 1, wherein: during the numerical processing, a Tokenizer-like Keras is used for carrying out serialized text operation, including establishing a bag-of-words model TF-IDF, and when the request parameter analysis of the hacking attack can discover that the dimension and the semantic are not factors which closely influence the success of the attack, word2vec can be used for carrying out text vectorization.

7. The NLP-based Web attack detection method according to claim 1, wherein: the modeling comprises three models, namely an ANN model, a TEXTCNN model and a DQN model.

8. The NLP-based Web attack detection method according to claim 7, wherein: in the ANN model, positive integers are converted into dense vectors with fixed sizes through word Embedding layer Embedding, the length of the dense vectors with the whole fixed sizes is shortened through global pooling, the average value is obtained, the dimensionality of the next layer of network is reduced, and the deviation phenomenon of the estimated average value caused by parameter errors is restrained to be output through modeling through the ANN model.

9. The NLP-based Web attack detection method according to claim 8, wherein: the same operation as the ANN model was employed in the TEXTCNN model.

10. The NLP-based Web attack detection method according to claim 7, wherein: in the DQN model, vectorization is carried out on state information obtained by interacting with the environment each time, an action is obtained according to the state information, and then the action and the state are transmitted to a neural network to obtain a Q value.

Images