CN115865472A - Request intercepting method and system based on log analysis - Google Patents

Request intercepting method and system based on log analysis Download PDF

Info

Publication number
CN115865472A
CN115865472A CN202211508936.6A CN202211508936A CN115865472A CN 115865472 A CN115865472 A CN 115865472A CN 202211508936 A CN202211508936 A CN 202211508936A CN 115865472 A CN115865472 A CN 115865472A
Authority
CN
China
Prior art keywords
log
request
log data
data
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211508936.6A
Other languages
Chinese (zh)
Inventor
刘博�
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
CCB Finetech Co Ltd
Original Assignee
China Construction Bank Corp
CCB Finetech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp, CCB Finetech Co Ltd filed Critical China Construction Bank Corp
Priority to CN202211508936.6A priority Critical patent/CN115865472A/en
Publication of CN115865472A publication Critical patent/CN115865472A/en
Pending legal-status Critical Current

Links

Images

Abstract

The invention relates to a request intercepting method and a system based on log analysis, which comprises the steps of collecting log data related to a user request through a log recovery component; the log recovery component performs distributed off-line storage and distribution and real-time distribution on the log data; carrying out classification processing on distributed offline storage and distribution log data by a log analyzer, and automatically learning by a machine learning algorithm to generate a discrimination model; transmitting the discrimination model to a request interceptor of the server; the method comprises the steps that a request interceptor obtains log data distributed in real time, the request interceptor is mounted on a server through a sidecar mode, and the request interceptor determines attributes of user requests according to a discrimination model; and judging whether to intercept the user request according to the attribute of the user request. The traditional centralized firewall intercepting mode is modified, the risk brought to the system by single-point faults is reduced, and meanwhile, the traditional intercepting rule adding mode can be effectively supplemented by adding the intercepting rules in a self-learning mode.

Description

Request intercepting method and system based on log analysis
Technical Field
The invention relates to the technical field of computers, in particular to a request intercepting method and system based on log analysis.
Background
With the gradual deepening of informatization and the rapid development of shared and open internet, the network security threat presents the characteristics of increasing diversification and unknown property, and computer systems increasingly contain contents which cannot be revealed by client privacy, client security and the like and can cause major business problems once the contents are revealed. Therefore, importance is attached to the security of the application system.
The traditional information security means is generally based on the filtering of request texts, the filtering is carried out in advance by configuring rules in the protection means such as a firewall and the like in advance and adding a related characteristic regular expression of a suspected dangerous request, so that the natural hysteresis is caused, the attack of vulnerabilities such as 0day and the like is met, only the analysis after the incident is needed, and the attack is usually ended at this time. In the face of targets with high attack value, hackers tend to adopt centralized and short-time explosive attacks, and the adoption of the protection mode can cause the target system to be easily exposed to security risks.
In the current bank safety protection situation, protection is generally based on a protection wall, the technology is more perfect due to long-time development of the protection mode, but the protection mode is also based on a passive protection concept, namely based on a determined rule, and then related request verification and interception are carried out.
Under a new technical environment, a third-generation network security system taking technologies such as machine learning, artificial intelligence, big data security analysis and the like as a core is urgently needed to be established in the industry because the traditional network security defense means for carrying out rule matching and feature code technology on logs cannot cope with novel network threats, most of the existing log analysis methods are based on field knowledge and depend on manual inspection or rule compiling modes to carry out feature identification and rule establishment manually, and the existing log analysis methods gradually evolve from independent, simple, direct and easy exposure into attacks such as organized, targeted and long-lasting APT and the like along with network intrusion attacks. The traditional manual security detection method still has difficulty in rapidly and efficiently detecting the network attack from massive log files. The manual log detection method based on expert experience and safety rules consumes a large amount of manpower when facing massive safety log files, and has the problems of low efficiency, high cost, serious false report and missed report and the like.
Disclosure of Invention
The intercepting method of the traditional centralized firewall is modified from centralized type to distributed type, the risk brought to the system by single-point faults is reduced, the service pressure of single points is reduced, the pressure is dispersed to different machines, simultaneously, log data are distributed in real time by using log data acquired in real time, the intercepting rule is increased in a self-learning mode, and effective relevant supplement can be carried out on the traditional intercepting rule increasing mode. The ordinary state and the attacked state are distinguished by long-term tracking of the log in the early stage and an artificial intelligence means, the attack is completely identified by automatic operation of the system, the request is automatically intercepted, and the malicious request is intercepted at the fastest speed. The hacker is prevented from quickly attacking the system by means of fast-combat DDOS attack and the like, and the protection capability of network intrusion behavior is further improved.
In order to achieve the above purpose, the technical scheme adopted by the invention comprises the following steps:
a request intercepting method based on log analysis comprises the following steps:
collecting log data related to a user request through a log recovery component;
the log recovery component performs distributed off-line storage and distribution and real-time distribution on the log data;
carrying out classification processing on the distributed off-line storage and distribution log data by a log analyzer, and carrying out automatic learning by a machine learning algorithm to generate a discrimination model;
transmitting the discrimination model to a request interceptor of a server;
the request interceptors acquire the log data distributed in real time, and are mounted on servers in a sidecar mode, wherein one server corresponds to one request interceptor;
the request interceptor determines the attribute of the user request according to the discrimination model;
and judging whether to intercept the user request according to the attribute of the user request.
Further, the automatically learning by a machine learning algorithm and generating the discriminant model includes:
carrying out data cleaning on the log data, removing repeated data and simultaneously supplementing vacant log fields;
performing feature extraction on the cleaned log data to generate feature log information;
and carrying out data annotation on the characteristic log information.
Further, the data tagging the characterized log information comprises:
the annotation type comprises normal log data and abnormal log data;
the normal log data is log data with the running time less than or equal to the preset running time and the data returning to be normal;
the abnormal log data is log data with the running time larger than the preset running time and/or the data return exception.
Further, the attributes of the user request comprise legal request and illegal request;
judging whether to intercept the user request according to the attribute of the user request comprises the following steps: and if the attribute of the user request is an illegal request, the request interceptor intercepts the illegal user request.
Further, the method further comprises: and transmitting the discriminant model to a request interceptor of a server, wherein the discriminant model corresponds to the latest version.
Furthermore, the discrimination model internally comprises a plurality of neural network models;
the plurality of neural network models includes: at least two of an XGB model, a LightGBM model, an RF model, an MLP model, and an LSTM model.
The invention also relates to a request intercepting system based on log analysis, which comprises:
the log collection module is used for collecting log data related to the user request through the log recovery assembly;
the distribution module is used for the log recovery component to perform distributed off-line storage distribution and real-time distribution on the log data;
the processing module is used for carrying out classification processing on the distributed off-line storage and distribution log data by a log analyzer, automatically learning by a machine learning algorithm and generating a discrimination model;
the transmission module is used for transmitting the discrimination model to a request interceptor of a server;
an obtaining module, configured to obtain the log data distributed in real time by the request interceptors, where the request interceptors are mounted on servers in a sidecar mode, and one of the servers corresponds to one of the request interceptors;
the determining module is used for determining the attribute of the user request according to the discrimination model by the request interceptor;
and the judging module is used for judging whether to intercept the user request according to the attribute of the user request.
The invention also relates to a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the above-mentioned method.
The invention also relates to an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor implements the above-mentioned method.
The invention also relates to a computer program product comprising a computer program and/or instructions, characterized in that the computer program and/or instructions, when executed by a processor, implement the steps of the above-mentioned method.
The beneficial effects of the invention are as follows:
the intercepting method of the traditional centralized firewall is modified from centralized type to distributed type, the risk brought to the system by single-point faults is reduced, the service pressure of single points is reduced, the pressure is dispersed to different machines, simultaneously, log data are distributed in real time by using log data acquired in real time, the intercepting rule is increased in a self-learning mode, and effective relevant supplement can be carried out on the traditional intercepting rule increasing mode. The ordinary state and the attacked state are distinguished by long-term tracking of the log in the early stage and an artificial intelligence means, the attack is completely identified by automatic operation of the system, the request is automatically intercepted, and the malicious request is intercepted at the fastest speed. The hacker is prevented from quickly attacking the system by means of fast-combat DDOS attack and the like, and the protection capability of network intrusion behavior is further improved.
Drawings
Fig. 1 is a schematic flow chart of a request intercepting method based on log analysis according to the present invention.
FIG. 2 is a schematic diagram of a request intercepting system based on log analysis according to the present invention.
Detailed Description
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only examples or embodiments of the present description, and that for a person skilled in the art, the present description can also be applied to other similar scenarios on the basis of these drawings without inventive effort. Unless otherwise apparent from the context, or otherwise indicated, like reference numbers in the figures refer to the same structure or operation.
It should be understood that "system," "device," "unit," and/or "module" as used herein is a method for distinguishing between different components, elements, parts, portions, or assemblies of different levels. However, other words may be substituted by other expressions if they accomplish the same purpose.
As used in this specification and the appended claims, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are intended to be inclusive in the plural, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements.
Flow charts are used in this description to illustrate operations performed by a system according to embodiments of the present description. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or several steps of operations may be removed from the processes.
The traditional information security means is generally based on the filtering of request texts, and the filtering is performed in advance by configuring rules in protective means such as firewalls and the like in advance and adding a relevant characteristic regular expression of suspected dangerous requests, so that natural hysteresis is caused, and the attacks of vulnerabilities such as 0day and the like can only depend on the analysis after the incident, and the attacks are usually ended at this time. In the face of targets with high attack value, hackers tend to adopt centralized and short-time explosive attacks, and the adoption of the protection mode can cause the target system to be easily exposed to security risks.
In the current bank safety protection situation, protection is generally based on a protection wall, the technology is more perfect due to long-time development of the protection mode, but the protection mode is also based on a passive protection concept, namely based on a determined rule, and then related request verification and interception are carried out.
According to the method and the device, the ordinary state and the attacked state are distinguished by long-term tracking of the early-stage log and an artificial intelligence means, the attack is completely identified through automatic operation of the system, the request is automatically intercepted, and the malicious request is intercepted at the fastest speed.
The first aspect of the present invention relates to a request intercepting method based on log analysis, the flow of which is shown in fig. 1, and the method comprises the following steps:
collecting log data related to a user request through a log recovery component;
specifically, in this embodiment, a fileteam plug-in is used to collect log data, and when a change in the log file is detected, incremental collection is performed on the changed data. The file change means that the content of the log file changes, during acquisition, a plurality of log data can be acquired from a plurality of websites simultaneously and parallelly, the log data can also be acquired from a plurality of servers of one website simultaneously and parallelly, and during real-time acquisition, the acquisition time and the data transmission rate can be controlled. The log recovery component performs distributed off-line storage distribution and real-time distribution on the log data;
specifically, the kafka cluster is used for converting the collected log data into a message queue, and after the log data is converted into the message queue, the data is distributed in real time: on one hand, the log data is sent to a log analyzer for automatic learning; and on the other hand, sending the log data to the request interceptor.
Transmitting the log data to a log analyzer;
the log analyzer classifies the log data, automatically learns the log data by a machine learning algorithm, and generates a discrimination model;
in the log analyzer, the logs are sorted and sorted, stored by a semi-structured data structure and automatically learned by supervised machine learning in a machine learning algorithm. And carrying out data cleaning on the log data, removing repeated data and simultaneously supplementing vacant log fields.
Specifically, (1) cleaning of vacancy value: ignoring the tuple, filling the vacancy value manually, filling the vacancy value by using a global variable, and filling the vacancy value by using probability statistic function values of an average value, a median value, a maximum value and a minimum value of the attribute, wherein the vacancy value is set as a null field;
(2) And (3) cleaning repeated data: the idea of eliminating repeated records is 'sorting and merging', firstly, the records in a database are sorted, then whether the records are repeated is detected by comparing whether adjacent records are the same, the repeated logs are eliminated after the fact that the data of the repeated logs do not contain time sequence characteristics and accumulative characteristics is determined, and an algorithm for eliminating the repeated logs is a priority queue algorithm;
(3) Cleaning of error data: the error field is set to null or the error data is directly deleted.
Performing feature extraction on the cleaned log data to generate feature log information; and carrying out data annotation on the characteristic log information. And marking the result with the running time less than or equal to the preset running time and the data returning to be normal as normal log data, marking the result with the data returning to be abnormal and/or the running time more than the preset running time as abnormal log data, and obtaining a discrimination model which is exclusive to a certain project or a certain type of server by a continuous self-reinforcement learning means, wherein the mode can adopt a mode of a decision tree.
In some embodiments, the training process of the discriminant model comprises: extracting log dimension characteristics from log data; performing time domain polymerization processing on the log dimension characteristics to obtain time domain dimension characteristics; and training the behavior detection model based on the time domain dimension characteristics.
It can be understood that log dimensional features, which are also called node dimensional features, may be extracted from log data, and in this step, features included in each log are extracted according to original sample log information, so that the log dimensional features are obtained.
Secondly, performing time domain aggregation processing on the log dimension features, for example, performing feature aggregation of a day dimension, a week dimension or a session dimension to obtain time domain dimension features.
In some embodiments, performing time domain aggregation processing on the log dimension feature to obtain a time domain dimension feature includes: and performing time domain aggregation processing on the log dimension characteristics based on the characteristic frequency information and the characteristic category information to obtain time domain dimension characteristics.
It can be understood that, in the process of performing time domain aggregation processing on the log dimension feature, time domain aggregation processing may be performed on the log dimension feature based on the feature frequency information and the feature category information to obtain the time domain dimension feature.
The time-domain dimensional features obtained by performing time-domain aggregation processing based on the feature frequency information and the feature category information include frequency features, the frequency features mainly include features corresponding to the frequency of a user request within a certain period of specified time, for example, the time-dimensional features may include: number of user requests, etc.
In some embodiments, the discriminative model includes a plurality of neural network models therein; the training process of the discriminant model comprises the following steps: respectively inputting the log data into a plurality of neural network models of the discrimination model to obtain a user request attribute result output by each neural network model and determine a weight parameter of each neural network model; and establishing a discrimination model based on the plurality of neural network models and the corresponding weight parameters.
It can be understood that, during the training of the discriminant model, the log data may be respectively input into a plurality of neural network models corresponding to the discriminant model, each neural network model may output a user request attribute detection result, the user request attribute detection result output by each neural network model is compared with the log data, and a weight parameter of each neural network model is determined according to the comparison result, and during the construction of the discriminant model, each neural network model and the corresponding weight parameter may be used to determine a final discriminant model, for example, a role proportion exerted by each neural network model during the attribute detection process of the user request may be determined according to the weight parameter, so as to determine the attribute of the user request output by the behavior detection model.
The plurality of neural network models includes: at least two of the XGB model, the LightGBM model, the RF model, the MLP model, and the LSTM model.
It can be understood that the discriminant model is a neural network model, the discriminant model may be a convolutional neural network model, a full convolutional neural network model, or a residual neural network model, or may be another type of neural network, and the discriminant model may also be formed by aggregating a plurality of neural network models, where the specific type of the discriminant model is not limited herein.
The neural network model belongs to Artificial Neural Networks (ANNs), also called Neural Networks (NNs) for short, or connection models (ConnectionModels), and is an algorithmic mathematical model simulating animal neural network behavior characteristics and performing distributed parallel information processing. The network achieves the purpose of processing information by adjusting the mutual connection relation among a large number of nodes in the network depending on the complexity of the system.
The deep learning neural network used by the discrimination model can pick out the features in the log data, each feature is used for obtaining an output result, each output result is compared with a sample label, the features meeting the requirements through comparison can be reserved, the features meeting the requirements through comparison are ignored through the Loss parameter, the core features needing to be memorized can be finally learned through continuous iterative training of a large amount of input log data, different core features are classified, and finally newly input log data information can be discriminated according to the core features.
It is worth mentioning that the machine learning algorithm has a good detection effect on abnormal attack modes, and relatively speaking, the machine learning model is light, and the overall model performance is superior to most deep learning models for large-scale log data.
Transmitting the discrimination model to a request interceptor of a server;
and sending the discrimination model to a request interceptor corresponding to the service, and intelligently judging whether each request belongs to a legal request or not by the request interceptor according to the latest model at any time and intercepting illegal requests.
The method comprises the steps that a user request is obtained by the request interceptors, the request interceptors are mounted on servers through a sidecar mode, and one server corresponds to one request interceptor;
in the current bank safety protection situation, protection is generally based on a protection wall, and the protection mode leads to more perfect technology after long-term development, but is also based on the concept of passive protection, namely based on a determined rule, and then related request verification and interception are carried out. The core idea is as follows: a guard's mechanism is established, and only one single instance of authentication is performed. There are the following problems: (1) Firewalls cannot resist the latest emerging vulnerabilities of unset policies. The operation of the firewall can be only carried out according to the set rule of the firewall, and active discovery cannot be carried out. And (2) the firewall has failure problems as a node. Each request passes through the firewall, which causes the firewall to become a key node of the system, and if the firewall fails, the service is easily unable to complete the response. (3) Firewalls cannot block most legally open ports. Firewalls cannot defend against requests disguised as normal service.
The request interceptor that this application adopted is mainly responsible for according to the relevant interception of rule to the request of self, and the mechanism of traditional firewall is similar to the request interceptor, but its biggest difference with traditional firewall lies in: the request interceptor is mounted on the server via the sidecar mode, rather than being separate as in a conventional firewall. I.e. one request interceptor for each server. In this way, the single point traffic pressure of the firewall is dispersed. The centralized type is changed into the distributed type, the risk brought to the system by single-point faults is reduced, and meanwhile, the service pressure of single points is reduced, so that the pressure is dispersed to different machines.
The request interceptor determines the attribute of the user request according to the discrimination model;
judging whether to intercept the user request according to the attribute of the user request;
the attribute of the user request comprises a legal request and an illegal request, and if the attribute of the user request is the illegal request, the request interceptor intercepts the illegal user request.
In some embodiments, the method further comprises notifying an administrator of the interception event when the interception occurs, and the information of the interception event comprises: SQL injection, XSS cross-site scripting attack, ELI file inclusion, URL coding bypass, sensitive word file scanning, vulnerability exploitation, scanning attack and error-display testing. According to the detection result of the interception event, if judging the interception event is an attack behavior, alarming the administrator, and displaying the detection result, wherein the method comprises the following steps: event Id, time of transmission event name, event type, threat level, attack IP, impact IP, raw log data, etc. This allows effective measures to be taken immediately upon the occurrence of a network attack.
The interception mode of the traditional centralized firewall is modified, the centralized mode is changed into the distributed mode, the risk brought to the system by single-point faults is reduced, the service pressure of single points is reduced, the pressure is dispersed to different machines, simultaneously, log data collected in real time are utilized and distributed in real time, the interception rules are increased through a self-learning mode, the discrimination model is continuously updated, effective relevant supplement can be carried out on the traditional interception rule increasing mode, hackers are prevented from rapidly attacking the system in modes of fast-combat DDOS attack and the like, and the protection capability of network intrusion behaviors is further improved.
The ordinary state and the attacked state are distinguished by long-term tracking of the log in the early stage and an artificial intelligence means, the attack is completely identified by automatic operation of the system, the request is automatically intercepted, and the malicious request is intercepted at the fastest speed.
Another aspect of the present invention also relates to a request intercepting system based on log analysis, whose structure is shown in fig. 2, including:
the log collection module is used for collecting log data related to the user request through the log recovery assembly;
the distribution module is used for the log recovery component to perform distributed off-line storage distribution and real-time distribution on the log data;
the processing module is used for carrying out classification processing on the distributed off-line storage and distribution log data by a log analyzer, automatically learning by a machine learning algorithm and generating a discrimination model;
the transmission module is used for transmitting the discrimination model to a request interceptor of a server;
an obtaining module, configured to obtain the log data distributed in real time by the request interceptors, where the request interceptors are mounted on servers in a sidecar mode, and one of the servers corresponds to one of the request interceptors;
the determining module is used for determining the attribute of the user request according to the discrimination model by the request interceptor;
and the judging module is used for judging whether to intercept the user request according to the attribute of the user request.
By using this system, the above-described arithmetic processing method can be executed, and a corresponding technical effect can be achieved.
An embodiment of the present invention further provides a computer-readable storage medium capable of implementing all the steps of the request intercepting method based on log analysis in the above embodiment, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements all the steps of the request intercepting method based on log analysis in the above embodiment.
Embodiments of the present invention further provide an electronic device for executing the method, as an implementation apparatus of the method, the electronic device at least includes a processor and a memory, and particularly, the memory stores data and related computer programs required for executing the method, and the processor calls the data and the programs in the memory to execute all steps of the implementation method, so as to obtain corresponding technical effects.
Preferably, the electronic device may comprise a bus architecture, which may include any number of interconnected buses and bridges that link together various circuits, including one or more processors and memory. The bus may also link various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a means for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor in performing operations.
Additionally, the electronic device may further include a communication module, an input unit, an audio processor, a display, a power source, and the like. The processor (or controller, operational controls) employed may include a microprocessor or other processor device and/or logic device that receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices, and may store the above related data information, and may further store a program for executing the related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used for providing input to the processor, and can be a key or a touch input device; the power supply is used for supplying power to the electronic equipment; the display is used for displaying display objects such as images and characters, and may be an LCD display, for example. The communication module is a transmitter/receiver that transmits and receives signals via an antenna. The communication module (transmitter/receiver) is coupled to the processor to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal. Based on different communication technologies, a plurality of communication modules, such as a cellular network module, a bluetooth module and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone to implement the usual telecommunication functions. The audio processor may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor is also coupled to the central processor, so that recording on the local machine can be realized through the microphone, and sound stored on the local machine can be played through the loudspeaker.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a system for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the invention.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (10)

1. A request intercepting method based on log analysis is characterized by comprising the following steps:
collecting log data related to a user request through a log recovery component;
the log recovery component performs distributed off-line storage distribution and real-time distribution on the log data;
carrying out classification processing on the distributed off-line storage and distribution log data by a log analyzer, and carrying out automatic learning by a machine learning algorithm to generate a discrimination model;
transmitting the discrimination model to a request interceptor of a server;
the request interceptors acquire the log data distributed in real time, and are mounted on servers in a sidecar mode, wherein one server corresponds to one request interceptor;
the request interceptor determines the attribute of the user request according to the discrimination model;
and judging whether to intercept the user request according to the attribute of the user request.
2. The method of claim 1, wherein the automatically learning with a machine learning algorithm to generate a discriminant model comprises:
performing data cleaning on the log data, removing repeated data, and simultaneously supplementing vacant log fields;
performing feature extraction on the cleaned log data to generate feature log information;
and carrying out data annotation on the characteristic log information.
3. The method of claim 2, wherein the data tagging the characterized log information comprises:
the marking type comprises normal log data and abnormal log data;
the running time is less than or equal to the preset running time, and the log data with normal data return is marked as normal log data;
and marking the log data with the running time larger than the preset running time and/or the data return exception as the exception log data.
4. The method of claim 3, wherein the attributes of the user request include legitimate requests and illegitimate requests;
judging whether to intercept the user request according to the attribute of the user request comprises the following steps:
and if the attribute of the user request is an illegal request, the request interceptor intercepts the illegal user request.
5. The method of claim 4, wherein the method further comprises: and transmitting the discriminant model to a request interceptor of a server, wherein the discriminant model corresponds to the latest version.
6. The method of any one of claims 1 to 5, wherein the discriminant model internally comprises a plurality of neural network models;
the plurality of neural network models includes: at least two of an XGB model, a LightGBM model, an RF model, an MLP model, and an LSTM model.
7. A request intercepting system based on log analysis, comprising:
the log collection module is used for collecting log data related to the user request through the log recovery component;
the distribution module is used for the log recovery component to perform distributed off-line storage distribution and real-time distribution on the log data;
the processing module is used for carrying out classification processing on the distributed off-line storage and distribution log data by a log analyzer, automatically learning by a machine learning algorithm and generating a discrimination model;
the transmission module is used for transmitting the discrimination model to a request interceptor of a server;
an obtaining module, configured to obtain the log data distributed in real time by the request interceptors, where the request interceptors are mounted on servers in a sidecar mode, and one of the servers corresponds to one of the request interceptors;
the determining module is used for determining the attribute of the user request according to the discrimination model by the request interceptor;
and the judging module is used for judging whether to intercept the user request according to the attribute of the user request.
8. A computer-readable storage medium, characterized in that a computer program is stored on the storage medium, which computer program, when being executed by a processor, carries out the method of any one of claims 1 to 6.
9. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 6 when executing the computer program.
10. A computer program product comprising a computer program and/or instructions, characterized in that the computer program and/or instructions, when executed by a processor, implement the steps of the method of any one of claims 1 to 6.
CN202211508936.6A 2022-11-29 2022-11-29 Request intercepting method and system based on log analysis Pending CN115865472A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211508936.6A CN115865472A (en) 2022-11-29 2022-11-29 Request intercepting method and system based on log analysis

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211508936.6A CN115865472A (en) 2022-11-29 2022-11-29 Request intercepting method and system based on log analysis

Publications (1)

Publication Number Publication Date
CN115865472A true CN115865472A (en) 2023-03-28

Family

ID=85667589

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211508936.6A Pending CN115865472A (en) 2022-11-29 2022-11-29 Request intercepting method and system based on log analysis

Country Status (1)

Country Link
CN (1) CN115865472A (en)

Similar Documents

Publication Publication Date Title
Kanimozhi et al. Artificial intelligence based network intrusion detection with hyper-parameter optimization tuning on the realistic cyber dataset CSE-CIC-IDS2018 using cloud computing
US20210273949A1 (en) Treating Data Flows Differently Based on Level of Interest
US10601848B1 (en) Cyber-security system and method for weak indicator detection and correlation to generate strong indicators
CN111193719A (en) Network intrusion protection system
EP2863611B1 (en) Device for detecting cyber attack based on event analysis and method thereof
CN111245793A (en) Method and device for analyzing abnormity of network data
CA3120156A1 (en) Systems and methods for behavioral threat detection
Abdallah et al. Intrusion detection systems using supervised machine learning techniques: a survey
WO2009037333A2 (en) Intrusion detection method and system
CN115134099B (en) Network attack behavior analysis method and device based on full flow
US20230012220A1 (en) Method for determining likely malicious behavior based on abnormal behavior pattern comparison
US11153332B2 (en) Systems and methods for behavioral threat detection
Fallah et al. Android malware detection using network traffic based on sequential deep learning models
CN113411297A (en) Situation awareness defense method and system based on attribute access control
CN117478433B (en) Network and information security dynamic early warning system
CN113660222A (en) Situation awareness defense method and system based on mandatory access control
CN111049828B (en) Network attack detection and response method and system
CN117454376A (en) Industrial Internet data security detection response and tracing method and device
CN115766235A (en) Network security early warning system and early warning method
Roponena et al. Towards a Human-in-the-Loop Intelligent Intrusion Detection System.
CN115865472A (en) Request intercepting method and system based on log analysis
Peleh et al. Intelligent detection of DDoS attacks in SDN networks
Rani et al. Analysis of machine learning and deep learning intrusion detection system in Internet of Things network
Li et al. IoT Honeypot Scanning and Detection System Based on Authorization Mechanism
Ramos et al. A Machine Learning Based Approach to Detect Stealthy Cobalt Strike C &C Activities from Encrypted Network Traffic

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination