CN111917740A - Abnormal flow alarm log detection method, device, equipment and medium - Google Patents
Abnormal flow alarm log detection method, device, equipment and medium Download PDFInfo
- Publication number
- CN111917740A CN111917740A CN202010680405.XA CN202010680405A CN111917740A CN 111917740 A CN111917740 A CN 111917740A CN 202010680405 A CN202010680405 A CN 202010680405A CN 111917740 A CN111917740 A CN 111917740A
- Authority
- CN
- China
- Prior art keywords
- alarm
- alarm log
- flow
- log
- traffic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Abstract
The application discloses a method, a device, equipment and a medium for detecting an abnormal flow alarm log, wherein the method comprises the following steps: generating a first flow alarm log and a second flow alarm log corresponding to an original flow packet according to a first preset alarm rule and a second preset alarm rule respectively, wherein the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule; labeling the first flow alarm log, and labeling the second flow alarm log according to the label in the first flow alarm log; taking the second flow alarm log with the label as sample data, and training a preset flow alarm log classification model by using the sample data; classifying the acquired traffic alarm log to be detected by using the trained traffic alarm log classification model so as to determine whether the traffic alarm log to be detected is an abnormal traffic alarm log. Therefore, the detection accuracy can be improved, the false alarm rate is reduced, and the flow threat detection capability is enhanced.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for detecting an abnormal traffic alarm log.
Background
The flow alarm data refers to data extracted by a network flow analysis system according to a certain rule after capturing and detecting network access flow data. In the cloud internet era, attackers such as hackers and the like usually attack enterprise websites and business systems on the traffic level by using threat means such as vulnerability attack and the like. Therefore, the flow alarm log corresponding to the flow can be correspondingly detected so as to determine whether abnormal alarm exists or not, so that management of the enterprise website or the business system is facilitated. In the process of detecting the flow alarm log, the model can be trained by a machine learning method to detect the flow alarm log so as to manage the network or the system according to the detection result. In the machine learning process, a large amount of sample data of the flow alarm log is needed to be trained, the condition of insufficient sample amount exists in the sample preparation stage, and the condition of overfitting easily occurs when the sample amount is too small, so that the detection accuracy is low, and more false alarms exist.
Disclosure of Invention
In view of this, an object of the present application is to provide a method, an apparatus, a device, and a medium for detecting an abnormal traffic alarm log, which can improve detection accuracy and reduce false alarm rate. The specific scheme is as follows:
in a first aspect, the present application discloses a method for detecting an abnormal traffic alarm log, including:
generating a first flow alarm log and a second flow alarm log corresponding to an original flow packet according to a first preset alarm rule and a second preset alarm rule respectively, wherein the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule;
labeling the first flow alarm log, and labeling the second flow alarm log according to the label in the first flow alarm log;
taking the second flow alarm log with the label as sample data, and training a preset flow alarm log classification model by using the sample data;
classifying the acquired traffic alarm log to be detected by using the trained traffic alarm log classification model so as to determine whether the traffic alarm log to be detected is an abnormal traffic alarm log.
Optionally, after generating the first traffic alarm log and the second traffic alarm log corresponding to the original traffic packet according to the first preset alarm rule and the second preset alarm rule, the method further includes:
deleting the flow alarm logs which do not belong to the preset alarm type in the first flow alarm log and the second flow alarm log respectively;
and deleting the flow alarm logs meeting preset requirements in the reserved first flow alarm log and the reserved second flow alarm log, wherein the preset requirements comprise that data in the flow alarm logs cannot be analyzed and/or an access address corresponding to the flow alarm logs is a preset access address.
Optionally, before tagging the first traffic alarm log, the method further includes:
judging whether the alarm quantity of the first flow alarm log is less than or equal to the alarm quantity of the second flow alarm log;
and if the alarm quantity of the first flow alarm log is less than or equal to the alarm quantity of the second flow alarm log, labeling the first flow alarm log.
Optionally, the tagging the first traffic alarm log comprises
Acquiring label information of the first flow alarm log, wherein the label information comprises a false alarm label and a non-false alarm label;
and labeling the first flow alarm log according to the label information.
Optionally, the generating a first traffic alarm log and a second traffic alarm log corresponding to the original traffic packet according to the first preset alarm rule and the second preset alarm rule respectively includes:
and generating a first traffic alarm log and a second traffic alarm log corresponding to the original traffic packet according to the first preset alarm rule and the second preset alarm rule respectively through suricata.
Optionally, the tagging the second traffic alarm log according to the tag in the first traffic alarm log includes:
comparing each flow alarm log in the first flow alarm log with each flow alarm log in the second flow alarm log;
and performing label synchronization processing on the same traffic alarm logs in the first traffic alarm log and the second traffic alarm log.
Optionally, after comparing each traffic alarm log in the first traffic alarm log with each traffic alarm log in the second traffic alarm log, the method further includes:
and marking a false alarm label on the flow alarm log which exists in the second flow alarm log and does not exist in the first flow alarm log.
In a second aspect, the present application discloses an abnormal traffic alarm log detection device, including:
the log generation module is used for generating a first flow alarm log and a second flow alarm log corresponding to an original flow packet according to a first preset alarm rule and a second preset alarm rule respectively, wherein the precision of the first preset alarm rule is higher than that of the second preset alarm rule;
the labeling module is used for labeling the first flow alarm log and labeling the second flow alarm log according to the label in the first flow alarm log;
the model training module is used for taking the second flow alarm log with the label as sample data and training a preset flow alarm log classification model by using the sample data;
and the detection module is used for classifying the acquired traffic alarm log to be detected by using the trained traffic alarm log classification model so as to determine whether the traffic alarm log to be detected is an abnormal traffic alarm log.
In a third aspect, the present application discloses an electronic device, comprising:
a memory and a processor;
wherein the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the abnormal flow alarm log detection method disclosed above.
In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the abnormal traffic alert log detection method disclosed in the foregoing.
According to the method, a first flow alarm log and a second flow alarm log corresponding to an original flow packet are generated according to a first preset alarm rule and a second preset alarm rule respectively, wherein the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule, then the first flow alarm log is labeled, the second flow alarm log is labeled according to the label in the first flow alarm log, then the second flow alarm log with the label is used as sample data, a preset flow alarm log classification model is trained, and then the obtained sample data to-be-detected flow alarm log is classified by the trained flow alarm log classification model to determine whether the to-be-detected flow alarm log is an abnormal flow alarm log. Therefore, the original flow packet and the preset alarm rule can be used for generating sample data so as to solve the problem that a training sample is insufficient in machine learning, then the generated training sample is used for training the flow alarm log classification model, and after the flow alarm log classification model is trained, the trained flow alarm log classification model can be used for obtaining the flow alarm log to be detected so as to determine whether the flow alarm log to be detected is an abnormal flow alarm log. Therefore, a large amount of sample data is generated firstly, and a model is trained by utilizing the generated large amount of sample data, so that the detection accuracy can be improved, the false alarm rate can be reduced, and the flow threat detection capability can be enhanced.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a method for detecting an abnormal traffic alarm log according to the present disclosure;
fig. 2 is a flowchart of a specific abnormal traffic alarm log detection method disclosed in the present application;
FIG. 3 is a schematic diagram illustrating a relationship between a first traffic alarm log and a second traffic alarm log according to the present disclosure;
FIG. 4 is a flow chart of a particular method for detecting an abnormal traffic alert log disclosed herein;
fig. 5 is a schematic structural diagram of an abnormal traffic alarm log detection device disclosed in the present application;
fig. 6 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, an embodiment of the present application discloses a method for detecting an abnormal traffic alarm log, where the method includes:
step S11: and generating a first flow alarm log and a second flow alarm log corresponding to the original flow packet according to a first preset alarm rule and a second preset alarm rule respectively, wherein the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule.
In a specific implementation process, a first traffic alarm log and a second traffic alarm log corresponding to an original traffic are generated according to a first preset alarm rule and a second alarm rule, respectively, where the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule. The first preset alarm rule may be obtained first, and the obtained first preset alarm rule is modified to be used as the second preset alarm rule, so that the accuracy of the first preset alarm rule is greater than that of the second preset alarm rule. For example, the first preset alarm rule is alert http any- > any any (msg: "sql injection attack detected"; content: "select"; http _ uri; content: "from"; http _ uri; distance: 0;). And then the first preset alarm rule is modified to be used as a second preset alarm rule, wherein the second preset alarm rule is alert http any- > any any (msg: 'sql injection attack detected'; 'content:' select ';' http _ uri; content: 'from'; 'http _ uri'). This may cause the second traffic alarm log to include the content of the first traffic alarm log.
Before generating the first traffic alarm log and the second traffic alarm log, the original traffic packet needs to be captured. In practical application, the traffic packet can be grabbed on the server by using the tcpdump method, and the grabbed traffic packet is stored as a pcap suffix file as an original traffic packet. In particular, the capturing of traffic packets can be performed on the server by executing a command of "tcpdump-XvvennSs 0-i ens22 tcp-w http.
Step S12: and labeling the first flow alarm log, and labeling the second flow alarm log according to the label in the first flow alarm log.
After the first traffic alarm log and the second traffic alarm log are generated, the traffic alarm log is required to be labeled, and the second traffic alarm log is labeled according to the label in the first traffic alarm log. Specifically, label information of the first flow alarm log may be obtained, where the label information includes a false alarm label and a non-false alarm label; and labeling the first flow alarm log according to the label information.
Since the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule, the second traffic alarm log includes a log in the first traffic alarm log, and therefore the second traffic alarm log can be labeled according to the label in the first traffic alarm log.
Step S13: and taking the second flow alarm log with the label as sample data, and training a preset flow alarm log classification model by using the sample data.
After the second traffic alarm log is labeled according to the label in the first traffic alarm log, the second traffic alarm log with the label can be obtained, the second traffic alarm log with the label can be used as sample data, and a preset traffic alarm log classification model is trained by using the sample data. Before the second flow alarm log is used for training a preset flow alarm log classification model, the method further comprises the following steps: and constructing the traffic alarm log classification model, wherein the traffic alarm log classification model can be a classifier.
Step S14: classifying the acquired traffic alarm log to be detected by using the trained traffic alarm log classification model so as to determine whether the traffic alarm log to be detected is an abnormal traffic alarm log.
After a preset flow alarm log model is trained by using a second flow alarm log with a label, a trained flow alarm log classification model is obtained, the trained flow alarm log classification model can divide input logs into a normal flow alarm log and an abnormal flow alarm log, and the abnormal flow alarm log is also a false alarm flow alarm log. Therefore, the trained traffic alarm log classification model can be used for classifying the acquired traffic alarm log to be detected, so that whether the traffic alarm log to be detected is an abnormal traffic alarm log can be determined.
According to the method, a first flow alarm log and a second flow alarm log corresponding to an original flow packet are generated according to a first preset alarm rule and a second preset alarm rule respectively, wherein the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule, then the first flow alarm log is labeled, the second flow alarm log is labeled according to the label in the first flow alarm log, then the second flow alarm log with the label is used as sample data, a preset flow alarm log classification model is trained, and then the obtained sample data to-be-detected flow alarm log is classified by the trained flow alarm log classification model to determine whether the to-be-detected flow alarm log is an abnormal flow alarm log. Therefore, the original flow packet and the preset alarm rule can be used for generating sample data so as to solve the problem that a training sample is insufficient in machine learning, then the generated training sample is used for training the flow alarm log classification model, and after the flow alarm log classification model is trained, the trained flow alarm log classification model can be used for obtaining the flow alarm log to be detected so as to determine whether the flow alarm log to be detected is an abnormal flow alarm log. Therefore, a large amount of sample data is generated firstly, and a model is trained by utilizing the generated large amount of sample data, so that the detection accuracy can be improved, the false alarm rate can be reduced, and the flow threat detection capability can be enhanced.
Referring to fig. 2, an embodiment of the present application discloses a specific method for detecting an abnormal traffic alarm log, where the method includes:
step S21: and generating a first traffic alarm log and a second traffic alarm log corresponding to the original traffic packet according to a first preset alarm rule and a second preset alarm rule respectively through suricata, wherein the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule.
In practical application, suricata can be used to generate a first traffic alarm log and a second traffic alarm log corresponding to an original traffic packet according to a first preset alarm rule and a second preset alarm rule, respectively, where the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule. Specifically, the first preset alarm rule and the second preset alarm rule may be loaded by modifying a configuration file of suricata, and two log files are respectively generated in a suricata-r http. Referring to fig. 3, the second traffic alarm log includes a log of the first traffic alarm log. Each line in the first flow alarm log and the second flow alarm log is in a json format and represents one flow alarm log.
Step S22: and deleting the flow alarm logs which do not belong to the preset alarm type in the first flow alarm log and the second flow alarm log respectively.
After obtaining the first traffic alarm log and the second traffic alarm log, the method further needs to perform corresponding processing on the first traffic alarm log and the second traffic alarm log, including: and deleting the flow alarm logs which do not belong to the preset alarm type in the first flow alarm log and the second flow alarm log respectively. The preset alarm type can be determined according to actual conditions. For example, the data of the http protocol needs to be tagged, the appProtocol field of json in each row in the first traffic alarm log and the second traffic alarm log can be reserved for http data through python, and the rest of the data is deleted.
Step S23: and deleting the flow alarm logs meeting preset requirements in the reserved first flow alarm log and the reserved second flow alarm log, wherein the preset requirements comprise that data in the flow alarm logs cannot be analyzed and/or an access address corresponding to the flow alarm logs is a preset access address.
After the logs belonging to the preset alarm type in the first traffic alarm log and the second traffic alarm log are reserved, the traffic alarm logs meeting preset requirements in the reserved first traffic alarm log and the reserved second traffic alarm log are also required to be deleted, wherein the preset requirements comprise that data in the traffic alarm logs cannot be analyzed and/or that an access address corresponding to the traffic alarm logs is a preset access address. The reserved first traffic alarm log and the second traffic alarm log may have a large amount of invalid data which cannot be resolved or generated by the service, so that the amount needs to be deleted. For example, if there is a part of information transmitted in the traffic system as a byte stream, the part of information is not readable, and the generated log is a messy code, but matches with a preset alarm rule, the part of data in the first traffic alarm log and the second traffic alarm log may be filtered by extracting a feature and compiling a script python, and the part of data is removed. Specifically, the quality control is performed on the data in the first traffic alarm log and the second traffic alarm log, so that the availability of the subsequently obtained sample data is high.
Step S24: and labeling the first flow alarm log, and labeling the second flow alarm log according to the label in the first flow alarm log.
After the first traffic alarm log and the second traffic alarm log are processed correspondingly, the first traffic alarm log can be labeled.
Before tagging the first traffic alarm log, the method further comprises: judging whether the alarm quantity of the first flow alarm log is less than or equal to the alarm quantity of the second flow alarm log; and if the alarm quantity of the first flow alarm log is less than or equal to the alarm quantity of the second flow alarm log, labeling the first flow alarm log. In practical application, in order to label the second traffic alarm log according to the label of the first traffic alarm log, the alarm amount in the first traffic alarm log needs to be less than or equal to the alarm amount in the second traffic alarm log. And if the alarm quantity in the first flow alarm log is greater than that in the second flow alarm log, acquiring the first preset alarm rule and the second alarm rule. Specifically, the number of alarms in the first traffic alarm log and the second traffic alarm log may be counted by using a script such as python.
And when the alarm quantity in the first flow alarm log is less than or equal to the alarm quantity in the second flow alarm log, marking the first flow alarm log with a label. Labeling the first flow alarm log, wherein the labeling comprises obtaining label information of the first flow alarm log, and the label information comprises a false alarm label and a non-false alarm label; and labeling the first flow alarm log according to the label information. The first flow alarm log can also be sent to the ES through logstack
(elastic search) and screening and labeling the alarm data by means of kibana and the like. For example, if a field exists in the data to uniquely determine the alarm, the field is used as the value of the record, and whether the value is misreported or not is recorded.
After the first traffic alarm log is labeled, the second traffic alarm log can be labeled according to the first traffic alarm log. Labeling the second traffic alarm log according to the label in the first traffic alarm log, including: comparing each flow alarm log in the first flow alarm log with each flow alarm log in the second flow alarm log; and performing label synchronization processing on the first traffic alarm log and the same traffic alarm log in the first traffic alarm log. And marking a false alarm label on the flow alarm log which exists in the second flow alarm log and does not exist in the first flow alarm log. Specifically, each flow alarm log in the first flow alarm log is compared with each flow alarm log in the second flow alarm log, a label synchronization process is performed on the common flow alarm log in the first flow alarm log and the second flow alarm log, and a misjudgment label is marked on the flow alarm log which is present in the second flow alarm log but not present in the first flow alarm log.
Step S25: and taking the second flow alarm log with the label as sample data, and training a preset flow alarm log classification model by using the sample data.
Step S26: classifying the acquired traffic alarm log to be detected by using the trained traffic alarm log classification model so as to determine whether the traffic alarm log to be detected is an abnormal traffic alarm log.
The specific implementation processes of step S25 and step S26 may refer to the contents disclosed in the foregoing embodiments, and are not repeated.
Referring to fig. 4, a partial flowchart of the abnormal traffic alarm log detection method is shown. Firstly, two suricata rules, namely a first preset alarm rule and a second preset alarm rule, are obtained, and a flow packet is captured, and the packet capture can be performed by using the PCAP. And then generating 2 traffic alarm logs, namely a first traffic alarm log and a second traffic alarm log, by suricata according to two suricata rules and the captured traffic packets. And performing type filtering on the two flow alarm logs, reserving the required type logs, and performing quality control on the data reserved in the two flow alarm logs, namely deleting the data which cannot be decoded and the like. And then screening and labeling false alarms in the flow alarm log generated by the first rule, wherein the flow alarm log generated by the first rule is also the first flow alarm log. And comparing the flow alarm log generated by the first rule with the flow alarm log generated by the second rule, and labeling data in the flow alarm log generated by the second rule in batches.
Referring to fig. 5, an embodiment of the present application discloses an abnormal traffic alarm log detection device, including:
the log generating module 11 is configured to generate a first traffic alarm log and a second traffic alarm log corresponding to an original traffic packet according to a first preset alarm rule and a second preset alarm rule, respectively, where accuracy of the first preset alarm rule is higher than that of the second preset alarm rule;
a labeling module 12, configured to label the first traffic alarm log, and label the second traffic alarm log according to a label in the first traffic alarm log;
the model training module 13 is configured to use the second traffic alarm log with the label as sample data, and train a preset traffic alarm log classification model by using the sample data;
the detection module 14 is configured to classify the acquired traffic alarm log to be detected by using the trained traffic alarm log classification model, so as to determine whether the traffic alarm log to be detected is an abnormal traffic alarm log.
According to the method, a first flow alarm log and a second flow alarm log corresponding to an original flow packet are generated according to a first preset alarm rule and a second preset alarm rule respectively, wherein the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule, then the first flow alarm log is labeled, the second flow alarm log is labeled according to the label in the first flow alarm log, then the second flow alarm log with the label is used as sample data, a preset flow alarm log classification model is trained, and then the obtained sample data to-be-detected flow alarm log is classified by the trained flow alarm log classification model to determine whether the to-be-detected flow alarm log is an abnormal flow alarm log. Therefore, the original flow packet and the preset alarm rule can be used for generating sample data so as to solve the problem that a training sample is insufficient in machine learning, then the generated training sample is used for training the flow alarm log classification model, and after the flow alarm log classification model is trained, the trained flow alarm log classification model can be used for obtaining the flow alarm log to be detected so as to determine whether the flow alarm log to be detected is an abnormal flow alarm log. Therefore, a large amount of sample data is generated firstly, and a model is trained by utilizing the generated large amount of sample data, so that the detection accuracy can be improved, the false alarm rate can be reduced, and the flow threat detection capability can be enhanced.
Further, the abnormal flow alarm log detection device further includes:
the first data processing module is used for deleting the flow alarm logs which do not belong to the preset alarm type in the first flow alarm log and the second flow alarm log respectively;
and the second data processing module is used for deleting the flow alarm logs meeting preset requirements in the reserved first flow alarm logs and the reserved second flow alarm logs, wherein the preset requirements comprise that data in the flow alarm logs cannot be analyzed and/or that the access addresses corresponding to the flow alarm logs are preset access addresses.
Further, the abnormal flow alarm log detection device further includes:
the judging module is used for judging whether the alarm quantity of the first flow alarm log is less than or equal to the alarm quantity of the second flow alarm log;
correspondingly, the tagging module 12 is configured to tag the first traffic alarm log when the alarm amount of the first traffic alarm log is less than or equal to the alarm amount of the second traffic alarm log.
Specifically, the labeling module 12 includes:
an information obtaining unit, configured to obtain tag information of the first flow alarm log, where the tag information includes a false alarm tag and a non-false alarm tag;
and the first labeling unit is used for labeling the first flow alarm log according to the label information.
In practical application, the log generating module 11 is specifically configured to:
and generating a first traffic alarm log and a second traffic alarm log corresponding to the original traffic packet according to the first preset alarm rule and the second preset alarm rule respectively through suricata.
Specifically, the labeling module 12 includes:
a comparison unit, configured to compare each of the first traffic alarm logs with each of the second traffic alarm logs respectively;
and the label synchronization unit is used for performing label synchronization processing on the same traffic alarm logs in the first traffic alarm log and the second traffic alarm log.
Specifically, the labeling module 12 includes:
and the second labeling unit is used for labeling the flow alarm logs which exist in the second flow alarm log and do not exist in the first flow alarm log with false alarm labels.
Referring to fig. 6, a schematic structural diagram of an electronic device 20 provided in the embodiment of the present application is shown, where the electronic device 20 can implement the abnormal traffic alarm log detection method disclosed in the foregoing embodiment.
In general, the electronic device 20 in the present embodiment includes: a processor 21 and a memory 22.
The processor 21 may include one or more processing cores, such as a four-core processor, an eight-core processor, and so on. The processor 21 may be implemented by at least one hardware of a DSP (digital signal processing), an FPGA (field-programmable gate array), and a PLA (programmable logic array). The processor 21 may also include a main processor and a coprocessor, where the main processor is a processor for processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a GPU (graphics processing unit) which is responsible for rendering and drawing images to be displayed on the display screen. In some embodiments, the processor 21 may include an AI (artificial intelligence) processor for processing computing operations related to machine learning.
Memory 22 may include one or more computer-readable storage media, which may be non-transitory. Memory 22 may also include high speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 22 is at least used for storing the following computer program 221, wherein after being loaded and executed by the processor 21, the computer program can implement the method steps of detecting the abnormal traffic alarm log disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 22 may also include an operating system 222, data 223, and the like, and the storage manner may be a transient storage or a permanent storage. The operating system 222 may be Windows, Unix, Linux, or the like. Data 223 may include a wide variety of data.
In some embodiments, the electronic device 20 may further include a display 23, an input/output interface 24, a communication interface 25, a sensor 26, a power supply 27, and a communication bus 28.
Those skilled in the art will appreciate that the configuration shown in FIG. 6 is not limiting of electronic device 20 and may include more or fewer components than those shown.
Further, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, where the computer program is executed by a processor to implement the abnormal flow alarm log detection method disclosed in any of the foregoing embodiments.
The specific process of the above abnormal traffic alarm log detection method may refer to the corresponding content disclosed in the foregoing embodiments, and is not described herein again.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of other elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The method, the device, the equipment and the medium for detecting the abnormal flow alarm log provided by the application are introduced in detail, a specific example is applied in the method to explain the principle and the implementation mode of the application, and the description of the embodiment is only used for helping to understand the method and the core idea of the application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A method for detecting an abnormal flow alarm log is characterized by comprising the following steps:
generating a first flow alarm log and a second flow alarm log corresponding to an original flow packet according to a first preset alarm rule and a second preset alarm rule respectively, wherein the accuracy of the first preset alarm rule is higher than that of the second preset alarm rule;
labeling the first flow alarm log, and labeling the second flow alarm log according to the label in the first flow alarm log;
taking the second flow alarm log with the label as sample data, and training a preset flow alarm log classification model by using the sample data;
classifying the acquired traffic alarm log to be detected by using the trained traffic alarm log classification model so as to determine whether the traffic alarm log to be detected is an abnormal traffic alarm log.
2. The method for detecting an abnormal traffic alarm log according to claim 1, wherein after the first traffic alarm log and the second traffic alarm log corresponding to the original traffic packet are generated according to the first preset alarm rule and the second preset alarm rule, the method further comprises:
deleting the flow alarm logs which do not belong to the preset alarm type in the first flow alarm log and the second flow alarm log respectively;
and deleting the flow alarm logs meeting preset requirements in the reserved first flow alarm log and the reserved second flow alarm log, wherein the preset requirements comprise that data in the flow alarm logs cannot be analyzed and/or an access address corresponding to the flow alarm logs is a preset access address.
3. The method according to claim 1, wherein before tagging the first traffic alarm log, the method further comprises:
judging whether the alarm quantity of the first flow alarm log is less than or equal to the alarm quantity of the second flow alarm log;
and if the alarm quantity of the first flow alarm log is less than or equal to the alarm quantity of the second flow alarm log, labeling the first flow alarm log.
4. The abnormal traffic alarm log detection method according to claim 1, wherein the labeling the first traffic alarm log comprises
Acquiring label information of the first flow alarm log, wherein the label information comprises a false alarm label and a non-false alarm label;
and labeling the first flow alarm log according to the label information.
5. The method for detecting the abnormal traffic alarm log according to claim 1, wherein the generating a first traffic alarm log and a second traffic alarm log corresponding to an original traffic packet according to a first preset alarm rule and a second preset alarm rule respectively comprises:
and generating a first traffic alarm log and a second traffic alarm log corresponding to the original traffic packet according to the first preset alarm rule and the second preset alarm rule respectively through suricata.
6. The method for detecting the abnormal traffic alarm log according to any one of claims 1 to 5, wherein the labeling the second traffic alarm log according to the label in the first traffic alarm log comprises:
comparing each flow alarm log in the first flow alarm log with each flow alarm log in the second flow alarm log;
and performing label synchronization processing on the same traffic alarm logs in the first traffic alarm log and the second traffic alarm log.
7. The method according to claim 6, wherein after comparing each of the first flow alarm logs with each of the second flow alarm logs, the method further comprises:
and marking a false alarm label on the flow alarm log which exists in the second flow alarm log and does not exist in the first flow alarm log.
8. An abnormal flow alarm log detection device, comprising:
the log generation module is used for generating a first flow alarm log and a second flow alarm log corresponding to an original flow packet according to a first preset alarm rule and a second preset alarm rule respectively, wherein the precision of the first preset alarm rule is higher than that of the second preset alarm rule;
the labeling module is used for labeling the first flow alarm log and labeling the second flow alarm log according to the label in the first flow alarm log;
the model training module is used for taking the second flow alarm log with the label as sample data and training a preset flow alarm log classification model by using the sample data;
and the detection module is used for classifying the acquired traffic alarm log to be detected by using the trained traffic alarm log classification model so as to determine whether the traffic alarm log to be detected is an abnormal traffic alarm log.
9. An electronic device, comprising:
a memory and a processor;
wherein the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the abnormal traffic alarm log detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the abnormal traffic alert log detection method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010680405.XA CN111917740B (en) | 2020-07-15 | 2020-07-15 | Abnormal flow alarm log detection method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010680405.XA CN111917740B (en) | 2020-07-15 | 2020-07-15 | Abnormal flow alarm log detection method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111917740A true CN111917740A (en) | 2020-11-10 |
| CN111917740B CN111917740B (en) | 2022-08-26 |
Family
ID=73281182
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010680405.XA Active CN111917740B (en) | 2020-07-15 | 2020-07-15 | Abnormal flow alarm log detection method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111917740B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685277A (en) * | 2020-12-31 | 2021-04-20 | 海光信息技术股份有限公司 | Warning information checking method and device, electronic equipment and readable storage medium |
| CN112711516A (en) * | 2021-03-26 | 2021-04-27 | 腾讯科技（深圳）有限公司 | Data processing method and related device |
| CN112948211A (en) * | 2021-02-26 | 2021-06-11 | 杭州安恒信息技术股份有限公司 | Alarm method, device, equipment and medium based on log processing |
| CN112953756A (en) * | 2021-01-22 | 2021-06-11 | 武汉武钢绿色城市技术发展有限公司 | Network loop elimination method based on python |
| CN113052338A (en) * | 2021-03-31 | 2021-06-29 | 上海天旦网络科技发展有限公司 | Operation and maintenance alarm rule generation method and system based on rule and model enhancement |
| CN113852591A (en) * | 2021-06-08 | 2021-12-28 | 天翼智慧家庭科技有限公司 | Camera abnormal access identification and alarm method based on improved quartile bit difference method |
| CN114070642A (en) * | 2021-11-26 | 2022-02-18 | 中国电信股份有限公司 | Network security detection method, system, device and storage medium |
| CN114466009A (en) * | 2021-12-22 | 2022-05-10 | 天翼云科技有限公司 | Data processing method, edge super-fusion terminal, cloud terminal and readable storage medium |
| CN115766079A (en) * | 2022-10-10 | 2023-03-07 | 北京明朝万达科技股份有限公司 | Flow data processing method and device, electronic equipment and readable storage medium |
| CN116860578A (en) * | 2023-07-07 | 2023-10-10 | 广州守恶网络科技有限公司 | Network and information security log management system and method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109344862A (en) * | 2018-08-21 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Acquisition methods, device, computer equipment and the storage medium of positive sample |
| US20190236460A1 (en) * | 2018-01-29 | 2019-08-01 | Salesforce.Com, Inc. | Machine learnt match rules |
| CN110717551A (en) * | 2019-10-18 | 2020-01-21 | 中国电子信息产业集团有限公司第六研究所 | Training method and device of flow identification model and electronic equipment |
| US20200042829A1 (en) * | 2017-07-24 | 2020-02-06 | Huawei Technologies Co., Ltd. | Classification Model Training Method and Apparatus |
| WO2020037898A1 (en) * | 2018-08-23 | 2020-02-27 | 平安科技（深圳）有限公司 | Face feature point detection method and apparatus, computer device, and storage medium |
| CN111222648A (en) * | 2020-01-15 | 2020-06-02 | 深圳前海微众银行股份有限公司 | Semi-supervised machine learning optimization method, device, equipment and storage medium |
| CN111277606A (en) * | 2020-02-10 | 2020-06-12 | 北京邮电大学 | Detection model training method, detection method and device, and storage medium |
| CN111291895A (en) * | 2020-01-17 | 2020-06-16 | 支付宝(杭州)信息技术有限公司 | Sample generation and training method and device for combined feature evaluation model |
-
2020
- 2020-07-15 CN CN202010680405.XA patent/CN111917740B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200042829A1 (en) * | 2017-07-24 | 2020-02-06 | Huawei Technologies Co., Ltd. | Classification Model Training Method and Apparatus |
| US20190236460A1 (en) * | 2018-01-29 | 2019-08-01 | Salesforce.Com, Inc. | Machine learnt match rules |
| CN109344862A (en) * | 2018-08-21 | 2019-02-15 | 中国平安人寿保险股份有限公司 | Acquisition methods, device, computer equipment and the storage medium of positive sample |
| WO2020037898A1 (en) * | 2018-08-23 | 2020-02-27 | 平安科技（深圳）有限公司 | Face feature point detection method and apparatus, computer device, and storage medium |
| CN110717551A (en) * | 2019-10-18 | 2020-01-21 | 中国电子信息产业集团有限公司第六研究所 | Training method and device of flow identification model and electronic equipment |
| CN111222648A (en) * | 2020-01-15 | 2020-06-02 | 深圳前海微众银行股份有限公司 | Semi-supervised machine learning optimization method, device, equipment and storage medium |
| CN111291895A (en) * | 2020-01-17 | 2020-06-16 | 支付宝(杭州)信息技术有限公司 | Sample generation and training method and device for combined feature evaluation model |
| CN111277606A (en) * | 2020-02-10 | 2020-06-12 | 北京邮电大学 | Detection model training method, detection method and device, and storage medium |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685277A (en) * | 2020-12-31 | 2021-04-20 | 海光信息技术股份有限公司 | Warning information checking method and device, electronic equipment and readable storage medium |
| CN112953756A (en) * | 2021-01-22 | 2021-06-11 | 武汉武钢绿色城市技术发展有限公司 | Network loop elimination method based on python |
| CN112948211A (en) * | 2021-02-26 | 2021-06-11 | 杭州安恒信息技术股份有限公司 | Alarm method, device, equipment and medium based on log processing |
| CN112711516A (en) * | 2021-03-26 | 2021-04-27 | 腾讯科技（深圳）有限公司 | Data processing method and related device |
| CN113052338A (en) * | 2021-03-31 | 2021-06-29 | 上海天旦网络科技发展有限公司 | Operation and maintenance alarm rule generation method and system based on rule and model enhancement |
| CN113852591A (en) * | 2021-06-08 | 2021-12-28 | 天翼智慧家庭科技有限公司 | Camera abnormal access identification and alarm method based on improved quartile bit difference method |
| CN113852591B (en) * | 2021-06-08 | 2023-09-22 | 天翼数字生活科技有限公司 | Camera abnormal access identification and alarm method based on improved four-level difference method |
| CN114070642A (en) * | 2021-11-26 | 2022-02-18 | 中国电信股份有限公司 | Network security detection method, system, device and storage medium |
| CN114466009A (en) * | 2021-12-22 | 2022-05-10 | 天翼云科技有限公司 | Data processing method, edge super-fusion terminal, cloud terminal and readable storage medium |
| CN115766079A (en) * | 2022-10-10 | 2023-03-07 | 北京明朝万达科技股份有限公司 | Flow data processing method and device, electronic equipment and readable storage medium |
| CN115766079B (en) * | 2022-10-10 | 2023-12-05 | 北京明朝万达科技股份有限公司 | Traffic data processing method and device, electronic equipment and readable storage medium |
| CN116860578A (en) * | 2023-07-07 | 2023-10-10 | 广州守恶网络科技有限公司 | Network and information security log management system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111917740B (en) | 2022-08-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111917740B (en) | Abnormal flow alarm log detection method, device, equipment and medium | |
| US9954805B2 (en) | Graymail filtering-based on user preferences | |
| CN108718298B (en) | Malicious external connection flow detection method and device | |
| CN109492118B (en) | Data detection method and detection device | |
| CN107395650B (en) | Method and device for identifying Trojan back connection based on sandbox detection file | |
| CN111368289B (en) | Malicious software detection method and device | |
| CN113051543B (en) | Cloud service security verification method and cloud service system in big data environment | |
| CN113507461B (en) | Network monitoring system and network monitoring method based on big data | |
| CN111177795A (en) | Method, device and computer storage medium for identifying video tampering by using block chain | |
| CN110602030A (en) | Network intrusion blocking method, server and computer readable medium | |
| CN111049783A (en) | Network attack detection method, device, equipment and storage medium | |
| CN111464510B (en) | Network real-time intrusion detection method based on rapid gradient lifting tree classification model | |
| CN113472803A (en) | Vulnerability attack state detection method and device, computer equipment and storage medium | |
| CN108156127B (en) | Network attack mode judging device, judging method and computer readable storage medium thereof | |
| CN113472580B (en) | Alarm system and alarm method based on dynamic loading mechanism | |
| CN113886829A (en) | Method and device for detecting defect host, electronic equipment and storage medium | |
| CN115643044A (en) | Data processing method, device, server and storage medium | |
| CN113810342B (en) | Intrusion detection method, device, equipment and medium | |
| CN114756401B (en) | Abnormal node detection method, device, equipment and medium based on log | |
| CN111526110A (en) | Method, device, equipment and medium for detecting unauthorized login of email account | |
| CN113596051B (en) | Detection method, detection apparatus, electronic device, medium, and computer program | |
| US20230291764A1 (en) | Content-based socially-engineered threat classifier | |
| CN113065132B (en) | Method and device for detecting confusion of macro program, electronic equipment and storage medium | |
| CN113765852B (en) | Data packet detection method, system, storage medium and computing device | |
| CN113810386B (en) | Method and device for extracting training data for network security from big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |