CN108763031B - A log-based threat intelligence detection method and device - Google Patents

Abstract

The embodiment of the invention discloses a threat information detection method and a device based on logs, wherein the method comprises the following steps: acquiring log files of different file types, analyzing the log files, matching different threat indicator IOC types, and adding the analyzed files of different IOC types to a detection engine queue; acquiring a target analysis file from the detection engine queue, and determining a corresponding target query mode according to the IOC type of the target analysis file; and if threat intelligence exists in the target analysis file according to the target query mode, generating threat alarm information, such as a lost host or a malicious file. The log files of different file types are analyzed, and the target analysis file is inquired in a corresponding defect detection inquiry mode, so that large-batch data can be processed simultaneously, and the detection efficiency of network security detection on mass data is greatly improved.

Description

A log-based threat intelligence detection method and device

technical field

Embodiments of the present invention relate to the technical field of network security, and in particular, to a log-based threat intelligence detection method and device.

Background technique

With the rapid development of computer technology and network applications, the amount of network information data is increasing, and the data security of massive data has become more and more important. Today's data generated by various applications such as social networks, mobile communications, network video and audio, e-commerce, sensor networks, and scientific experiments can often generate massive data of tens of millions, billions, or even billions and tens of billions. The security of the network requires the detection of these massive data to ensure the safe operation of the network.

The prior art mainly adopts a conventional single-type log detection method, which can cope with a network with a small amount of data. However, for a massive data network, the efficiency of the detection method obviously cannot meet the requirements, resulting in a significant reduction in the network operation speed. At the same time, the coverage of data security detection is small.

In the process of implementing the embodiments of the present invention, the inventor finds that the detection efficiency of the existing method for network security detection of massive data is too low.

SUMMARY OF THE INVENTION

Due to the above problems existing in the existing methods, the embodiments of the present invention provide a log-based threat intelligence detection method and device.

In a first aspect, an embodiment of the present invention provides a log-based threat intelligence detection method, including:

Obtain log files of different file types, parse the log files, match different threat indicator IOC types, obtain parsed files, and add the parsed files to the detection engine queue;

Obtain the target parsing file from the detection engine queue, and determine the target query mode of the corresponding loss detection according to the IOC type of the target parsing file;

If it is found that threat intelligence exists in the target analysis file according to the target query method, threat alarm information is generated.

Optionally, the log file is parsed to obtain parsed files of different threat indicator IOC types, specifically including:

The log file is parsed, the IOC type of the log file is determined by the primary key, and malicious domain names, IPs or malicious files in the log file are detected, and parsed files of different IOC types are obtained.

Optionally, generating threat warning information if it is found that threat intelligence exists in the target analysis file according to the target query method, specifically including:

Perform batch query on the target parsing file according to the target query mode;

If it is found that there is threat intelligence in the target analysis file, threat alarm information is generated.

Optionally, after obtaining log files of different file types, parsing the log files, obtaining parsed files of different threat indicator IOC types, and adding the parsed files of different IOC types to the detection engine queue, also include:

If the data amount of the parsed files of the different IOC types is greater than the threshold, the parsed files of the different IOC types are temporarily stored in the database.

Optionally, before generating threat warning information, if it is found that threat intelligence exists in the target analysis file according to the target query method, the method further includes:

If there is an exception when querying the target parsing file according to the target query mode, the abnormal value queue is stored.

Optionally, the method further includes:

The corresponding display type is determined according to the type of the threat alarm information, the threat alarm information is sent to the display for display according to the display type, an alarm log is generated according to the threat alarm information, and the threat alarm information and the The alarm log is stored.

In a second aspect, an embodiment of the present invention further provides a log-based threat intelligence detection device, including:

a file parsing module, configured to obtain log files of different file types, parse the log files, match different threat index IOC types, obtain parsing files, and add the parsing files to the detection engine queue;

A corresponding query module, configured to obtain a target parsing file from the detection engine queue, and determine a corresponding target query mode for trap detection according to the IOC type of the target parsing file;

A threat warning module, configured to generate threat warning information if threat intelligence exists in the target analysis file according to the target query method.

Optionally, the file parsing module is specifically configured to parse the log file, detect the malicious domain name, IP or malicious file in the log file by using the primary key method of the IOC type of the log file, and obtain different IOCs. Type of parsing file.

Optionally, the threat alarm module is specifically used for:

Perform batch query on the target parsing file according to the target query mode;

If it is found that there is threat intelligence in the target analysis file, threat alarm information is generated.

Optionally, the device further includes:

A file temporary storage module, configured to temporarily store the parsed files of different IOC types in a database if the data amount of the parsed files of different IOC types is greater than a threshold.

Optionally, the device further includes:

The exception saving module is configured to save the abnormal value queue if there is an exception when querying the target parsing file according to the target query mode.

Optionally, the device further includes:

An alarm display module is configured to determine a corresponding display type according to the type of the threat alarm information, send the threat alarm information to a display for display according to the display type, generate an alarm log according to the threat alarm information, and record the The threat alarm information and the alarm log are stored.

In a third aspect, an embodiment of the present invention further provides an electronic device, including:

at least one processor; and

at least one memory communicatively coupled to the processor, wherein:

The memory stores program instructions executable by the processor, the processor invoking the program instructions capable of performing the above-described method.

In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores a computer program, and the computer program causes the computer to execute the above method.

It can be seen from the above technical solutions that the embodiment of the present invention can process large batches of data at the same time by analyzing log files of different file types, and querying the target analysis file by using the corresponding query method of failure detection, which greatly improves the processing efficiency of massive data. Detection efficiency of network security detection.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative efforts.

FIG. 1 is a schematic flowchart of a log-based threat intelligence detection method according to an embodiment of the present invention;

FIG. 2 is a schematic flowchart of a method for detecting a loss according to an embodiment of the present invention;

3 is a schematic diagram of a result of a log-based threat intelligence detection method provided by an embodiment of the present invention;

4 is a schematic diagram of a result of a log-based threat intelligence detection method provided by another embodiment of the present invention;

5 is a schematic structural diagram of a log-based threat intelligence detection device according to an embodiment of the present invention;

FIG. 6 is a logical block diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. The following examples are only used to illustrate the technical solutions of the present invention more clearly, and cannot be used to limit the protection scope of the present invention.

FIG. 1 shows a schematic flowchart of a log-based threat intelligence detection method provided in this embodiment, including:

S101. Obtain log files of different file types, parse the log files, match different threat indicator IOC types, obtain parsed files, and add the parsed files to a detection engine queue.

The file types of log files may include various file methods, such as file methods, logstash methods, or API methods. Specific file types may include log traffic such as JSON, Syslog, Netflow, DNS, and HTTP, as well as log records of the SEIM system. By obtaining log files of different file types, it is convenient to aggregate and parse various types of logs, maximize compatibility and parsing of different types of log formats, and facilitate subsequent detection and matching.

The detection engine queue is a queue for storing parsed files of different IOC types for subsequent failure detection.

S102. Acquire a target parsing file from the detection engine queue, and determine a corresponding target query mode for failure detection according to the IOC type of the target parsing file.

Among them, the query methods of the failure detection include cc (command and control server) query, sinkhole (slot hole) query and dga (DNS domain name generated by a special algorithm) query.

S103. If it is found that threat intelligence exists in the target analysis file according to the target query method, generate threat alarm information.

Specifically, as shown in Figure 2, after obtaining data from the detection engine queue, query the corresponding database through cc, sinkhole or dga, and determine whether the query is successful; if the query is successful, generate threat alarm information, otherwise save the failure field .

This embodiment is divided into four parts as a whole, a log receiving part, a log parsing part, a detection engine part and a log alarm part. By using the above four parts, different from common log detection methods, it can receive various types of log formats, and has excellent performance processing. Through the log processing process of the four major parts, it can effectively combine threat intelligence for accurate log matching.

This embodiment parses log files of different file types, and queries the target parsing file by using the corresponding query method of failure detection, which can process large batches of data at the same time, and greatly improves the detection efficiency of network security detection for massive data.

Further, on the basis of the above method embodiments, the log file is parsed as described in S101 to obtain parsed files of different threat indicator IOC types, specifically including:

The log file is parsed, the IOC type of the log file is determined by the primary key, and malicious domain names, IPs or malicious files in the log file are detected, and parsed files of different IOC types are obtained.

Specifically, through two main module engines, various log formats are detected and processed, and the IOC (Indicators Of Compromise threat indicator) in the intelligence is matched by the main KEY method to detect malicious domain names, IPs or malicious files, and provide detailed information. For attack source tracing, as shown in Figures 3 and 4, it can detect and trace the source of various types of logs on the existing network, such as logs, traffic logs, and file transfers, among various types of threat intelligence information and various types of threat intelligence. .

This embodiment provides an efficient log storage and alarm detection mechanism by matching with massive amounts of intelligence, and can use various existing intelligence information for accurate detection and alarm in the case of heterogeneous data sources with a large amount of logs; An effective way of log analysis, combined with threat intelligence sources, to efficiently discover abnormal attacks and provide traceability.

Further, on the basis of the above method embodiments, S103 specifically includes:

S1031. Perform a batch query on the target parsing file according to the target query mode.

S1032. If it is found that threat intelligence exists in the target analysis file, generate threat alarm information.

Specifically, the compromise detection is the core part of this embodiment, which is used for matching detection of the compromise detection threat intelligence with the original log, and checking the compromised host and abnormal behavior logs. The specific process is shown in Figure 2: First, obtain data from the detection engine queue, and perform batch cc, dga, and sinkhole queries (kc mode); then obtain host, ip, sha1, MD5 from the parsed list, and query kc in batches of 1000 Or kt; then start the process check_compromise.py, assemble the alarm information into a structure through the functions: check_cc, check_dga, check_sinkhole and construct_alert_info, and send it to the corresponding alarm module through the socket.

Through failure detection, malicious domain names, malicious IPs, and malicious files can be efficiently detected; by matching common network logs, traffic logs, and behavior logs, malicious threats can be accurately found and traced in effective IOC intelligence.

Further, on the basis of the above method embodiments, after S101, the method further includes:

S1012. If the data amount of the parsed files of different IOC types is greater than a threshold, temporarily store the parsed files of different IOC types in a database.

Specifically, different data sources are received and parsed through Logstash, and logs with a large amount of data are temporarily stored in a cache queue, so that the computer can read and process from the temporarily stored database. To prevent the data storage from taking up too much memory when the log traffic is too large, you can temporarily store the logs in the kc library. After the query and alarm fields are assembled, an alarm is generated and the information that has been queried in the kc library is deleted.

Further, on the basis of the above method embodiments, before S103, the method further includes:

S1023. If there is an abnormality when querying the target parsing file according to the target query mode, store the abnormal value in a queue.

Specifically, when an abnormal file spreads or appears in the user network, it is necessary to discover malicious samples in time and generate an alarm to provide details of the malicious samples. The data can be obtained from the file reputation detection queue to perform batch file reputation query (kt mode); and executed through the key process check_filerepputation.py, the alarm information is assembled into a structure through the functions: check_info and construct_alert_info, and sent to the corresponding through the socket. Alarm module; when the query is abnormal, save the key list of the abnormality, and let the process sleep if the queue is empty.

Further, on the basis of the above method embodiments, the method further includes:

S104. Determine a corresponding display type according to the type of the threat alarm information, send the threat alarm information to a display for display according to the display type, generate an alarm log according to the threat alarm information, and send the threat alarm information and the alarm log are stored.

Specifically, the threat alarm information detected by matching the threat intelligence is displayed on the front end through the alarm, and the original field log is written into the database. For the alarms matched by different intelligence types, the design details of the alarm fields are slightly different for the convenience of users.

Specifically, the process log_parse.py can be executed, the log is received by the function recv_log, the log is parsed by the function parse_log, and the log is written into the kc engine and cached by the function write_kc.

In this embodiment, by receiving a large number of alarm logs and without storing the original logs for the time being, the malicious threat alarm generated by the detection can be stored and displayed on the interface of the original alarm.

FIG. 5 shows a schematic structural diagram of a log-based threat intelligence detection device provided in this embodiment. The device includes: a file parsing module 501, a corresponding query module 502, and a threat alarm module 503, wherein:

The file parsing module 501 is configured to obtain log files of different file types, parse the log files, match different threat indicator IOC types, obtain parsing files, and add the parsing files to the detection engine queue;

The corresponding query module 502 is configured to obtain a target analysis file from the detection engine queue, and determine a corresponding target query mode for trap detection according to the IOC type of the target analysis file;

The threat alarm module 503 is configured to generate threat alarm information if threat intelligence exists in the target analysis file according to the target query method.

Specifically, the file parsing module 501 obtains log files of different file types, parses the log files, matches different threat indicator IOC types, obtains parsing files, and adds the parsing files to the detection engine queue; the The corresponding query module 502 obtains the target analysis file from the detection engine queue, and determines the corresponding target query mode for failure detection according to the IOC type of the target analysis file; the threat alarm module 503, if the threat alarm module 503 queries according to the target query method. If threat intelligence exists in the target parsing file, threat warning information is generated.

In this embodiment, by analyzing log files of different file types, and querying the target parsing file by using the corresponding failure detection query method, large batches of data can be processed simultaneously, and the detection efficiency of network security detection of massive data can be greatly improved.

Further, on the basis of the above-mentioned apparatus embodiment, the file parsing module 501 is specifically configured to parse the log file, and detect the malicious domain name in the log file by means of the primary key of the IOC type of the log file. , IP or malicious files to get parsed files of different IOC types.

Further, on the basis of the above device embodiments, the threat alarm module 503 is specifically used for:

Perform batch query on the target parsing file according to the target query mode;

If it is found that there is threat intelligence in the target analysis file, threat alarm information is generated.

Further, on the basis of the above device embodiments, the device further includes:

A file temporary storage module, configured to temporarily store the parsed files of different IOC types in a database if the data amount of the parsed files of different IOC types is greater than a threshold.

Further, on the basis of the above device embodiments, the device further includes:

The exception saving module is configured to save the abnormal value queue if there is an exception when querying the target parsing file according to the target query mode.

Further, on the basis of the above device embodiments, the device further includes:

An alarm display module is configured to determine a corresponding display type according to the type of the threat alarm information, send the threat alarm information to a display for display according to the display type, generate an alarm log according to the threat alarm information, and record the The threat alarm information and the alarm log are stored.

The log-based threat intelligence detection apparatus described in this embodiment can be used to execute the foregoing method embodiments, and its principles and technical effects are similar, and details are not repeated here.

6 , the electronic device includes: a processor (processor) 601, a memory (memory) 602 and a bus 603;

in,

The processor 601 and the memory 602 communicate with each other through the bus 603;

The processor 601 is configured to call program instructions in the memory 602 to execute the methods provided by the above method embodiments.

This embodiment discloses a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer program The methods provided by the above method embodiments can be executed.

This embodiment provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the methods provided by the above method embodiments.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

It should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be used for The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (12)

1. A log-based threat intelligence detection method is characterized in that, comprising: Obtain log files of different file types, parse the log files, match different threat indicator IOC types, obtain parsed files, and add the parsed files to the detection engine queue; Obtain the target parsing file from the detection engine queue, and determine the corresponding target query mode for failure detection according to the IOC type of the target analysis file; wherein, the target query mode for failure detection includes command and control server query, slot hole Queries and DNS queries generated by special algorithms; If it is found that threat intelligence exists in the target analysis file according to the target query method, generating threat warning information; Wherein, the said log file is parsed to obtain parsed files of different threat indicator IOC types, specifically including: The log file is parsed, the IOC type of the log file is determined by the primary key mode, and malicious domain names, IPs or malicious files in the log file are detected, and parsed files of different IOC types are obtained. 2. The method according to claim 1, characterized in that, if the target analysis file is queried according to the target query method to find that there is threat intelligence in the target analysis file, generating threat warning information, specifically comprising: Perform batch query on the target parsing file according to the target query mode; If it is found that there is threat intelligence in the target analysis file, threat alarm information is generated. 3. The method according to claim 1, wherein the log files of different file types are obtained, the log files are parsed, the analysis files of different threat indicator IOC types are obtained, and the different IOC types are analyzed. After the parsing file is added to the detection engine queue, it also includes: If the data amount of the parsed files of the different IOC types is greater than the threshold, the parsed files of the different IOC types are temporarily stored in the database. 4. The method according to claim 1, characterized in that, before generating threat warning information, further comprising: If there is an exception when querying the target parsing file according to the target query mode, the abnormal value queue is stored. 5. The method according to any one of claims 1-4, wherein the method further comprises: The corresponding display type is determined according to the type of the threat alarm information, the threat alarm information is sent to the display for display according to the display type, an alarm log is generated according to the threat alarm information, and the threat alarm information and the The alarm log is stored. 6. A log-based threat intelligence detection device, comprising: a file parsing module, configured to obtain log files of different file types, parse the log files, match different threat index IOC types, obtain parsing files, and add the parsing files to the detection engine queue; The corresponding query module is used to obtain the target analysis file from the detection engine queue, and determine the target query mode of the corresponding loss detection according to the IOC type of the target analysis file; wherein, the target query mode of the failure detection includes commands and Control server queries, slot queries and DNS queries generated by special algorithms; a threat warning module, configured to generate threat warning information if threat intelligence exists in the target analysis file according to the target query method; The file parsing module is specifically configured to parse the log file, determine the IOC type of the log file by means of a primary key, and detect malicious domain names, IPs or malicious files in the log file, and obtain different IOC types parsing file. 7. The device according to claim 6, wherein the threat alarm module is specifically used for: Perform batch query on the target parsing file according to the target query mode; If it is found that there is threat intelligence in the target analysis file, threat alarm information is generated. 8. The apparatus of claim 6, wherein the apparatus further comprises: A file temporary storage module, configured to temporarily store the parsed files of different IOC types in a database if the data amount of the parsed files of different IOC types is greater than a threshold. 9. The apparatus of claim 6, wherein the apparatus further comprises: The exception saving module is configured to save the abnormal value queue if there is an exception when querying the target parsing file according to the target query mode. 10. The device according to any one of claims 6-9, wherein the device further comprises: An alarm display module is configured to determine a corresponding display type according to the type of the threat alarm information, send the threat alarm information to a display for display according to the display type, generate an alarm log according to the threat alarm information, and record the The threat alarm information and the alarm log are stored. 11. An electronic device, characterized in that, comprising: at least one processor; and at least one memory communicatively coupled to the processor, wherein: The memory stores program instructions executable by the processor, and the processor invokes the program instructions to perform the method as claimed in any one of claims 1 to 5. 12. A non-transitory computer-readable storage medium, characterized in that, the non-transitory computer-readable storage medium stores a computer program, and the computer program causes the computer to execute any one of claims 1 to 5. Methods.

Images