CN115733701A - Method, device, electronic equipment and storage medium for collecting threat intelligence sources - Google Patents

Abstract

The embodiment of the invention relates to the technical field of internet security, in particular to a method and a device for collecting threat information sources, electronic equipment and a storage medium. The method comprises the following steps: obtaining source addresses of all target threat information sources; detecting and evaluating threat intelligence in the target threat intelligence sources based on source addresses of the target threat intelligence sources to obtain evaluation results corresponding to the target threat intelligence sources; and reporting the source address and the evaluation result corresponding to each target threat intelligence source. The scheme can automatically collect the target threat information source to enhance the collection capability of the threat information source and further increase the collection amount of the threat information.

Description

Method, device, electronic equipment and storage medium for collecting threat intelligence sources

technical field

Embodiments of the present invention relate to the technical field of Internet security, and in particular, to a method, device, electronic device, and storage medium for collecting threat intelligence sources.

Background technique

As the world today is in the stage of rapid development of the Internet, the network attack and defense environment is becoming more and more complex. Quick and accurate elimination of network threats has become a necessary requirement in today's network environment. Threat intelligence, as a medium for providing network security information, is the ballast stone to ensure network security, a stabilizer to help network security, and a key point to promote the development of network security.

The existing collection methods of threat intelligence sources are mainly to artificially search and collect threat intelligence sources, which will undoubtedly affect the collection efficiency of threat intelligence sources.

Therefore, there is an urgent need for a new method of collecting threat intelligence sources.

Contents of the invention

In order to solve the problem of low collection efficiency of artificially collected threat information sources, embodiments of the present invention provide a method, device, electronic device, and storage medium for collecting threat information sources.

In the first aspect, an embodiment of the present invention provides a method for collecting threat intelligence sources, including:

Obtain the source address of each target threat intelligence source;

Based on the source address of the target threat intelligence source, detect and evaluate the threat intelligence in the target threat intelligence source, so as to obtain the evaluation result of each target threat intelligence source;

Reporting the source address of each target threat intelligence source and the evaluation result.

Preferably, the acquisition method of the source address of the target threat intelligence source includes a first acquisition method based on known threat intelligence sources and a second acquisition method based on the GitHub platform.

Preferably, the first obtaining method is to obtain the source address of each target threat intelligence source in the following manner:

Obtain a list of source addresses of known threat intelligence sources;

For each known threat intelligence source in the source address list, execute:

Access the source address of currently known threat intelligence feeds;

Obtain the website link in the access page of the current known threat intelligence source, so as to determine the source address of the target threat intelligence source corresponding to the current known threat intelligence source.

Preferably, after obtaining the website link in the access page of the current known threat intelligence source, and before determining the source address of the target threat intelligence source corresponding to the current known threat intelligence source, the method further includes:

Obtain the domain name of the current known threat intelligence source based on the source address of the current known threat intelligence source;

Determine whether each of the website links in the access page of the current known threat intelligence source is the same as the domain name of the current known threat intelligence source;

A website link different from the domain name of the current known threat intelligence source is determined as the source address of the target threat intelligence source corresponding to the current known threat intelligence source.

Preferably, the second obtaining method is to obtain the source address of each target threat intelligence source in the following manner:

Identify multiple keywords to use in searching for targeted threat intelligence feeds;

Search for each of the keywords on the GitHub platform, and determine each warehouse address obtained from the search as the source address of the target threat intelligence source.

Preferably, based on the source address of the target threat intelligence source, the threat intelligence in the target threat intelligence source is detected and evaluated, so as to obtain the evaluation result of each target threat intelligence source, including:

For each of the targeted threat intelligence feeds described, do:

Based on the source address of the current target threat intelligence source, detect the amount of threat intelligence in the current target threat intelligence source;

Based on the type of each threat intelligence detected in the current target threat intelligence source, determine whether the format of each threat intelligence meets the requirements;

Determine whether each threat intelligence in the current target threat intelligence source has been included in the pre-built threat intelligence library;

Get an assessment of the current target threat intelligence feed.

Preferably, after the judging whether each threat intelligence in the current target threat intelligence source has been included in the pre-built threat intelligence library, before the evaluation result of the current target threat intelligence source is obtained, the method further includes:

For each threat intelligence in the currently targeted threat intelligence feed, execute:

If the current threat intelligence has been included in the threat intelligence library, obtaining the threat type of the current threat intelligence from the threat intelligence library, so as to add the threat type of the current threat intelligence to the evaluation result of the current target threat intelligence source

In the second aspect, the embodiment of the present invention also provides a threat intelligence source collection device, including:

an obtaining unit, configured to obtain the source address of each target threat intelligence source;

An evaluation unit, configured to detect and evaluate threat intelligence in the target threat intelligence source based on the source address of the target threat intelligence source, so as to obtain an evaluation result corresponding to the target threat intelligence source;

A reporting unit, configured to report the source address corresponding to the target threat intelligence source and the evaluation result.

In a third aspect, an embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the computer program described in any embodiment of this specification can be realized. described method.

In a fourth aspect, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed in a computer, the computer is instructed to execute the method described in any embodiment of this specification .

Embodiments of the present invention provide a method, device, electronic device, and storage medium for collecting threat intelligence sources. First, the source address of each target threat intelligence source is automatically acquired, and based on the acquired source address of each target threat intelligence source, , to detect and evaluate the threat intelligence in each target threat intelligence source to obtain the evaluation result of each target threat intelligence source; finally, report the source address and corresponding evaluation result of each target threat intelligence source, so as to Automatically collecting target threat intelligence sources can enhance the collection capabilities of threat intelligence sources and further increase the amount of threat intelligence collection.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a flow chart of a method for collecting threat intelligence sources provided by an embodiment of the present invention;

FIG. 2 is a hardware architecture diagram of a computing device provided by an embodiment of the present invention;

Fig. 3 is a structural diagram of a collection device of a threat intelligence source provided by an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection of the present invention. scope.

As mentioned above, the existing collection methods of threat intelligence sources are mainly to artificially search and collect threat intelligence sources, which will undoubtedly affect the collection efficiency of threat intelligence sources.

In order to solve the above technical problems, the inventor may consider automatically obtaining the source address of each target threat intelligence source. However, the quality of each target threat intelligence source obtained automatically is uneven. Based on the source address of each target threat intelligence source, the The threat intelligence in each target threat intelligence source is detected and evaluated, and the source address and corresponding evaluation results of each target threat intelligence source are reported, so as to complete the automatic search and collection of threat intelligence sources.

The specific implementation of the above idea is described below.

Please refer to FIG. 1, an embodiment of the present invention provides a method for collecting threat intelligence sources, the method includes:

Step 100: Obtain the source address of each target threat intelligence source;

Step 102: Based on the source address of the target threat intelligence source, detect and evaluate the threat intelligence in the target threat intelligence source, so as to obtain the evaluation results of each target threat intelligence source;

Step 104: Report the source addresses and evaluation results of each target threat intelligence source.

In the embodiment of the present invention, firstly, the source address of each target threat intelligence source is automatically acquired, and based on the acquired source address of each target threat intelligence source, the threat intelligence in each target threat intelligence source is detected and evaluated, so as to Obtain the evaluation result of each target threat intelligence source; finally, report the source address and corresponding evaluation result of each target threat intelligence source, so as to automatically collect the target threat intelligence source, which can enhance the collection ability of threat intelligence sources , to further increase the collection of threat intelligence.

The execution manner of each step shown in FIG. 1 is described below.

For step 100:

In some implementations, the manner of obtaining the source address of the target threat intelligence source includes a first manner based on known threat intelligence sources and a second manner based on the GitHub platform.

It should be noted that known threat intelligence sources contain several threat intelligence sources and their attribution website links. These attribution website links have not been collected, so the source of the target threat intelligence source can be obtained based on known threat intelligence sources address. GitHub is a hosting platform for open source and private software projects. The platform may contain many threat intelligence warehouses. Therefore, the source address of the target threat intelligence source can also be obtained from the GitHub platform. Through the first acquisition method based on known threat intelligence sources and the second acquisition method based on the GitHub platform, the number of source addresses of target threat intelligence sources can be obtained, and the collection capability of target threat intelligence sources can be enhanced to further expand the threat intelligence library. Storage of threat intelligence in China.

In some implementation manners, the first obtaining method is to obtain the source address of each target threat intelligence source in the following manner:

Obtain a list of source addresses of known threat intelligence sources;

For each known threat intelligence source in the list of source addresses, execute:

Access the source address of currently known threat intelligence feeds;

Obtain the website link in the access page of the current known threat intelligence source, so as to determine the source address of the target threat intelligence source corresponding to the current known threat intelligence source.

In this embodiment, first obtain the source address list of known threat intelligence sources, use crawler technology to visit the source addresses of each known threat intelligence source in turn, and assign the attribution in the access page of each known threat intelligence source Website links are retained, so as to determine each retained website link as the source address of the target threat intelligence source.

In some embodiments, after the step "obtaining the website link in the access page of the current known threat intelligence source" and before the step "determining the source address of the target threat intelligence source corresponding to the current known threat intelligence source", it also includes :

Obtain the domain name of the current known threat intelligence source based on the source address of the current known threat intelligence source;

Determine whether each website link in the access page of the current known threat intelligence source is the same as the domain name of the current known threat intelligence source;

A website link different from the domain name of the current known threat intelligence source is determined as the source address of the target threat intelligence source corresponding to the current known threat intelligence source.

In this embodiment, since the website link contained in the known threat intelligence source is the attribution of the threat information in the known threat intelligence source, then the website link with the same domain name as the known threat intelligence source among these website links Most of them are explanations of the threat intelligence in the known threat intelligence source, and the threat intelligence in the known threat intelligence source has been included in the threat intelligence library, so it is necessary to combine these with the known threat intelligence source Website links with the same domain name are excluded, so that many worthless threat intelligence sources can be screened out.

In some implementation manners, the second obtaining method is to obtain the source address of each target threat intelligence source in the following manner:

Identify multiple keywords to use in searching for targeted threat intelligence feeds;

Search for each keyword in the GitHub platform, and determine each warehouse address obtained from the search as the source address of the target threat intelligence source.

In this embodiment, "IOC", "Blacklist", "Intelligence" and "Malicious" can be used as keywords, and each keyword can be recursively searched on the GitHub platform, and each searched warehouse address can be determined as the target The source address of the threat intelligence source, and clone the files in these warehouse addresses to the local, so as to obtain the files in these warehouse addresses conveniently and quickly.

For step 102:

In order to improve the collectability of each target threat intelligence source automatically obtained through step 100, the threat intelligence in each target threat intelligence source can be detected and evaluated based on the source address of each target threat intelligence source, so as to obtain each target Assessment report for threat intelligence feeds.

In some implementations, step 102 may include:

For each targeted threat intelligence feed, execute:

Based on the source address of the current target threat intelligence source, detect the amount of threat intelligence in the current target threat intelligence source;

Based on the type of each threat intelligence detected in the current target threat intelligence source, determine whether the format of each threat intelligence meets the requirements;

Determine whether each threat intelligence in the current target threat intelligence source has been included in the pre-built threat intelligence library;

Get an assessment of the current target threat intelligence feed.

In this embodiment, since the target threat intelligence source obtained in step 100 may contain blogs, there will be threat intelligence in these blogs, but the number is small, and they belong to news blogs, so it is necessary to analyze the threats contained in each threat intelligence source. The amount of intelligence to detect. In addition, threat intelligence contains multiple types, such as IP type, domain name type, URL type (link type) and fingerprint type, etc. This embodiment can detect the type of each threat intelligence in the target threat intelligence source, and Based on the type of each threat intelligence, it is judged whether the format of each threat intelligence meets the requirements. In addition, it is also necessary to determine whether each threat intelligence in the target threat intelligence source has been included in the threat intelligence library. So far, the evaluation results of each target threat intelligence source can be obtained.

In some embodiments, after the step of "determining whether each threat intelligence in the current target threat intelligence source has been included in the pre-built threat intelligence library", before the step of "obtaining the evaluation result of the current target threat intelligence source" ,Also includes:

For each threat intelligence in the currently targeted threat intelligence feed, execute:

If the current threat intelligence has been included in the threat intelligence database, the threat type of the current threat intelligence is obtained from the threat intelligence database, so as to add the threat type of the current threat intelligence to the evaluation result of the current target threat intelligence source.

In this embodiment, in order to obtain a more comprehensive evaluation result, the threat type of the threat intelligence that has been included in the threat intelligence library can also be retrieved from the threat intelligence library and added to the corresponding evaluation result, so that The evaluation result contains the threat type of the collected threat intelligence, which is convenient for the collectors in step 104 to evaluate and judge the target threat intelligence source.

It should be noted that the threat types in this embodiment can be divided into three categories: threat, no threat, and unknown, and may also contain more detailed threat type levels, so the threat types are not specifically limited here.

For step 104:

In the embodiment of the present invention, the source address and evaluation results of each target threat intelligence source are sent to the mailbox of the threat intelligence collector in the form of email, and the threat intelligence in each target threat intelligence source is further analyzed by the collector. research and collection work.

As shown in FIG. 2 and FIG. 3 , an embodiment of the present invention provides a threat intelligence source collection device. The device embodiments can be implemented by software, or by hardware or a combination of software and hardware. From the perspective of hardware, as shown in Figure 2, it is a hardware architecture diagram of an electronic device where a threat intelligence source collection device provided by an embodiment of the present invention is located, except for the processor, memory, and network interface shown in Figure 2 , and non-volatile memory, the electronic device where the device in the embodiment is located may also generally include other hardware, such as a forwarding chip responsible for processing messages, and the like. Taking software implementation as an example, as shown in Figure 3, as a device in a logical sense, it is formed by reading the corresponding computer program in the non-volatile memory into the memory and running it through the CPU of the electronic device where it is located.

As shown in Figure 3, a threat intelligence source collection device provided in this embodiment includes:

An obtaining unit 301, configured to obtain the source address of each target threat intelligence source;

The evaluation unit 302 is configured to detect and evaluate the threat intelligence in the target threat intelligence source based on the source address of the target threat intelligence source, so as to obtain the evaluation results of each target threat intelligence source;

The reporting unit 303 is configured to report the source address and evaluation result of each target threat intelligence source.

In an embodiment of the present invention, the acquisition means of the source address of the target threat intelligence source in the acquisition unit 301 includes a first acquisition manner based on known threat intelligence sources and a second acquisition manner based on the GitHub platform.

In an embodiment of the present invention, in the obtaining unit 301, the first obtaining method is to obtain the source address of each target threat intelligence source in the following manner:

Obtain a list of source addresses of known threat intelligence sources;

For each known threat intelligence source in the list of source addresses, execute:

Access the source address of currently known threat intelligence feeds;

Obtain the website link in the access page of the current known threat intelligence source, so as to determine the source address of the target threat intelligence source corresponding to the current known threat intelligence source.

In an embodiment of the present invention, after the acquisition unit 301 acquires the website link in the access page of the current known threat intelligence source, before performing the determination of the source address of the target threat intelligence source corresponding to the current known threat intelligence source, Also used for:

Obtain the domain name of the current known threat intelligence source based on the source address of the current known threat intelligence source;

Determine whether each website link in the access page of the current known threat intelligence source is the same as the domain name of the current known threat intelligence source;

A website link different from the domain name of the current known threat intelligence source is determined as the source address of the target threat intelligence source corresponding to the current known threat intelligence source.

In an embodiment of the present invention, in the obtaining unit 301, the second obtaining method is to obtain the source address of each target threat intelligence source in the following manner:

Identify multiple keywords to use in searching for targeted threat intelligence feeds;

Search for each keyword in the GitHub platform, and determine each warehouse address obtained from the search as the source address of the target threat intelligence source.

In one embodiment of the present invention, the evaluation unit 302 is used to perform:

For each targeted threat intelligence feed, execute:

Based on the source address of the current target threat intelligence source, detect the amount of threat intelligence in the current target threat intelligence source;

Based on the type of each threat intelligence detected in the current target threat intelligence source, determine whether the format of each threat intelligence meets the requirements;

Determine whether each threat intelligence in the current target threat intelligence source has been included in the pre-built threat intelligence library;

Get an assessment of the current target threat intelligence feed.

In one embodiment of the present invention, the evaluation unit 302, after executing the judgment whether each threat intelligence in the current target threat intelligence source has been included in the pre-built threat intelligence library, executes the process of obtaining the current target threat intelligence source Before evaluating the results, also include:

For each threat intelligence in the currently targeted threat intelligence feed, execute:

If the current threat intelligence has been included in the threat intelligence database, the threat type of the current threat intelligence is obtained from the threat intelligence database, so as to add the threat type of the current threat intelligence to the evaluation result of the current target threat intelligence source.

It can be understood that the structure shown in the embodiment of the present invention does not constitute a specific limitation on a threat intelligence source collection device. In some other embodiments of the present invention, an apparatus for collecting threat intelligence sources may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange different components. The illustrated components may be realized in hardware, software, or a combination of software and hardware.

The information interaction and execution process among the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific content can refer to the description in the method embodiment of the present invention, and will not be repeated here.

An embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, a threat in any embodiment of the present invention is realized. Methods of collecting information sources.

The embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes any implementation of the present invention. A collection method of threat intelligence source in the example.

Specifically, a system or device equipped with a storage medium may be provided, on which a software program code for realizing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, so the program code and the storage medium storing the program code constitute a part of the present invention.

Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), Tape, non-volatile memory card, and ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

In addition, it should be clear that not only by executing the program codes read by the computer, but also by making the operating system on the computer complete some or all of the actual operations through instructions based on the program codes, so as to implement the above-mentioned embodiments. function of any one of the embodiments.

In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or written into the memory provided in the expansion module connected to the computer, and then based on the program code The instruction causes the CPU installed on the expansion board or the expansion module to perform some or all of the actual operations, thereby realizing the functions of any one of the above-mentioned embodiments.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or sequence. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (10)

1. A method for collecting threat intelligence sources, comprising: Obtain the source address of each target threat intelligence source; Based on the source address of the target threat intelligence source, detect and evaluate the threat intelligence in the target threat intelligence source, so as to obtain the evaluation result of each target threat intelligence source; Reporting the source address of each target threat intelligence source and the evaluation result. 2. The method according to claim 1, wherein the acquisition method of the source address of the target threat intelligence source includes a first acquisition method based on known threat intelligence sources and a second acquisition method based on the GitHub platform. 3. The method according to claim 2, wherein the first obtaining method is to obtain the source address of each target threat intelligence source in the following manner: Obtain a list of source addresses of known threat intelligence sources; For each known threat intelligence source in the source address list, execute: Access the source address of currently known threat intelligence feeds; Obtain the website link in the access page of the current known threat intelligence source, so as to determine the source address of the target threat intelligence source corresponding to the current known threat intelligence source. 4. The method according to claim 3, wherein after said obtaining the website link in the access page of the current known threat intelligence source, after said determining the target threat intelligence source corresponding to the current known threat intelligence source Before the source address of the , also include: Obtain the domain name of the current known threat intelligence source based on the source address of the current known threat intelligence source; Determine whether each of the website links in the access page of the current known threat intelligence source is the same as the domain name of the current known threat intelligence source; A website link different from the domain name of the current known threat intelligence source is determined as the source address of the target threat intelligence source corresponding to the current known threat intelligence source. 5. The method according to claim 2, wherein the second obtaining method is to obtain the source address of each target threat intelligence source in the following manner: Identify multiple keywords to use in searching for targeted threat intelligence feeds; Search for each of the keywords on the GitHub platform, and determine each warehouse address obtained from the search as the source address of the target threat intelligence source. 6. The method according to any one of claims 2-5, wherein the threat intelligence in the target threat intelligence source is detected and evaluated based on the source address of the target threat intelligence source, to An assessment of each of the stated targeted threat intelligence sources was obtained, including: For each of the targeted threat intelligence feeds described, do: Based on the source address of the current target threat intelligence source, detect the amount of threat intelligence in the current target threat intelligence source; Based on the type of each threat intelligence detected in the current target threat intelligence source, determine whether the format of each threat intelligence meets the requirements; Determine whether each threat intelligence in the current target threat intelligence source has been included in the pre-built threat intelligence library; Get an assessment of the current target threat intelligence feed. 7. The method according to claim 6, characterized in that, after said judging whether each threat intelligence source in the current target threat intelligence source has been included in a pre-built threat intelligence library, after said obtaining the current target Before assessing the results of threat intelligence feeds, also include: For each threat intelligence in the currently targeted threat intelligence feed, execute: If the current threat intelligence has been included in the threat intelligence database, the threat type of the current threat intelligence is acquired from the threat intelligence database, so as to add the threat type of the current threat intelligence to the evaluation result of the current target threat intelligence source. 8. A collection device for threat intelligence sources, characterized in that it comprises: an obtaining unit, configured to obtain the source address of each target threat intelligence source; An evaluation unit, configured to detect and evaluate threat intelligence in the target threat intelligence source based on the source address of the target threat intelligence source, so as to obtain an evaluation result corresponding to the target threat intelligence source; A reporting unit, configured to report the source address corresponding to the target threat intelligence source and the evaluation result. 9. An electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1-7 is implemented. 10. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed in a computer, it causes the computer to execute the method according to any one of claims 1-7.

Images