KR20210083936A - System for collecting cyber threat information - Google Patents

Abstract

Provided is a cyber threat information collection system including: an information collection part monitoring a plurality of collection channels providing cyber threat information in real time, and collecting a situation propagation text, software vulnerability information, invasion incident information, malignant code information, hacking domain information, zombie IP sent email information, a vulnerability analysis report, a malignant code analysis report and an invasion analysis report; and an integrated management part extracting an invasion index including an IP, a domain, a hash, an email, and a security vulnerability code by processing the variety of information collected through the information collection part, in a natural language, and classifying types of the variety of information collected through the information collection part, based on the invasion index. Therefore, the present invention is capable of flexibly handling a newly added or deleted collection channel.

Description

Cyber threat information collection system {SYSTEM FOR COLLECTING CYBER THREAT INFORMATION}

The present invention relates to a cyber threat information collection system, and more particularly, to a cyber threat information collection system using crawling and API.

Cyber threat information (CTI) is data that is being used as an indicator to identify potential malicious activity in the threat of intrusion incidents that are constantly increasing and becoming more intelligent.

Cyber Threat Intelligence means to collect and analyze threats from external as well as internal organizations, and integrate and apply them to internal security systems. The current domestic and foreign cyber threats are continuously increasing and becoming more intelligent, and they are developing into systematic and systematic threats.

Accordingly, there is a need for a technology capable of stably collecting data collected from various collection channels and flexibly responding to newly added or deleted collection channels in order to respond to intelligent cyber threats.

An object of the present invention is to provide a cyber threat information collection system capable of stably collecting data collected from various collection channels for intelligent cyber threat response and flexibly responding to newly added or deleted collection channels.

According to one embodiment, a system for collecting cyber threat information is provided. The cyber threat information collection system monitors a plurality of collection channels providing the cyber threat information in real time, and from the plurality of collection channels, situation propagation message, software vulnerability information, infringement accident information, malicious code information, hacking domain information, malicious code An information collection unit that collects domain information, zombie IP sending email information, vulnerability analysis report, malicious code analysis report, and infringement accident analysis report, and natural language processing of various information collected through the information collection unit to obtain IP, domain, hash, and an integrated management unit for extracting a breach index including an email and a security vulnerability code, and classifying the types of information collected through the information collection unit based on the breach index.

The information collection unit collects data from a collection channel that provides cyber threat information without input data, a general collection module that collects data, and a collection channel that provides cyber threat information related to input data when input data is received. It may include a recursive query collection module.

The integrated management unit extracts text from various information collected through the information collection unit, separates the text into sentence units, extracts a token from the separated sentence, performs text normalization using the token, and A preprocessing module for allocating part-of-speech information to a token, an entity name recognition module for classifying proper nouns based on the token, and cross-referencing analysis to determine whether a word is the same entity by comparing preceding words and phrases with current words and phrases, It may include a cross-reference analysis module for performing dependency analysis between the two.

The integrated management unit extracts the breach index, attack tactics and tools, attack techniques and attack procedures (TTPs) through the cross-reference analysis and dependency analysis, and the intrusion index and the attack tactics and tools, attack techniques and attacks It may further include a semantic analysis module for extracting metadata describing the procedure.

The integrated management unit manages the plurality of collection channels based on an interface management module for managing network communication, internal and external interfaces, and a pre-stored management table model, and assigns preset environment settings to the plurality of collection channels It may include a database model management module that

The integrated management unit, the collection channel control module for transmitting the total collection channel status registered in the integrated management unit, the currently available collection channel status, the total idle collection channel status, and whether the task of the collection channel to which the task is assigned has been completed to the user terminal may further include.

The integrated management unit may further include a scheduler for adding or deleting a schedule of a collection channel to a pre-stored scheduling event.

The integrated management unit may further include a process management module that initializes process information registered in the integrated management unit, collects errors occurring in the plurality of collection channels, and stores the error information in a database.

The interface management module may transmit statistics on the information collection amount for each period, resource, or channel to the user terminal.

Data collected from various collection channels can be reliably collected to respond to advanced cyber threats, and it can respond flexibly to newly added or deleted collection channels.

1 is a block diagram of a cyber threat information collection system according to an embodiment.
2 is a block diagram of an information collection unit of a cyber threat information collection system according to an embodiment.
3 is a block diagram of an integrated management unit of a cyber threat information collection system according to an exemplary embodiment.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

1 is a block diagram of a cyber threat information collection system according to an embodiment. 2 is a block diagram of an information collection unit of a cyber threat information collection system according to an embodiment. 3 is a block diagram of an integrated management unit of a cyber threat information collection system according to an exemplary embodiment.

Referring to FIG. 1 , the cyber threat information collection system 100 according to an embodiment includes an information collection unit 110 , an integrated management unit 120 , and a database 130 .

The information collection unit 110 monitors a plurality of collection channels 10 providing cyber threat information (CTI) in real time, and a situational message, software vulnerability information, and infringement incident from the plurality of collection channels 10 . Information, malicious code information, hacking domain information, malicious domain information, zombie IP sending email information, vulnerability analysis report, malicious code analysis report, breach analysis report, and open source information (Open Source Intelligence, OSINT) are collected.

The plurality of collection channels 10 are, as an embodiment, MC Finder (Malicious Code-Finder), KISA Internet Infringement Response Center (KISC), Expoit-DB, US Vulnerability Sharing Center (NVD), OTX (Open Threat Exchange), It may include at least one of Malware Information Sharing Platform & Threat Sharing (MISP), C-share, DNSBL, VirusShare, zone-h.org, malwaredomainlist.com, Threat Crowd, Cymon, EmergingThreats, Talos Intelligence, bambenk.

As an embodiment, the information collection unit 110 may parse and store information provided by the MC finder, collect, parse, and store software (S/W) vulnerability information from Expoit-DB, and may collect, parse and store software (S/W) vulnerability information from NVD. S/W) Vulnerability information and Security Vulnerability Joint Assessment System (CVSS) information can be parsed and stored, OTX-based intrusion incident information can be collected from OTX, MISP-based incident information can be collected from MISP, and Export Using API, you can collect, parse, and store intrusion incident information from C-share, collect and store single malicious code and a large amount of malicious code from VirusShare, and collect and parse hacking domain information from zone-h.org It can collect, parse, and store malicious domain information from malwaredomainlist.com, collect and store zombie IP sending email information from a zombie IP and botnet group, and collect and store malicious code hash, domain, IP, and password information from Threat Crowd. You can collect information provided as a feed, etc., collect blacklist IP information and related IP information from Cymon, collect blacklist IP information from EmergingThreats, Talos Vulnerability analysis reports can be collected from Intelligence, and C&C IP and domain information can be collected from bambenk.

As an embodiment, the information collection unit 110 may collect IP and domain information related to the inquiry target domain from DNS and PTR (it may inquire a DNS record), and obtain registration information for the inquiry target domain from whois. It is possible to collect static behavior analysis for malicious code from the malicious code static behavior analysis channel, and collect analysis request results from the malicious code static behavior analysis channel, and search similar malicious codes through linking with the malicious code similarity analysis system. You can request analysis of similar malicious code from the channel, collect analysis request results from the similar malicious code inquiry channel, search for malicious code based on the malicious code hash value in VirusTotal, and collect related information. , IP, domain-based related information can be collected.

The information collection unit 110 collects information in a plurality of collection channels by utilizing a crawling technology and API. The information collection unit 110 performs a task target and order selection (Indexing) step, a task creation (Payload) step, and a collection execution (Parsing) step.

The information collection unit 110 detects the work progress rate (based on the unit set in Events) for data collection in the task target and order selection (Indexing) step, and performs distributed processing according to the amount of work to be collected. In the operation target and order selection (indexing) step, the operation target may be a URL of a web page to be collected, pagination and line number of a table in the web page, a specific data file path, and the like.

The information collection unit 110 divides the indexing generated in the task creation (Payload) step, the task target and the order selection (Indexing) step according to the amount of work.

The information collection unit 110 parses the actual data through a defined collection engine (eg, crawling) in the collection parsing step. The information collection unit 110 allocates and processes a previously generated job as a job target created by sub-indexing in the job creation process. The information collection unit 110 records the error in the payload and indexing that was in the work when a work error occurs. The information collection unit 110 generates Events, Correlation, and Meta information object (JSON) like one Event every time it is stored in the database 130 in an Indexing unit (Events unit), stores one Event, and stores it in the database. (Mongodb, bson)

Referring to FIG. 2 , the information collection unit 110 may include a general collection module 111 and a recursive inquiry collection module 1120 .

The general collection module 111 collects data from a collection channel that provides cyber threat information without input data.

As an embodiment, the general collection module 111 may periodically or permanently collect data in consideration of the characteristics of a collection target channel. The general collection module 111 is an embodiment, and may control whether parallel processing is performed and a work cycle according to a user's settings.

When input data is received, the recursive inquiry collection module 112 collects data from a collection channel that provides cyber threat information related to the input data.

As an embodiment, the recursive inquiry collection module 112 may periodically or permanently collect data in consideration of the characteristics of a collection target channel. As an embodiment, the recursive inquiry collection module 112 may control whether to perform parallel processing and a work cycle according to a user's settings. As an embodiment, the recursive inquiry collection module 112 may adjust the collection period when there is a restriction (eg, limit on the number of inquiries per day, etc.) according to the collection channel.

The integrated management unit 120 extracts an Indicator of Compromise (IoC) including IP, domain, hash, e-mail, and security vulnerability code by natural language processing of various information collected through the information collection unit 110, and infringes The types of various types of information collected through the information collection unit 110 are classified based on the index.

In order to configure the expanded cyber threat information database (Mongodb), the integrated management unit 120 defines detailed classification according to data types based on the breach index.

The integrated management unit 120 regards the cyber threat information collected through OSINT as a past history. The reason cyber threat information is viewed as a past history is that it is difficult to be published in real time on the OSINT site from the point of view of the actual incident.

Past history information (Events) may be expressed in units of bundles or groups of IoC data collected at the same time through the same OSINT site in the system. As an embodiment, when specific table information is crawled on the Web, one row of the corresponding table may be generated on an event basis.

Table 1 shows the information collection.

'event_id' is used for management purposes in the database 130 to store the corresponding past history information (Events), and is uniquely used based on string information of each data. 'ctime' and 'dtime' are date/time data related to data collection. For data in the form of time/date without UTC notation, the local time zone of the server is applied as the domestic standard time (UTC+09:00) through the Whois query. 'channel' and 'origin' are classifications for OSINT sites, and channels can be classified based on the OSINT site domain name. An origin may be defined by classifying according to a collection method and an event type within the same channel. 'ioc_list' is ID information about IoC collected through a corresponding event, and since a user starts a search within the IoC data category when a user queries the database 130, the corresponding information may be included for expansion and convenience. Data with the same type and value are continuously referred to as the same 'collect_id' through the 'ioc_list' key value in the document in Events, and the document corresponding to 'resource' among the values of 'label' (in this case, mainly the search target resource ), you can check the past collection history for a specific resource through the number of times it is referenced.

Table 2 shows an example of major resource and attribute classification.

The main criterion for classifying the main resource and the attribute is the criterion for whether the resource can be viewed as malicious, such as a violation index, and the meta collection storage structure can be defined in the form shown in Table 3 on the database 130. can

For example, the integrated management unit 120 stores the data type as 'ip' when a string representing each data in the bundle of past history information (Events) is IP, and stores an arbitrary IP '192.168.0.1 In the case of string data in the form of ', it can be stored as a value.

The main purpose of the cyber threat information database storage structure is to store string values of resources (Meta) and to go through various past histories (Events) as well as past histories (Events) of a specific IoC. It is to create a relationship with a new IoC that was not in the previous one. For this, it is necessary to express the relationship between IoCs, and the storage structure can be defined as shown in Table 4.

Referring to FIG. 3 , the integrated management unit 120 may include a preprocessing module 121 , an entity name recognition module 122 , a cross-reference analysis module 123 , and a semantic analysis module 124 , and the integrated management unit ( 120 may perform natural language processing through the preprocessing module 121 , the entity name recognition module 122 , the cross-reference analysis module 123 , and the semantic analysis module 124 .

The preprocessing module 121 extracts text from various information collected through the information collection unit 110, separates the text into sentence units, extracts a token from the separated sentence, and performs text normalization using the token, , allocating part-of-speech information to the token.

Specifically, the pre-processing module 121 extracts text from document files (eg, PDF, DOC, HWP) collected in various forms, separates the extracted text into sentence units, and tokens to facilitate analysis of the separated sentences. Extracts in units, analyzes tokens in a general form, performs text normalization to increase the efficiency of analysis by reducing the number of words, and allocates part-of-speech information to tokens using algorithms such as Decision Tree, Hidden Markov Models, and Support Vector Machines .

The entity name recognition module 122 classifies proper nouns such as a person's name and a place name based on the token.

The cross-reference analysis module 123 compares preceding words and phrases with current words and phrases to perform cross-reference analysis to determine whether they are the same entity, and performs dependency analysis between words.

The semantic analysis module 124 extracts indices of compromise (IoC), attack tactics and tools, attack techniques and attack procedures (TTPs) through cross-reference analysis and dependency analysis, and extracts indices and attack tactics and tools, attack techniques and Extracts meta data describing the attack procedure (eg time value, attack group name, description, etc.). The semantic analysis module 124 generates an attack pattern by connecting the extracted breach index (IoC), attack tactics and tools, attack techniques and attack procedures (TTPs), and metadata.

The integrated management unit 120 may include an interface management module 125 , a database model management module 126 , a collection channel control module 127 , a scheduler 128 , and a process management module 129 as an embodiment. have.

The interface management module 125 manages network communication, internal and external interfaces.

The interface management module 125 transmits statistics on the amount of information collected by period, resource, or channel to the user terminal 300 .

The database model management module 126 manages the plurality of collection channels 10 based on the previously stored management table model, and allocates preset environment settings to the plurality of collection channels 10 . The environment setting items may be preset by the user, and the integrated management unit 120 receives the preset environment setting items from the user terminal 300 .

The collection channel control module 127 displays the total collection channel status registered in the integrated management unit 120, the currently available collection channel status, the total idle collection channel status, and whether the job of the collection channel to which the job is assigned has been completed, the user terminal 300 ) is sent to

The scheduler 128 adds or deletes the schedule of the collection channel to the pre-stored scheduling event.

The process management module 129 initializes process information registered in the integrated management unit 120 , collects errors occurring in the plurality of collection channels 10 , and stores the error information in the database 130 .

As an embodiment, the database 130 may store various types of information collected through the information collection unit 110 , cyber threat information classified through the integrated management unit 120 , and a management table model.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improved forms of the present invention are also provided by those skilled in the art using the basic concept of the present invention as defined in the following claims. is within the scope of the right.

Claims (9)

A system for collecting cyber threat information, comprising:
A plurality of collection channels providing the cyber threat information are monitored in real time, and situation propagation messages, software vulnerability information, infringement accident information, malicious code information, hacking domain information, malicious domain information, and zombie IP are sent from the plurality of collection channels. an information collection unit that collects information, vulnerability analysis reports, malicious code analysis reports, and incident analysis reports; and
Natural language processing of various information collected through the information collection unit extracts a breach index including IP, domain, hash, email, and security vulnerability code, and the types of information collected through the information collection unit based on the breach index Integrated management department to classify
A cyber threat information collection system comprising a. In claim 1,
The information collection unit,
A general collection module that collects data from a collection channel that provides cyber threat information without input data, and
A cyber threat information collection system, comprising a recursive inquiry collection module for collecting data from a collection channel that provides cyber threat information related to the input data when input data is received. In claim 1,
The integrated management unit,
Extracts text from various pieces of information collected through the information collection unit, separates the text into sentence units, extracts a token from the separated sentence, performs text normalization using the token, and adds part-of-speech information to the token. Allocating preprocessing module,
An entity name recognition module for classifying proper nouns based on the token, and
A cyber threat information collection system comprising a cross-reference analysis module that compares preceding words and phrases with current words and phrases to determine whether they are the same entity, and performs dependency analysis between words. In claim 3,
The integrated management unit,
The intrusion index, attack tactics and tools, attack techniques and attack procedures (TTPs) are extracted through the cross-reference analysis and dependency analysis, and meta describing the intrusion index and the attack tactics and tools, attack techniques and attack procedures A cyber threat information collection system further comprising a semantic analysis module for extracting data. In claim 1,
The integrated management unit,
an interface management module for managing network communications, internal and external interfaces, and
and a database model management module for managing the plurality of collection channels based on a pre-stored management table model and allocating preset environment settings to the plurality of collection channels. In claim 5,
The integrated management unit,
Further comprising a collection channel control module for transmitting the total collection channel status registered in the integrated management unit, the currently available collection channel status, the total idle collection channel status, and whether the job of the collection channel to which the job is assigned has been completed, to the user terminal; Cyber threat information collection system. In claim 6,
The integrated management unit,
A cyber threat information collection system, further comprising a scheduler for adding or deleting a schedule of a collection channel to a pre-stored scheduling event. In claim 7,
The integrated management unit,
and a process management module for initializing process information registered in the integrated management unit, collecting errors occurring in the plurality of collection channels, and storing the error information in a database. In claim 5,
The interface management module,
A cyber threat information collection system that transmits statistics on the amount of information collected by period, resource, or channel to a user terminal.

Images