KR20220158533A - Malicious site detection method - Google Patents

Abstract

Disclosed is a method for detecting a malicious site. The method for detecting a malicious site according to the present invention comprises the steps of: collecting a training data set including a plurality of types of training data, and extracting a feature from the training data set; generating an artificial intelligence model by training an artificial neural network to classify normal and malignity using the feature as input data; collecting web traffic in real time; obtaining a plurality of types of web collection data by using the web traffic, and extracting a feature from the plurality of types of web collection data; and providing the feature extracted from the web collection data to the artificial intelligence model, and determining whether the web traffic is harmful based on a result value output by the artificial intelligence model. Accordingly, a suspected malicious domain can be automatically classified through the artificial intelligence model.

Description

Malicious site detection method {MALICIOUS SITE DETECTION METHOD}

The present invention relates to a method for detecting malicious sites, which is capable of obtaining whether or not harmful to web traffic is obtained by extracting features from web collected data and inputting the extracted features to an artificial intelligence model.

In general, a harmful site detection method is an event analysis method of a security device or security system, and can be said to be an analysis based on a pattern or threshold matching a predefined security policy when an event occurs.

And in these technologies, it is difficult to preemptively detect malicious sites before a security incident occurs, and it is difficult to detect malicious sites based on web packets and web site information without file analysis.

The present invention is intended to solve the above problems, and an object of the present invention is to extract features from web collected data and then input the extracted features to an artificial intelligence model to obtain whether web traffic is harmful or not, a malicious site. It is to provide a detection method.

A malicious site detection method according to the present invention includes the steps of collecting a training data set including a plurality of types of training data, extracting a feature from the training data set, and using the feature as input data. and generating an artificial intelligence model by training an artificial neural network to classify normal and malicious data, collecting web traffic in real time, obtaining the plurality of types of web collection data using the web traffic, and Extracting a feature from web collected data of a type, and providing the feature extracted from the web collected data to the artificial intelligence model, based on the result value output by the artificial intelligence model and determining whether the web traffic is harmful.

In this case, the plurality of types of web collection data may include URL, URI, HTML, JavaScript, and host.

In this case, the characteristics extracted from the plurality of types of web collected data include length information of components constituting the web collected data, information on the number of the components, whether the components are in a specific language, and whether the components are specific It may include the number representing the shape, the entropy of the component, the number of tags, whether a specific value exists, whether a specific word exists, and the number of functions.

Meanwhile, when the web traffic is collected, determining whether the collected web traffic is a feature extraction target may be further included.

In this case, the step of determining whether the collected web traffic is a feature extraction target may include the collected web traffic based on whether a file exists, whether it corresponds to a whitelisted domain, the type of a response code, and whether it is an HTML document. It is possible to determine whether a feature is to be extracted.

According to the present invention, there is an advantage in that web traffic is collected in real time, features are extracted so that AI analysis is possible, and suspected malicious domains can be automatically classified through an AI model.

In addition, according to the present invention, even when file download is impossible, there is an advantage in that a malicious site can be preemptively and quickly detected only with web packets or web site data collected at the network level.

Harmful site information detected in this way can be applied to security services and security devices to prevent cyber security threats so that threat information can be preemptively responded to, and threats can be removed in advance through linkage with the security platform. You can make sure that people can use the Internet service safely. In addition, new value can be created in the information security business through the present invention.

1 is a block diagram illustrating an apparatus for detecting malicious sites according to the present invention.
2 is a diagram for explaining a feature extraction method according to the present invention.
3 is a diagram for explaining in detail a feature extraction method according to the present invention.
4 is a diagram for explaining a method of checking whether web traffic is normal in real time using web traffic according to the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In implementing the present invention, components may be subdivided for convenience of description, but these components may be implemented in one device or module, or one component may be divided into multiple devices or modules may be implemented in

1 is a block diagram illustrating an apparatus for detecting malicious sites according to the present invention.

The malicious site detection device 100 according to the present invention may include a collection unit 110 , a control unit 120 and a memory 130 .

The collection unit 110 includes a communication circuit or communication module for communicating with an external device using a wired/wireless communication technology, and may transmit/receive data with the external device.

In detail, the collecting unit 110 is connected to a network end and can collect web traffic passing through the network end in real time.

In addition, the collection unit 110 may collect data by being connected to a phishing site, Korea Internet & Security Agency (KISA), or a telecommunications company.

The memory 130 may store programs or other data for the operation of the malicious site detection device.

The controller 120 may control overall operations of the malicious site detection device 100 .

In addition, the malicious site detection device 100 is equipped with an artificial neural network or artificial intelligence model, and in this case, one or more commands constituting the artificial neural network or artificial intelligence model may be stored in the memory 130 .

2 is a diagram for explaining a feature extraction method according to the present invention.

First, in this specification, data collected to train an artificial neural network is referred to as training data.

The controller 120 may collect a training data set through the collection unit 110 .

Specifically, the training data set may include URI (Uniform Resource Identifier) data. Also, the URI data may include public URI data and non-public URI data.

Public URI data may be data collected from phishing sites (OpenPhish, Phistank, etc.).

In addition, the non-public URI may be data held by a subject (for example, a telecommunications company) that operates the malicious site detection device 100 or held by the Korea Internet & Security Agency (KISA).

Meanwhile, a training set may include a plurality of types of training data. Here, one training set may include URI data and various types of data related to the URI data. For example, multiple types of training data may include URL, URI, HTML, JavaScript, and host. In this case, the control unit 120 may obtain a plurality of types of training data through web crawling of the collected URI data.

Meanwhile, the controller 120 may obtain a correct answer value (label) corresponding to the training data set.

Specifically, when a specific training data set is obtained based on public URI data, the controller 120 may determine that a label corresponding to the specific training data set is malicious.

Also, when a specific training data set is obtained based on non-public URI data held by the Korea Internet & Security Agency (KISA), the controller 120 may determine that a label corresponding to the specific training data set is malicious.

On the other hand, when a specific training data set is obtained based on non-public URI data held by an entity (eg, a telecommunications company) that operates the malicious site detection device 100, the control unit 120 detects normal or malicious data through the SafeBrowsing API. information can be obtained. Also, the controller 120 may associate information on whether the training data set is normal or malicious as a label.

Meanwhile, the controller 120 may extract features from the training data set. In the present invention, 183 features were extracted. And the 183 features consist of 96 URI-based features, 43 HTML-based features, 28 JavaScript-based features, and 16 Host-based features.

3 is a diagram for explaining in detail a feature extraction method according to the present invention.

The control unit 120 may collect public URI data and non-public URI data (S305).

Meanwhile, as described above, it is also possible to first configure a training data set and then label the correct value, or configure the training data set after first labeling the URI data with the correct value.

The controller 120 may label correct values for public URI data and non-public URI data (S310).

Specifically, the control unit 120 may determine whether the collected URI data is data held by an entity (eg, a telecommunication company) that operates the malicious site detection device 100 (S315). And if the collected URI data is data held by a subject (for example, a telecommunications company) that operates the malicious site detection device 100, the control unit 120 calls the google safeBrowsing API (S320), and the corresponding URI data is normal. Information about recognition or malignancy may be obtained.

And, if the corresponding URI data is malicious (S325), the control unit 120 may label the corresponding URI data with an answer value of “malicious” (S330).

On the other hand, if the corresponding URI data is normal (S325), the control unit 120 may label the corresponding URI data with an answer value of “normal” (S335).

On the other hand, if the collected URI data is not data held by an entity (eg, a telecommunications company) that operates the malicious site detection device 100, the control unit 120 may determine whether the collected URI data is public data (S340). .

And if the collected URI data is open data, the control unit 120 may label the corresponding URI data with an answer value of “malicious” (S330).

Meanwhile, if the collected URI data is not open data, the controller 120 may determine whether the collected URI data is data owned by the Korea Internet & Security Agency (S345).

If the collected URI data is data held by the Korea Internet & Security Agency, the control unit 120 may label the corresponding URI data with an answer value of “malicious” (S330).

Meanwhile, the control unit 120 may determine whether a training data set including a URI is a feature extraction target (S350).

Specifically, the controller 120 may determine whether the training data set is a feature extraction target based on whether the file exists, whether it corresponds to a whitelist domain, the type of the response code, and whether it is an HTML document.

More specifically, the control unit 120 cannot acquire a file using URI data (when downloading a file is impossible) (S351), the URI data is not a white list domain (S352), and the form of the response code is 2XX (S353), and if the URI data is an HTML document (S354), features can be extracted from the training data set (S360).

Specific types of features will be described in detail later.

Meanwhile, the controller 120 may train an artificial neural network to classify normal and malignant features by using the features as input data.

Specifically, an artificial neural network (neural network) is a modeling of the operating principle of biological neurons and the connection relationship between neurons, and a number of neurons called nodes or processing elements are connected in the form of a layer structure. It may be an information processing system.

Also, an artificial neural network may be trained using input data. Here, training may refer to a process of determining parameters of an artificial neural network using input data in order to achieve a purpose such as classification, regression analysis, or clustering of input data. can As representative examples of parameters of an artificial neural network, a weight assigned to a synapse or a bias applied to a neuron may be cited.

In addition, the loss function can be used as an index (reference) for determining optimal model parameters in the training process of an artificial neural network. Learning in an artificial neural network means a process of manipulating the parameters (weights, biases, etc.) of the artificial neural network to reduce the loss function, and the purpose of learning can be seen as determining parameters that minimize the loss function.

Also, the controller 120 may train an artificial neural network based on a supervised learning algorithm.

In supervised learning, an artificial neural network is trained under a given label for input data. Here, the label may mean a correct answer (or a result value) to be inferred by the artificial neural network when input data is input to the artificial neural network.

In addition, the control unit 120 provides features extracted from a specific training data set to the artificial neural network as input data, and provides information on normal or malicious labeled in the specific training data set to the artificial neural network as an answer, thereby training the artificial neural network. can do.

Also, the controller 120 may train an artificial neural network based on input data (features) and differences between correct answers. That is, the controller 120 may calculate a loss function based on the difference between the input data (characteristic) and the correct answer, and adjust parameters of the artificial neural network using the calculated loss function.

In addition, the controller 120 may generate an artificial intelligence model in which parameters of the artificial neural network are optimized by repeating training of the artificial neural network using features extracted from various training data sets and correct answers corresponding to the various training data sets. .

4 is a diagram for explaining a method of checking whether web traffic is normal in real time using web traffic according to the present invention.

The controller 120 may be connected to the network end and collect web traffic passing through the network end in real time (S410). Here, web traffic may include at least one of web packets and website data.

Then, the control unit 120 may determine whether the collected web traffic is a feature extraction target (S420).

Specifically, the control unit 120 determines whether the collected web traffic is a target for feature extraction (i.e., normal or malicious) based on whether the file exists, whether it corresponds to a whitelist domain, the type of response code, and whether it is an HTML document. , whether it is an object to be judged using an artificial intelligence model) can be determined (420).

More specifically, the control unit 120 cannot acquire a file using web traffic (if it is impossible to download a file) (S421), the web traffic is not a white list domain (S422), and the form of the response code is 2XX (S423), and if the web traffic is an HTML document (S424), it can be determined that the collected web traffic is a subject for feature extraction (ie, normal or malicious, subject to be determined using an artificial intelligence model) (420).

Meanwhile, if the collected web traffic is a feature extraction target, the control unit 120 may obtain multiple types of web collected data using the web traffic and extract features from the multiple types of web collected data. (S430).

Here, the plurality of types of web collection data may include URL, URI, HTML, JavaScript, and host information, and may be obtained using web packets. Also, the control unit 120 may obtain domain information. For example, the control unit 120 may extract a URI from a web packet and obtain HTML and JavaScript information of a corresponding web page by accessing the extracted URI. For another example, the control unit 120 may extract a URI from a web packet and inquire about the extracted URI to Whois to obtain domain information (domain creation time, elapsed time after domain creation, etc.).

In the present invention, 183 features are exemplified, and the 183 features consist of 96 URI-based features, 43 HTML-based features, 28 JavaScript-based features, and 16 Host-based features. Examples of features are shown in Table 1.

Classification of characteristics Explanation URL url length excluding protocol URL The number of '-' in domain URL number of domain tokens URL length of url path URL length of url filename URL Length of the longest token among domain tokens URL Average token length of domain tokens URL Check whether the tld of the url is a valid tld URL Length of domain (including port number) (including netloc: port number) URL (without port number) Length of url hostname (hostname: without port number) URL Number of dots(.) in url URL Number of underscores(_) in url URL Number of equals(=) in url URL Number of slashes (/) in url (including two 'http://') URL number of dashes (-) in url URL Number of semicolon(;) in url URL The number of at(@) in url URL Number of percent(%) in url URL The number of plus(+) in url URL query length in url URL Number of query parameters in url URL Check if ip(IPv4/IPv6) exists in url URL Check url complexity (shannon-entropy) URL url consonant count check URL Check the number of url numbers (0 to 9) URL url chinese check URL Check if the url port is a valid port (None, 80, 443, 8080, 8443) URL_Path number of path tokens URL_Path Length of longest token among path tokens URL_Path Length of the shortest token among path tokens URL_Path Check the number of consonants in the path URL_Path Average token length of path token URI length of url tld URI [a i e o u] /total characters [a-z] URI consonants /total letters [a-z] URI The number of ldl types in url URI The number of ldl forms in the domain URI The number of ldl types in path URI The number of ldl types in the file name URI The number of ldl types in Arg URI Number of dld types in url URI Number of dld types in domain URI The number of dld types in the path URI The number of dld types in file names URI Number of dld types in Arg URI length of subdirectory URI length of fileName URI length of file extension URI length of arg URI path/url URI arg/url URI arg/domain URI domain/url URI path/domain URI arg/path URI Executable file URI Are you using port 80? URI Ratio of continuity (the sum of the longest values among alphabet, number, and special character types is divided by the length of the chord) URI value of longest variable URI number of digits in hostname URI number of digits in directory URI number of digits in filename URI number of digits in extension URI number of digits in query URI number of characters in url URI number of characters in host URI number of characters in directory URI number of characters in filename URI number of characters in extension URI number of characters in query URI Longest word length among subdirectories URI Longest word length among arguments URI Number of sensitive words in URL URI number of variables in query URI Number of special characters in url URI number of delimiters in domain URI number of delimiters in path URI Number of delimiters in full URL URI Numbers/full characters in url URI number/full character in domain URI Numbers/whole letters in DirectoryName URI Numbers/whole characters in filename URI number/full character in extension URI Numbers/whole characters in AfterPath URI Number of symbols in full URL URI The number of symbols in all domains URI number of symbols in directory name URI number of symbols in filename URI Number of Symbols in File Extension URI number of symbols in afterpath URI entropy of url URI domain entropy URI entropy of directoryname URI entropy of filename URI extension entropy URI afterpath entropy HTML The number of <iFrame> tags in html HTML The number of <script> tags in html HTML The number of <embed> tags in html HTML The number of <object> tags in html HTML The number of <div> tags in html HTML The number of <head> tags in html HTML The number of <body> tags in html HTML The number of <form> tags in html HTML The number of <a> tags in html HTML The number of <small> tags in html HTML The number of <span> tags in html HTML The number of <input> tags in html HTML The number of <applet> tags in html HTML Number of <img> tags in html HTML Number of <video> tags in html HTML The number of <audio> tags in html HTML Number of <meta> tags with 'refresh' attribute in html ex) <meta http-equiv="refresh" … HTML Length of <script>...</script> in html (excluding external links) HTML Number of spaces in html (only space) HTML The number of tags that refer to external addresses in html (when the link does not have its own hostname and the href attribute refers to 'http' or 'ftp') HTML html body document length HTML The number of tags with href,src attributes in script HTML The number of tags in html with the href attribute HTML The number of tokenized words based on space in the html value. HTML Number of tokenized words based on space in html value (remove duplicates) HTML Number of lines in html (remove duplicates) HTML Average word length in html (total word length / total number of words) HTML Check the word "log" in html HTML Check the word "pay" in html HTML Check for the word "free" in html HTML Check the word "access" in html HTML Check the word "bonus" in html HTML Check for the word "click" in html HTML Number of dots(.) in html HTML Number of hyphens(-) in html HTML Number of underscores(_) in html HTML Number of equals(=) in html HTML The number of slashes (/) in html HTML Number of hashes (#) in html HTML number of at(@) in html HTML number of dollars($) in html HTML Check if <title> tag exists in html HTML Check if html is the default webpage (apache, nginx, etc.) Javascript Number of eval() functions in script Javascript The number of setTimeout() functions in script Javascript The number of unescape() functions in script Javascript Number of escape() functions in script Javascript The number of parseInt() functions in script Javascript The number of fromCharCode() functions in script Javascript Number of ActiveXObject() functions in script Javascript The number of concat() functions in script Javascript The number of indexOf() functions in script Javascript The number of substring() functions in script Javascript The number of replace() functions in script Javascript The number of documentaddEventListener() functions in script Javascript The number of createElement() functions in script Javascript The number of getElementById() functions in script Javascript The number of attachEvent() functions in script Javascript The number of documentwrite() functions in script Javascript script document length Javascript Number of deduplicated words in script Javascript Number of lines in script (remove duplicates) Javascript Average word length in script (total word length / total number of words) Javascript Number of dots(.) in script Javascript Number of hyphens(-) in script Javascript Number of underscores(_) in script Javascript Number of equals(=) in script Javascript The number of slashes (/) in script Javascript The number of hashes (#) in script Javascript The number of at(@) in script Javascript The number of dollars($) in script Host period after domain creation (Based on Month) Host Domain remaining expiry date (by month) Host Period after the latest update of domain information (Based on Day) Host Whether update date value exists Host Whether create date value exists Host Existence of expiration date value Host Whether zipcode value exists Host whether the orgraniztion value exists Host whether dnssec is signed Host Whether city value exists Host Whether state value exists Host Whether the country value exists Host Whether to register whois server Host Whether referral url value exists Host Whether email value exists Host Alexa ranking (by 100,000)

As an example of a feature, a feature extracted from a plurality of types of web collected data may include length information of a component constituting the web collected data. For example, the characteristic may include the length of a component (url excluding the protocol) constituting the web collection data of the first type (URL). For another example, the feature may include the length of a component (fileName) constituting the web collection data of the second type (URI).

As another example of a feature, a feature extracted from a plurality of types of web collected data may include information on the number of components constituting the web collected data. For example, the feature may include the number of components (equals(=)) constituting the web collection data of the first type (URL). For another example, the characteristic may include the number of components (words removed from duplicates in script) constituting the web collection data of the fourth type (JavaScript).

As another example of a feature, a feature extracted from a plurality of types of web collected data may include whether a component is in a specific language. For example, the characteristic may include whether or not a component constituting the web collection data of the first type (URL) is Chinese.

As another example of a feature, a feature extracted from a plurality of types of web collected data may include the number of components representing a specific shape. For example, the feature may include the number of elements constituting the web collection data of the second type (URI) indicating the type of did.

As another example of a feature, a feature extracted from a plurality of types of web collected data may include entropy of a component. For example, the characteristic may include the entropy of the domain in the web collection data of the second type (URI).

As another example of a feature, a feature extracted from a plurality of types of web collected data may include the number of tags. For example, the characteristics may include the number of components (<body> tags) constituting the web collection data of the third type (HTML). For another example, the feature may include the number of components (number of tags with href attribute) constituting web collection data of the third type (HTML).

As another example of a feature, a feature extracted from a plurality of types of web collected data may include whether a specific value exists. For example, the characteristic may include whether a specific value (state value) is present in the web collection data of the fifth type (HOST).

As another example of a feature, a feature extracted from a plurality of types of web collected data may include the number of functions. For example, the feature may include the number of eval() functions in web collection data of the fourth type (Javascrip).

As another example of a feature, a feature extracted from a plurality of types of web collected data may include whether a specific word exists. For example, the feature may include whether a specific word (“free”) is present in the web collection data of the third type (HTNL).

Meanwhile, when a feature is extracted, AI analysis based on the extracted feature may be performed (S440).

Specifically, the controller 120 may provide features extracted from web collected data to an artificial intelligence model, and determine whether or not web traffic is harmful based on a result value output from the artificial intelligence model.

More specifically, when a feature extracted from web collection data is input, the artificial intelligence model may output a result value corresponding to the input feature based on the input feature. In this case, the artificial intelligence model may output a class (normal or malicious) corresponding to the input feature and a probability corresponding to the class.

Also, if the probability of maliciousness is output as higher than a preset value (eg, 0.5), the controller 120 may determine that the input feature is malicious (S450). Also, if the input feature is extracted from multiple types of web collected data, it may be determined that the first web traffic used to collect the multiple types of web collected data is traffic to a harmful site. Therefore, the control unit 120 may determine that the web traffic collected in real time is traffic to a harmful site (S460), and transmit a harmful site guide to the corresponding user.

Meanwhile, when the normal probability is output as a value exceeding a preset value (eg, 0.5), the controller 120 may determine that the input feature is normal (S450). Also, if the input feature is extracted from multiple types of web collected data, the controller 120 may determine that the first web traffic used to collect multiple types of web collected data is traffic to a normal site. . Accordingly, the control unit 120 may determine that the web traffic collected in real time is traffic to a normal site (S470).

The above-described present invention can be implemented as computer readable code on a medium on which a program is recorded. The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. there is Also, the computer may include a control unit. Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

100: malicious site detection device 110: collection unit
120: control unit 130: memory

Claims (5)

collecting a training data set including a plurality of types of training data and extracting a feature from the training data set;
generating an artificial intelligence model by training an artificial neural network to classify normal and malignant types using the features as input data;
Collecting web traffic in real time;
obtaining the plurality of types of web collected data using the web traffic and extracting features from the plurality of types of web collected data; and
Providing features extracted from the web collection data to the artificial intelligence model, and determining whether or not the web traffic is harmful based on a result value output by the artificial intelligence model;
How to detect malicious sites. According to claim 1,
The plurality of types of web collected data,
including URL, URI, HTML, JavaScript and Host
How to detect malicious sites. According to claim 2,
The characteristics extracted from the plurality of types of web collected data are:
Length information of elements constituting the web collection data, information on the number of elements, whether or not the element is in a specific language, the number of elements representing a specific form, entropy of the element, number of tags, specific including the presence or absence of a value, the presence or absence of a specific word, and the number of functions.
How to detect malicious sites. According to claim 1,
When the web traffic is collected, determining whether the collected web traffic is a feature extraction target; further comprising
How to detect malicious sites. According to claim 4,
The step of determining whether the collected web traffic is a feature extraction target,
Determining whether the collected web traffic is a feature extraction target based on whether the file exists, whether it corresponds to a whitelist domain, the type of response code, and whether it is an HTML document
How to detect malicious sites.

Images