WO2024089860A1

WO2024089860A1 - Classification device, classification method, and classification program

Info

Publication number: WO2024089860A1
Application number: PCT/JP2022/040260
Authority: WO
Inventors: 弘樹中野; 大紀千葉; 駿小出; 直翼福士
Original assignee: 日本電信電話株式会社
Priority date: 2022-10-27
Filing date: 2022-10-27
Publication date: 2024-05-02

Abstract

This classification device extracts, from Tweets pertaining to reports of phishing attacks that are collected by a collection device, feature amounts for each of text and an image included in the Tweets. The classification device subsequently carries out learning, using the feature amounts, with respect to teaching data labeled with a correct-answer label indicating whether the Tweets pertain to reports of phishing attacks, thereby training a classification model for classifying inputted Tweets with regard to whether the Tweets pertain to reports of phishing attacks. The classification device subsequently classifies the Tweets with regard to whether the Tweets pertain to reports of phishing attacks by using the trained classification model. The classification device then outputs the result of classifying the Tweets with regard to whether the Tweets pertain to reports of phishing attacks.

Description

Classification device, classification method, and classification program

The present invention relates to a classification device, a classification method, and a classification program for classifying posts related to security threat information.

　On social platforms, security experts as well as well-intentioned general users are sharing images (e.g. screenshots) of suspicious phishing attacks they have observed as a warning. If this information can be collected, analyzed, and extracted as quickly and accurately as possible, it will be useful in preventing phishing attacks.

Security blogs, security reports, social platforms, etc. are sources from which information on security threats such as phishing attacks can be extracted.

For example, as in

non-patent documents

3 and 4, natural language processing technology can be applied to blogs and reports that summarize threat information analyzed by security experts, and the data can be extracted as formatted data, making it possible to use it mechanically.

In addition, Non-Patent Document 5 compares and evaluates Twitter (registered trademark), Facebook (registered trademark), news sites, security blogs, security forums, etc. as sources of threat information, and reports that Twitter is superior in terms of both the quantity and quality of information that can be collected.

Non-Patent

Documents

6, 7, and 8 propose technology that focuses on specific users and keywords on Twitter and extracts threat-related URLs, domain names, hash values, IP addresses, vulnerability information, and other information from each user's tweets. It has been reported that this technology can obtain a large amount of useful threat information.

However, the above conventional technologies have the following problems:

(1) The Tweets that are the subject of information collection are limited. Conventional technology limits the subjects of information collection to specific user accounts, so it is not possible to collect information on reports of phishing attacks by various users. In addition, conventional technology collects only limited keywords such as "#phishing" and "#warning", so it can only collect a limited range of Tweets.

(2) Information extraction is limited to text in a certain format contained in Tweets. Reports of phishing attacks via Tweets include images such as screenshots, but the conventional technology extracts information only from text in Tweets. Therefore, the conventional technology cannot extract information contained in images. In addition, since users post information in various formats, the conventional technology, which is specialized in a certain format, can only extract limited information.

As a result, the conventional technology had the problem of being unable to extract useful security threat information. Therefore, the objective of the present invention is to solve the above-mentioned problem and extract useful security threat information.

In order to solve the above problems, the present invention is characterized by comprising a feature extraction unit that extracts features of each of the text and images contained in posts related to security threats on SNS (Social Networking Service) from the posts; a learning unit that uses the features to learn from training data in which each post is labeled with a correct answer as to whether it is a security threat or not, thereby learning a machine learning model for classifying an input post as to whether the post is a security threat or not, a classification unit that uses the trained machine learning model to classify an input post as to whether it is a security threat or not, and an output processing unit that outputs the results of the classification.

The present invention makes it possible to extract useful security threat information.

FIG. 1 is a diagram illustrating an example of a system configuration. FIG. 2A is a diagram illustrating an example of the configuration of a collection device. FIG. 2B is a flowchart illustrating an example of a processing procedure executed by the collection device. FIG. 3 is a diagram for explaining a specific example of a processing procedure executed by the collection device. FIG. 4 is a diagram showing an example of security keywords. Figure 5 is a diagram illustrating an example of generating co-occurrence keywords. FIG. 6 is a diagram showing an example of a Tweet that is the subject of data collection. FIG. 7 is a diagram for explaining the process of extracting a URL and a domain name from the text and image of a Tweet. FIG. 8A is a diagram illustrating an example of the configuration of a classification device. FIG. 8B is a flowchart illustrating an example of a processing procedure executed by the classification device. FIG. 9 is a diagram for explaining a specific example of a processing procedure executed by the classification device. FIG. 10 is a diagram showing an example of feature quantities generated from a Tweet. Figure 11 is a diagram showing an example of an Account Feature of a Tweet. Figure 12 shows an example of a Content Feature of a Tweet. FIG. 13 is a diagram showing an example of a URL Feature of a Tweet. Figure 14 shows an example of an OCR Feature of a Tweet. Figure 15 shows an example of a Visual Feature of a Tweet. Figure 16 shows an example of a Context Feature of a Tweet. FIG. 17 is a diagram showing an example of feature amounts selected by the selection unit in FIG. 8A. FIG. 18 shows the evaluation results of the classification accuracy of the system. FIG. 19 is a diagram showing the number of phishing attack reports and URLs related to phishing attacks extracted by the system during a given period. FIG. 20 is a diagram showing the results of comparing the system with OpenPhish. FIG. 21 is a diagram showing the comparison results between the system and PhishTank. FIG. 22 is a diagram showing the survey results of the number of reports by users and the number of phishing URLs. FIG. 23 is a diagram showing the effect of dynamically selecting keywords. FIG. 24 is a diagram illustrating a computer that executes a program.

Below, a form (embodiment) for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to this embodiment.

[overview]
First, an overview of a system including a collection device and a classification device according to the present embodiment will be described with reference to FIG.

Note that the SNS (Social Networking Service) posts handled by the system will be described as Twitter posts (Tweets) as an example, but are not limited to this. Also, SNS posts may be in either Japanese or English.

In addition, in this embodiment, the system will be described taking as an example a case where posts reporting phishing attacks are collected from SNS posts, but posts reporting security threats other than phishing attacks may also be collected.

The system, for example, quickly and accurately extracts tweets reporting phishing attacks from each user's tweets. For example, the system includes a collection device 10 and a classification device 20. The collection device 10 and the classification device 20 may be connected to each other so as to be able to communicate with each other via a network such as the Internet, or may be installed in the same device.

(1) Collection device 10: Collects a wide range of tweets that may be reports of phishing attacks. For example, the collection device 10 extracts keywords that co-occur in reports of phishing attacks (Co-occurrence Keywords). The collection device 10 then uses keywords related to security threats (Security Keywords) and the above-mentioned Co-occurrence Keywords to collect a wide range of tweets that may be reports of phishing attacks (Screened Tweets in Figure 1).

(2) Classification device 20: Classifies tweets reporting phishing attacks from among the tweets collected by collection device 10. For example, classification device 20 extracts text and image features of tweets reporting phishing attacks through machine learning, and uses the extracted features to classify each tweet as either a tweet reporting a phishing attack or another tweet.

In addition, after the classification device 20 classifies the Tweets, the collection device 10 may extract Co-occurrence Keywords from the group of Tweets classified as Tweets reporting phishing attacks. The collection device 10 may then use the extracted Co-occurrence Keywords to collect Tweets that may be reports of phishing attacks. In this way, the system can dynamically expand/reduce the keywords for collecting Tweets that may be reports of phishing attacks, and collect Tweets that should be collected at the appropriate time.

With such a system, it is possible to collect tweets reporting phishing attacks not only from security experts but also from well-intentioned general users. In addition, because the system collects tweets using a large number of keywords, it is possible to analyze reports of phishing attacks on a large scale.

The system can also accurately extract reports of phishing attacks from the large amount of collected Tweets. Furthermore, the system extracts information about phishing attacks from both the text and images contained in Tweets, making it possible to extract useful information that could not be obtained by simply analyzing the text of Tweets.

This system provides the following benefits in countering phishing attacks:
(1) It becomes possible to collect threat information from a wider range than the limited monitoring targets of conventional technology, making it possible to provide threat information from a new perspective.

(2) In particular, it will be possible to quickly provide threat information that can be used to counter phishing attacks targeting Japanese people, which has been in short supply until now.

(3) Applying the data obtained by this system to telecommunications carriers' filtering rules, etc., will lead to a reduction in the number of victims of phishing attacks, etc.

[Collection Device]
[Configuration example]
Next, a detailed description will be given of the collection device 10. First, a configuration example of the collection device 10 will be described with reference to Fig. 2A. The collection device 10 includes, for example, an input/output unit 11, a storage unit 12, and a control unit 13.

The input/output unit 11 is an interface that handles the input and output of various data. For example, the input/output unit 11 accepts input of Tweets collected from Twitter. In addition, the input/output unit 11 outputs Tweets that may be reports of phishing attacks extracted by the control unit 13 (Screened Tweets in FIG. 1 ).

The memory unit 12 stores data, programs, etc. that are referenced when the control unit 13 executes various processes. The memory unit 12 is realized, for example, by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The memory unit 12 stores, for example, security keywords, co-occurrence keywords, etc. extracted by the control unit 13.

The control unit 13 is responsible for controlling the entire collection device 10. The functions of the control unit 13 are realized, for example, by a CPU (Central Processing Unit) executing a program stored in the memory unit 12.

The control unit 13 includes, for example, a first collection unit 131, a keyword extraction unit 132, a second collection unit 133, and a data collection unit 134. Note that a URL/domain name extraction unit 135 and a selection unit 136, shown by dashed lines, may or may not be provided, and cases in which they are provided will be described later.

The first collection unit 131 uses Security Keywords, which are keywords related to security threats, to collect Tweets reporting phishing attacks from each user's Tweets.

The keyword extraction unit 132 extracts co-occurrence keywords, which are keywords that co-occur with more than a predetermined frequency, from tweets reporting phishing attacks collected by the first collection unit 131. Note that these co-occurrence keywords may be extracted from tweets classified by the classification device 20 as tweets reporting phishing attacks.

The second collection unit 133 uses the Co-occurrence Keywords to collect Tweets that may be reports of phishing attacks from the Tweets of each user. For example, the second collection unit 133 collects Tweets that contain Security Keywords and Co-occurrence Keywords in the text of the Tweet or in images linked to the Tweet from the Tweets of each user. The collected Tweets are stored, for example, in the memory unit 12.

The data collection unit 134 collects data necessary for input to the classification device 20. For example, the data collection unit 134 collects the following data from Tweets collected by the second collection unit 133: (1) Tweet character strings (e.g., hashtags, number of characters, etc.), (2) meta information linked to the Tweet (e.g., application information, presence or absence of defang, etc.), (3) information related to the Tweet's account (e.g., number of followers of the account, period of account registration, etc.), and (4) images included in the Tweet (e.g., up to four images linked to the Tweet, etc.). The collected data (collected data) is stored, for example, in the memory unit 12.

[Example of processing procedure]
Next, an example of a processing procedure executed by the collection device 10 will be described with reference to Fig. 2B. First, the first collection unit 131 of the collection device 10 collects tweets reporting phishing attacks using, for example, security keywords (S1: collection of tweets using security keywords). Then, the keyword extraction unit 132 extracts co-occurrence keywords, which are keywords that co-occur with a predetermined frequency or more, from the tweets reporting phishing attacks collected in S1 (S2: extraction of co-occurrence keywords).

After S2, the second collection unit 133 uses the Security Keywords and Co-occurrence Keywords to collect Tweets that may be reports of phishing attacks from each user's Tweets (S3). After that, the data collection unit 134 collects data necessary for input to the classification device 20 from the Tweets collected in S3 (S4).

By performing the above process, the collection device 10 can collect tweets that may be reports of phishing attacks.

The collection device 10 may also include a URL/domain name extraction unit 135 and a selection unit 136 as shown in FIG. 2A.

The URL/domain name extraction unit 135 extracts URLs and domain names from the text and images of the Tweets collected by the second collection unit 133. The selection unit 136 selects Tweets that are likely to be reports of phishing attacks from the Tweets collected by the second collection unit 133, based on the URLs or domain names extracted by the URL/domain name extraction unit 135.

For example, if a URL or domain included in a Tweet collected by the second collection unit 133 is not included in the list of URLs or domain names of legitimate websites, the selection unit 136 selects the Tweet as likely to be a report of a phishing attack. In addition, if the domain name of the URL included in the Tweet has been in use for less than a predetermined period, the selection unit 136 selects the Tweet as likely to be a report of a phishing attack. For example, the selection unit 136 selects a domain name that has been registered in WHOIS for less than a predetermined number of days as a Tweet that is likely to be a report of a phishing attack.

Then, the data collection unit 134 collects data (e.g., Tweet character strings, etc.) necessary for input to the classification device 20 from the Tweets selected by the selection unit 136.

In this way, the collection device 10 can collect tweets and their data that are more likely to be reports of phishing attacks from the collected tweets.

[Specific example of processing procedure]
Next, a specific example of the process executed by the collection device 10 will be described with reference to Fig. 3. Note that the collection device 10 will be described with reference to a case where it is equipped with a URL/domain name extraction unit 135 and a selection unit 136.

(1) Generating Keywords
The collection device 10 generates two types of keywords (Security Keywords and Co-occurrence Keywords) for searching for Tweets containing reports of phishing attacks.

(1-1) Security Keywords
First, the security keywords will be described. For example, the collection device 10 generates, as security keywords, keywords related to security threats and the media through which they are spread, such as "SMS" and "fake site," and keywords for sharing security threat information, such as "#phishing" and "#fraud" (see FIG. 4). Note that existing keywords related to security threats may be used as the security keywords.

(1-2) Security Keywords
Next, the co-occurrence keywords will be described. For example, the collection device 10 extracts co-occurring keywords (co-occurrence keywords) with a frequency exceeding a predetermined value only from reports of phishing attacks collected using security keywords as keys.

For example, the first collection unit 131 of the collection device 10 uses Security Keywords to collect Tweets reporting phishing attacks from each user's Tweets. The keyword extraction unit 132 then extracts Co-occurrence Keywords from the collected Tweets. For example, the keyword extraction unit 132 newly extracts Co-occurrence Keywords from the Tweets collected during each specified period.

For example, the keyword extraction unit 132 extracts proper nouns from the character strings of tweets for a given period of time, and calculates PMI (Pointwise Mutual Information) using the following formula (1). Note that X and Y in formula (1) are proper nouns contained in the tweets.

PMI(X,Y)=log(P(X,Y)/P(X)P(Y))…Equation (1)

Next, the keyword extraction unit 132 calculates the SoA using formula (2). In formula (2), W is a proper noun contained in the Tweet, and L is a label (security threat information or other).

SoA(W,L)=PMI(W,L)-PMI(W,￢L)...Equation (2)

Then, the keyword extraction unit 132 extracts proper nouns whose SoA exceeds a predetermined threshold. For example, tweets containing the security keyword "fraud" include tweets related to phishing reports shown in FIG. 5 (1) and tweets unrelated to phishing reports shown in FIG. 5 (2). The keyword extraction unit 132 extracts "Company d" and "SMS," proper nouns that appear frequently (whose SoA exceeds a predetermined threshold) only in tweets ((1)) related to phishing reports that contain "fraud," as co-occurrence keywords.

(2) Searching Tweets
Next, the collection device 10 collects data necessary for input to the classification device 20 from Twitter. For example, the second collection unit 133 collects Tweets that may be reports of phishing attacks from Tweets of each user by using the co-occurrence keywords extracted by the keyword extraction unit 132. In this way, the second collection unit 133 can collect Tweets that include URLs and domains of Potentially Phishing Sites, for example, as shown in FIG. 3.

In other words, the second collection unit 133 can collect Tweets (Screened Tweets) from among the Tweets of each user, excluding Tweets (Unrelated Tweets) related to Legitimate Sites. The data collection unit 134 collects the following data related to the Tweets collected by the second collection unit 133 (see FIG. 6).

Tweet string (e.g. hashtag, number of characters, etc.), meta information associated with the Tweet (e.g. application information, whether or not defanged, etc.), information about the Tweet's account (e.g. number of followers, period of account registration, etc.), images included in the Tweet (e.g. up to four images associated with the Tweet, etc.).

(3) Extracting URLs and Domain Names
Next, the URL/domain name extraction unit 135 of the collection device 10 extracts URLs and domain names from the text and images of the Tweets (Screened Tweets) collected by the second collection unit 133 .

For example, the URL/domain name extraction unit 135 applies optical character recognition to the image of the Tweet to extract a character string. In addition, if a defang (e.g., https -> ttps) is present in the character string of the Tweet, the URL/domain name extraction unit 135 restores it to its original state. The URL/domain name extraction unit 135 then extracts URLs and domain names from the character strings in the text and image of the Tweet using regular expressions. The URL/domain name extraction unit 135 then checks whether the extracted domain name exists in the Public Suffix List (see Reference 1) or the like.

Reference 1: “Public Suffix List”, https://publicsuffix.org/

Then, when the URL/domain name extraction unit 135 confirms that the extracted domain name exists, it extracts the domain name and a URL that includes the domain name. For example, the URL/domain name extraction unit 135 extracts the following URL and domain name from the Tweet shown in FIG. 7.

・URL: https://tinyurl.com/yph6pswp, https://atavollwei.duckdns.org/
Domain names: tinyurl.com, atavollwei.duckdns.org

(4) Screening Phishing-related URLs and Domain Names
Next, the selection unit 136 screens the URLs and domain names extracted by the URL/domain name extraction unit 135 for URLs and domain names related to phishing.

For example, if the extracted URL or domain name does not match the Allowlist (e.g., a list of URLs or domain names of legitimate websites) and is not a Long-lived Domain Name (e.g., a domain name that has been registered in WHOIS for a predetermined number of days or more), the selection unit 136 determines that the extracted URL and domain name are Potentially Phishing Sites. The selection unit 136 then selects Tweets that include URLs or domain names determined to be Potentially Phishing Sites as Tweets that are likely to be reports of phishing attacks.

On the other hand, if the extracted URL and domain name match the Allowlist or are Long-lived Domain Names, the selection unit 136 determines that the URL and domain name are Legitimate Sites.

For example, if the extracted domain name corresponds to a domain name of a predefined URL shortening service, the selection unit 136 passes the domain name. In addition, if the extracted domain name matches the Tranco List (see Reference 2), the selection unit 136 excludes the domain name as a domain name that is not related to phishing attacks.

- Reference 2: "A research-oriented top sites ranking hardened against manipulation - Tranco", https://tranco-list.eu/

The selection unit 136 also queries WHOIS for the extracted domain name, and if no information can be obtained, passes the domain name. Furthermore, based on the WHOIS information, the selection unit 136 excludes a domain name if it has been more than 365 days since it was registered, and passes the domain name if it has not been 365 days since it was registered. The selection unit 136 then selects, for example, a Tweet that contains at least one URL or domain name that has been passed in the above process as a Tweet that is likely to be a report of a phishing attack.

In this way, the collection device 10 can extract tweets from each user that are likely to be reports of phishing attacks.

[Classification device]
[Configuration example]
Next, a detailed description will be given of the classification device 20. First, a configuration example of the classification device 20 will be described with reference to Fig. 8A. The classification device 20 includes, for example, an input/output unit 21, a storage unit 22, and a control unit 23.

The input/output unit 21 is an interface that handles the input and output of various data. For example, the input/output unit 21 accepts input of tweets that may be reports of phishing attacks collected by the collection device 10 and the associated data. The input/output unit 21 also outputs the classification results obtained by the control unit 23.

The storage unit 22 stores data, programs, etc. referenced when the control unit 23 executes various processes. The storage unit 22 is realized by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 22 stores tweets that are likely to be reports of phishing attacks received by the input/output unit 21 and the data (collected data), etc. In addition, the storage unit 22 stores parameters of the classification model after the control unit 23 has learned the classification model.

The control unit 23 is responsible for controlling the entire classification device 20. The functions of the control unit 23 are realized, for example, by the CPU executing a program stored in the storage unit 22.

The control unit 23 includes, for example, a data acquisition unit 231, a feature extraction unit 232, a feature selection unit 233, a learning unit 234, a classification unit 235, and an output processing unit 236.

The data acquisition unit 231 acquires tweets and their data that are likely to be reports of phishing attacks from the collection device 10.

The feature extraction unit 232 extracts features from the Tweet and its data acquired by the data acquisition unit 231. For example, the feature extraction unit 232 extracts features from the text and image of the Tweet acquired by the data acquisition unit 231.

For example, the feature extraction unit 232 extracts, from a Tweet acquired by the data acquisition unit 231, features of the account of the Tweet, features of the content of the Tweet, features of the URL or domain name included in the Tweet, features of a character string obtained by optical character recognition of an image included in the post, features of an image included in the Tweet, features of the context of the text included in the Tweet, etc. Details of the extraction of Tweet features by the feature extraction unit 232 will be described later using specific examples.

The feature selection unit 233 selects, from among the features extracted by the feature extraction unit 232, features that are effective in classifying whether or not a tweet is related to a report of a phishing attack. For example, the feature selection method uses Boruta-SHAP (see References 3 and 4).

Reference 3: Kursa, Miron B. and Rudnicki, Witold R., “Feature Selection with the Boruta Package,” Journal of Statistical Software 2010.
Reference 4: “BorutaShap: A wrapper feature selection method which combines the Boruta feature selection algorithm with Shapley values,” https://zenodo.org/badge/latestdoi/255354538

For example, the feature selection unit 233 selects, from among the features extracted by the feature extraction unit 232, features that are effective for classifying whether or not a tweet is related to a report of a phishing attack, using the following procedure.

(1) First, the feature selecting unit 233 generates false features that include random values in addition to the features to be selected.
(2) Next, the feature selection unit 233 classifies the features to be selected and the false features using a decision tree-based algorithm, and calculates the variable importance of each feature.
(3) Next, the feature selecting unit 233 counts the variable importance of the feature to be selected calculated in (2) if it is greater than the variable importance of the false feature.
(4) The feature value selection unit 233 repeats the processes (1) to (3) multiple times and selects feature values that are determined to be statistically significant as feature values that are effective for classification.

The learning unit 234 learns a machine learning model (classification model) for classifying whether an input Tweet is a Tweet reporting a phishing attack or not through supervised learning using the features selected by the feature selection unit 233. For example, the learning unit 234 learns a classification model through supervised learning using the features selected by the feature selection unit 233 for teacher data related to phishing attacks (data to which each Tweet is assigned a correct answer label indicating whether it is a phishing attack or not).

The classification unit 235 uses the classification model learned by the learning unit 234 to classify whether the input Tweet is a Tweet reporting a phishing attack. The output processing unit 236 outputs the result of the classification of the Tweet by the classification unit 235.

[Example of processing procedure]
Next, an example of a processing procedure executed by the classification device 20 will be described with reference to Fig. 8B. First, the data acquisition unit 231 of the classification device 20 acquires Tweets and their data that are likely to be reports of phishing attacks collected by the collection device 10 (S11: Acquisition of collected data). After that, the feature extraction unit 232 extracts features from the Tweets and their data acquired by the data acquisition unit 231 (S12: Extraction of Tweet features).

After S12, the feature selection unit 233 selects, from the features extracted in S12, features that are effective for classifying whether or not a Tweet is a report of a phishing attack (S13). Then, the learning unit 234 uses the features selected in S13 for the teacher data related to phishing attacks to learn a classification model for classifying whether or not an input Tweet is a report of a phishing attack (S14).

After S14, the classification unit 235 uses the classification model learned in S14 to classify whether the input Tweet is a Tweet reporting a phishing attack (S15). Then, the output processing unit 236 outputs the result of the classification in S16 (S16).

[Specific example of processing procedure]
Next, a specific example of a processing procedure executed by the classification device 20 will be described with reference to FIG.

(5) Feature Engineering
First, the data acquisition unit 231 of the classification device 20 acquires Tweets (Screened Tweets) and their data collected by the collection device 10. Then, the feature extraction unit 232 extracts features from the Tweets and their data acquired by the data acquisition unit 231.

For example, as shown in FIG. 10, the feature extraction unit 232 generates a total of 27 features of six types: Account Feature (1) from the account of the Tweet, Content Feature (2) from information linked to the Tweet, URL Feature (3) from the extracted URL, OCR Feature (5) from character strings extracted by OCR, Visual Feature (6) from the appearance of the image, and Context Feature (4) from the context of the Tweet. Each feature is explained in detail below.

(5-1) Account Feature
In order to capture the characteristics of a Twitter user, the feature extraction unit 232 generates an Account Feature for each Tweet from information about the user's account (e.g., number of followings, number of followers, number of Tweets, number of media, number of lists, account registration date, etc.), as shown in FIG. 11 .

(5-2) Content Feature
In order to capture the characteristics of content that frequently appears in Tweets reporting phishing attacks, the feature extraction unit 232 generates a Content Feature for each Tweet from information linked to the Tweet itself (e.g., a character string, a mentioned user, a hashtag, an image, a URL or domain name, an application used in the Tweet, a defang type, etc.), as shown in FIG. 12 .

(5-3) URL Feature
In order to capture features related to the abuse of subdomains specific to phishing URLs and the abuse of specific top-level domains, the feature extraction unit 232 generates a URL Feature for each Tweet from the URL (or domain name) extracted from both the character string and image of the Tweet, as shown in Fig. 13. The URL Feature is, for example, the character string of the URL, the domain name, the path, the numbers included in the URL, the top-level domain, etc.

(5-4) OCR Feature
In order to capture characteristics of similar character strings in Tweets related to phishing attacks, the feature extraction unit 232 generates an OCR feature for each Tweet from character strings extracted by optical character recognition (OCR), as shown in Fig. 14. The OCR feature is, for example, a character string, a word, a symbol, a number, a URL, a domain name, etc.

(5-5) Visual Feature
In order to capture the commonality in the appearance of images contained in Tweets related to reports of phishing attacks, the feature extraction unit 232 generates a Visual Feature for each Tweet from the images associated with the Tweet.

The feature extraction unit 232 uses the Efficient Net model (see Reference 5), which has produced excellent results in image classification, to generate a fixed-dimensional vector of the image linked to the Tweet. The feature extraction unit 232 then compresses the dimension of the vector using Truncated SV (see Reference 6), which converts a sparse vector into a dense vector. The feature extraction unit 232 then treats the compressed vector as the Visual Feature of the image included in the Tweet.

・Reference 5: Tan, Mingxing and Le, Quoc. “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, ICML 2019.
・Reference 6: “The truncatedsvd as a method for regularization”, BIT Numerical Mathematics.

The feature extraction unit 232 converts images associated with Tweets into vectors with inherent dimensions using an Efficient Net model that has been pre-trained on a large number of images from the Image Net, as shown in FIG. 15, for example. The feature extraction unit 232 then compresses the converted vectors to a cumulative contribution rate of 99% in the training data using Truncated SV.

(5-6) Context Feature
In order to grasp the commonality of context in Tweets related to reports of phishing attacks, the feature extraction unit 232 generates a Context Feature for each Tweet from character strings in the Tweet.

The feature extraction unit 232 generates a fixed-dimensional vector from the character strings in the Tweet, for example, using the BERT model, which has shown excellent results in sentence classification. The feature extraction unit 232 then compresses the dimension of the vector using Truncated SV. The feature extraction unit 232 then sets the compressed vector as the Context Feature of the Tweet.

The feature extraction unit 232 converts the strings in the Tweet into vectors with inherent dimensions using a BERT model that has been pre-trained on a large number of strings from Wikipedia in English and Japanese, as shown in FIG. 16. The feature extraction unit 232 then compresses the converted vectors to a cumulative contribution rate of 99% in the training data using Truncated SV.

(6) Feature Selection
The feature selection unit 233 selects, from the group of features generated by the feature extraction unit 232 in (5), features that are effective (important) for classifying tweets reporting phishing attacks from other tweets.

In addition, Figure 17 shows examples of features that were determined to be important for classification as a result of feature selection.

Account Feature: 6 English (6 dimensions), 5 Japanese (5 dimensions)
Content Feature: 6 English types (9 dimensions), 4 Japanese types (7 dimensions)
URL Feature: 2 English (2D), 3 Japanese (3D)
OCR Feature: 3 types of English (3D), 3 types of Japanese (3D)
Visual Feature: English 9 dimensions, Japanese 5 dimensions
Context Feature: 58 dimensions in English, 33 dimensions in Japanese

Furthermore, among the Context Features shown in Figure 17, for App source (14), Twitter Web App, Twitter for iPhone (registered trademark), and Twitter for Android (registered trademark) were important in both languages, while PhishingPicker was important only in the case of English. Furthermore, for Defanged type (15), example[.]com was important in both languages, while hxxp was important only in the case of Japanese. Furthermore, among the URL Features shown in Figure 17, for Top-level domain (20), .xyz was important only when it was Japanese.

Finally, we were able to confirm that 87 English and 56 Japanese feature dimensions are important for classifying tweets reporting phishing attacks from other tweets.

(7) Offline Training
The learning unit 234 learns a classification model (machine learning model) using the features (feature vectors) selected by the feature selection unit 233 in (6) and training data (Ground-Truth Dataset) to which correct labels indicating whether or not the attack is a phishing attack have been assigned.

Algorithms that can be used to train classification models include, for example, Random Forest, Neural Network, Decision Tree, Support Vector Machine, Logistic Regression, Naive Bayes, Gradient Boosting, and Stochastic Gradient Descent. After evaluating these algorithms against training data, it was confirmed that it is preferable to use Random Forest for the following three reasons.

- Random Forest had better classification accuracy than any other algorithm.
- Random Forest performed at a stable speed in both the learning and estimation (classification) phases.
・Random Forest had a distributed feature importance for all six types of features.

(8) Online Classification
The classification unit 235 classifies the Tweets collected by the collection device 10 into Tweets related to reports of phishing attacks (positive) or not (negative) using the machine learning model (classification model) learned in (7). Then, the output processing unit 236 outputs the result of the classification.

The classification device 20 may extract proper nouns that appear in tweets classified as reports of phishing attacks, and the collection device 10 may use the proper nouns when extracting co-occurrence keywords.

[Evaluation results]
Next, the evaluation results of the system of this embodiment will be described. For example, it was confirmed that by using the features selected by the system, it is possible to classify tweets reporting phishing attacks with an accuracy of approximately 95% in both English and Japanese (see FIG. 18).

In addition, during the experimental period (August 1, 2021 to September 30, 2021), the system of this embodiment was able to extract 77,004 phishing attack reports (User Reports) and 85,027 phishing URLs (Phising URLs), as shown in FIG. 19.

Furthermore, when phishing URLs collected by the system of this embodiment were compared with those collected by the existing data feed OpenPhish (see Reference 7) (see FIG. 20), it was found that of the 4,802 phishing URLs common to both, the system of this embodiment was able to collect 2,686 phishing URLs (55.9% of the total) more quickly.

Reference 7: “OpenPhish - Phishing Intelligence”, https://openphish.com

Furthermore, when phishing URLs collected by the system of this embodiment were compared with those collected by PhishTank (see Reference 8), an existing data feed (see FIG. 21), it was found that of the 5,323 phishing URLs common to both, the system of this embodiment was able to collect 3,183 phishing URLs (59.8% of the total) more quickly.

- Reference 8: “PhishTank | Join the fight against phishing”, https://www.phishtank.com/.

Furthermore, when the number of phishing attack reports by users and the number of phishing URLs were investigated, it was confirmed that phishing attacks reported only once by users accounted for 49.8% of all phishing URLs (see Figure 22). In other words, it was confirmed that reports of phishing attacks from a wide range of users are highly likely to contain phishing URLs that are highly unique. From this, it was confirmed that collecting reports of phishing attacks from a wide range of users, as in the system of this embodiment, is extremely effective.

We also confirmed the effectiveness of using not only fixed keywords (Security Keywords) but also dynamic keywords (Co-occurrence Keywords) to collect tweets reporting phishing attacks (see Figure 23). As a result, we confirmed that using not only fixed keywords (Security Keywords) but also dynamic keywords (Co-occurrence Keywords) was able to extract +23.3% more User Reports (tweets reporting phishing attacks) than using only fixed keywords (Security Keywords). We also confirmed that using not only fixed keywords (Security Keywords) but also dynamic keywords (Co-occurrence Keywords) was able to extract +24.1% more phishing URLs.

From this, it was confirmed that collecting tweets using not only fixed keywords (Security Keywords) but also dynamic keywords (Co-occurrence Keywords), as in the system of this embodiment, is extremely effective in collecting information on phishing attacks.

[System configuration, etc.]
In addition, each component of each part shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[program]
The above-mentioned system can be implemented by installing a program as package software or online software on a desired computer. For example, the above-mentioned program can be executed by an information processing device to function as the above-mentioned system. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), as well as terminals such as PDAs (Personal Digital Assistants).

FIG. 24 is a diagram showing an example of a computer that executes a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these components is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define each process executed by the above-mentioned system are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the system is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

The data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes it.

The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN (Local Area Network), WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

REFERENCE SIGNS LIST 10

Collection device

11, 21 Input/

output unit

12, 22

Memory unit

13, 23 Control unit 20 Classification device 131 First collection unit 132 Keyword extraction unit 133 Second collection unit 134 Data collection unit 135 URL/domain name extraction unit 136 Selection unit 231 Data acquisition unit 232 Feature extraction unit 233 Feature selection unit 234 Learning unit 235 Classification unit 236 Output processing unit

Claims

A feature extraction unit that extracts features of text and images included in a post about a security threat on a social networking service (SNS);
a learning unit that performs learning using the features of training data in which each post is labeled with a correct answer as to whether or not the post is a post related to a security threat, thereby learning a machine learning model for classifying an input post as to whether or not the post is a post related to a security threat;
A classification unit that classifies an input post as whether or not the input post is a post related to a security threat using the trained machine learning model;
and an output processing unit that outputs a result of the classification.
The feature amount of the image included in the post is
The classification device according to claim 1 , further comprising a feature amount of the image and a feature amount of a character string obtained by optical character recognition of the image.
The feature amount is
The classification device of claim 1 , further comprising URL or domain name features extracted from the post text or images.
The feature amount is
The classification device according to claim 1, characterized in that the features are at least any of the following: a feature of the account of the poster of the post; a feature of the content of the post; a feature of a URL or domain name extracted from the text or image of the post; a feature of a character string obtained by optical recognition of the image included in the post; a feature of the image included in the post; and a feature of the context of the text included in the post.
a feature selection unit that selects a feature that is effective for classifying whether or not a post is related to a security threat from among the features extracted by the feature extraction unit,
The learning unit is
The classification device according to claim 1 , further comprising: a machine learning model that is trained using the selected feature quantity.
The feature amount selection unit is
The classification device according to claim 5 , further comprising: selecting a feature quantity effective for classifying whether or not a post is related to a security threat by Boruta-SHAP.
1. A classification method performed by a classification device, comprising:
extracting feature quantities of text and images contained in posts related to security threats on a social networking service (SNS);
A step of training a machine learning model for classifying an input post as whether or not the post is a security threat by performing training using the features on training data in which each post is labeled with a correct answer as to whether or not the post is a security threat;
A step of classifying an input post as being related to a security threat or not using the trained machine learning model;
and outputting a result of the classification.
extracting feature quantities of text and images contained in posts related to security threats on a social networking service (SNS);
A step of training a machine learning model for classifying an input post as whether or not the post is a security threat by performing training using the features on training data in which each post is labeled with a correct answer as to whether or not the post is a security threat;
A step of classifying an input post as being related to a security threat or not using the trained machine learning model;
and a step of outputting the results of the classification.