KR102008845B1 - Automatic classification method of unstructured data - Google Patents

Abstract

The present invention relates to a method for automatically classifying categories of unstructured data.
Collecting a string of unstructured data from a post posted on an Internet bulletin board; Performing preprocessing on the collected strings of unstructured data, including information extraction and string processing, and character tokenization processing, wherein the string is converted into a vector representation by performing preprocessing; Inputting the preprocessed vector representation into a CNN-LSTM classifier; And automatically categorizing the post's category by a CNN-LSTM classifier.

Description

Automatic classification of categories of unstructured data {AUTOMATIC CLASSIFICATION METHOD OF UNSTRUCTURED DATA}

The present invention relates to a method for automatically classifying categories of unstructured data. More particularly, the present invention relates to a method for classifying which categories a post corresponds to by automatically analyzing a post entered in a specific area such as a bulletin board on the Internet.

Unstructured data refers to data that does not follow a predefined structure, such as text, images, or video. Unstructured data varies with news, comments, social media data, emails, reports, and more.

Companies, institutions, and individuals produce unstructured data every hour. Most unstructured data, however, is not classified and is dead. Analysis is essential for such unstructured data to be meaningful and valuable information.

The first method of analyzing unstructured data is to use classification analysis or clustering analysis. The second method is to categorize into a specific category.

While the two analytical methods used manual and automated treatment methods, application by field is still difficult.

In general, automatic classification systems for text documents tend to rely on feature selection algorithms rather than learning algorithms themselves. Feature selection refers to a technique of selecting only features that contribute to differentiation among categories among features (or words) existing in a learning document.

As the amount of information and documents to be processed increases and becomes more and more complex, this not only slows down the news that needs to be delivered quickly, but also costs more due to the input of human resources. Therefore, the need for automation of document classification is increasing.

In addition, in order to automate document classification, a statistical classification method that simply designates a suitable category by simply using the frequency of words appearing in a document is used, or key words necessary for classification are extracted and based on the extracted words, K-NN, Research has been conducted using data mining algorithms such as decision trees, Bayesian networks, and artificial neural networks. Recently, the convolutional neural network (CNN), which is a deep learning algorithm, is known to be effective for natural language processing, and a sentence classification method using word2vec and CNN to express words as vectors has been proposed. Showed.

The sentence classification method using word2vec and CNN has the advantage of fast training and prediction time because of its simple structure, and it is evaluated as an SVM method that is evaluated as a high performance machine learning tool in various fields of natural language processing and text mining (Support Vector Machine). And Logistic Regression (LR) have improved the classification performance.

However, the sentence classification method using word2vec and CNN presented only the results of the performance type test for the English sentences, so the validity of the model could not be verified when applied to Korean document classification.

The present invention has been made based on the above-mentioned problem, and the sentence classification method using word2vec and CNN is applied to Korean to verify whether it is effective in classifying Korean documents and applying the article postings more accurately in Korean document classification. It is an object of the present invention to provide a method for automatically classifying.

According to an aspect of the present invention for solving the problems, a method for automatically classifying categories of unstructured data is provided. Specifically, the classification method may include collecting a string of unstructured data from a post posted on an Internet bulletin board; Performing preprocessing on the collected strings of unstructured data, including information extraction and string processing, and character tokenization processing, wherein the string is converted into a vector representation by performing preprocessing; Inputting the preprocessed vector representation into a CNN-LSTM classifier; And a step of automatically classifying the category of the post by the CNN-LSTM classifier.

In the above-described aspect, the information extraction and string processing are

Parsing an Excel file to extract body and category information; Processing newline characters and special characters in the string; Automatic spacing of the character string; And applying WPM to the auto spaced string,

The character tokenization process is

And converting the WPM-applied string into a vector representation by using the Word2Vec library.

Also in the above-described aspect, the vector representation input to the CNN-LSTM classifier has a predetermined input length for the CNN through zero-padding, and the height of the matrix of vectors (the number of tokens input to the CNN). Correspondence) is fixed.

Also in the above aspect, the step of automatically classifying the category of the post by the CNN-LSTM classifier,

Performing convolution through a predetermined filter to generate a plurality of feature maps; Performing one max pulling operation on each of the generated feature maps to obtain one feature from each feature map; Concatenating all output values to produce a top level feature vector having a fixed length; Configuring the generated top-level feature vector with a MultiRNNCell consisting of three layers of BasicLSTM Cell; Outputting through a Full Connection + Softmax layer; It is configured to include.

In addition, in the above-described aspect, the output through the Full Connection + Softmax layer is the probability distribution for the label, and the label with the highest output value becomes the predicted label of the given sentence.

According to the present invention, it is possible to provide a classification method for informal Korean text, which has improved performance and effectiveness, compared to the existing CNN-based classification method.

1 illustrates a WPM based lexical dictionary generation algorithm;
2 shows a CBOW / Skip-gram model of Word2Vec;
3 is a structural diagram of a CNN showing the structure of a CNN published by Lecun;
4 shows an LSTM block structure;
5 shows an LSTM network structure;
6 shows an LSTM diagram;
7 is a view showing a data preprocessing process in an embodiment according to the present invention;
8 shows an example of a CNN model utilizing word2vec;
9 is a view briefly showing a CNN model (based model) utilizing word2vec;
10 illustrates a CNN-LSTM model utilizing word2vec according to the present invention;
11 is a diagram showing the configuration of a model classifier for classifier performance analysis;
12 is a diagram showing a learning training diagram using Softmax regression;
13 illustrates performance in a CNN based classifier;
14 illustrates performance in an LSTM based classifier;
FIG. 15 is a diagram illustrating performance of each cell type in a CNN-LSTM based classifier. FIG.
16 shows CNN-LSTM performance using final parameters. And
17 is a diagram showing the performance using the final parameters for each model according to the number of training.

Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms.

In this specification, the embodiments are provided so that the disclosure of the present invention may be completed and the scope of the present invention may be completely provided to those skilled in the art. And the present invention is only defined by the scope of the claims. Thus, in some embodiments, well known components, well known operations and well known techniques are not described in detail in order to avoid obscuring the present invention.

Like reference numerals refer to like elements throughout. In addition, the terms used (discussed) herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. In addition, components and operations referred to as 'includes (or includes)' do not exclude the presence or addition of one or more other components and operations.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used in a sense that can be commonly understood by those skilled in the art. In addition, the terms defined in the commonly used dictionary are not ideally or excessively interpreted unless they are defined.

Related technologies required in the present invention include Word2Vec, CNN, LSTM, WPM and the like are as follows.

Table 1 Techniques Used in the Present Invention and Reasons for Use

Theoretical Background

1. Analytical data (posting inconveniences, improvements and ideas)

-Analysis target: Posts about 172 e-government sites

(Posts collected from 95 informational websites, 43 government representative websites, 31 complaint handling websites, and 3 public participation websites)

-Collection period: October 2013-December 2014

-Analysis quantity: 3,195 cases, 18,343 opinions

-Analysis items: Improvement of inconveniences (9,303) / Ideas (9,0409)

-Analysis content: 3 categories (services, information, system)

[Table 2] Example of a post

2. Natural Language Processing

2.1 Natural Language Processing Concepts

Website posts are composed of natural forms of language such as Korean, English, and numbers that can be recognized by humans. In order to proceed with the machine learning, the computer must express it in a form that can be understood. The related technology is called Natural Language Processing (NLP).

Natural language is distinguished from artificial languages such as computer programming languages. Artificial language is a language that humans set rules so that computers can understand it, so anyone can understand it when they learn the rules. Natural language, although humans set rules, over time, customary rules, informal rules, etc. Due to various changes, there are characteristics that cannot be understood by rules alone. The natural language processing process can be divided into four stages: morphological analysis, syntactic analysis, semantic analysis, and pragmatic analysis. have.

2.2 Auto Spacing System

The most basic task in text analysis is tokenization, which identifies and extracts words from text. Korean is regarded as a word by token, and it is required to use a space between words and words, unlike languages without word boundary marks such as Chinese or Japanese. In Korean, incorrect spacing causes ambiguity or noise in text analysis, which in turn hinders tokenization and decreases readability. As such, spacing in Korean is an important factor that affects machine readability as well as user readability for text. Spacing errors in sentences cause a lot of grammatical and semantic ambiguity, sometimes making stemming impossible.

Therefore, the present invention assumes that a user often writes a post without considering spacing in the Internet environment, and prepares a document by applying automatic spacing. Table 3 below shows an Internet post that is an informal document, and Table 4 shows an example of applying auto spacing to an article that is an informal document.

[Table 3] Example of an unofficial document Internet post

[Table 4] Example of applying automatic spacing to a post that is an informal document

2.3 Word Piece Model

WPM is a method for constructing a voice search system and automatically generates a vocabulary with minimal complexity without prior knowledge of the language. In conventional natural language processing, there are four types of morphological analysis, syntactic analysis, semantic analysis, and pragmatic analysis. However, WPM encodes a unit using a pronunciation set based on the International Phonetic Alphabet (PA). Create new units by combining them according to the frequency of use using statistical techniques. The WPM lexical dictionary generation algorithm consists of the process shown in FIG. 1.

The benefits of WPM are language-independent and statistical, so they can be applied to specific domains or even languages that do not yet have meaning. In the present invention, the WPM algorithm is applied to the document to which the auto spacing is applied as shown in Table 4 as shown in FIG.

[Table 5] below shows an example after applying WPM after applying automatic spacing.

[Table 5] Example of applying WPM after applying auto offset

In other words, WPM uses statistical techniques to combine syllables based on syllables to generate new vocabulary and to create dictionaries with vocabulary with high frequency. [Table 5] shows an example of tokenized sentences using WPM.

[Table 6] Comparative example of sentences before and after applying WPM

[Table 6] shows that WPM can be tokenized differently from morphological analysis. In the case of "Netane" in the "Quick Menu", it is divided into "Quick" and "Menu" when stemming, but it is divided into "Quick", "Me" and "New" when applying WPM. This is because the frequency of appearance of "Quick", "Me", and "New" is higher than that of "Quick Menu" in the corpus used. As such, WPM uses a statistical method, which is language independent and has the advantage of being applicable even in a specific domain or in a state in which meaning is not yet understood.

3. Vector representations of words and documents

Many machine learning algorithms cannot enter a string of text as it is for training, but require that the input be represented as a fixed-length feature vector for the computer to recognize. Prior to document classification, there is a need for a method of representing documents or words in a vector space. Accordingly, in the present invention, word2vec, which is a quantitative technique for expressing words and documents in the form of vectors containing order and meaning, was used. The vectors generated by these techniques can be used as inputs for various machine learning or artificial neural network technologies such as LR, DNN, CNN, LSTM, etc.

3.1 Word2Vec

Word2Vec was originally born from artificial neural network research. Words in the same context begin with the Distribution Hypothesis. word2vec learns through text documents and learns other words that appear as a word in a sentence in the artificial neural network as related words. Because the associated words are more likely to appear nearer in the document, in the process of repeating learning, two words with similar words are placed in a close vector space.

word2vec simply learns whether there is the same information before or after a word. Thus, very abstract verbs or adjectives can be difficult to learn compared to nouns. Nevertheless, looking at countless data, it is possible to learn the semantic relationship between verbs to some extent by grasping the regularity of what object verbs have.

For example, you might learn that break and broken will have similar objects, so the two verbs will have similar meanings. Also, if enough learning is done, the distance in the vector space of break and broken can be equal to the distance in the vector space of have and had. This means that you can learn the meaning of the past.

The model of word2vec is not a deep neural network (DNN). It is an artificial neural network consisting of one hidden layer without activation function and an output layer with softmax function. Therefore, the learning speed is much faster than the general deep neural network, so it is easy to learn very large data. The algorithm of word2vec internally learns sentences using two neural network models, CBOW (Continuous Bag Of Words) and SG (Skip Gram, below). Express in vector space.

The CBOW model has inputs surrounding words at t-2, t-1, t + 1, t + 2, and the output CBOW model has inputs surrounding words at t-2, t-1, t + 1, t + 2. The output is a model that predicts the t-th word. It is known to be several times faster than the SG model. In contrast, the SG model is a model in which the input is one t-th word and the output predicts the surrounding words of t-2, t-1, t + 1, and t + 2. It is known that its performance is slightly better than CBOW because it learns relatively few words well [16]. Therefore, in this paper, we use SG model to generate word vector representation.

4. Machine learning

4.1 Softmax regression

Softmax regression is a multiclass version of logistic regression. Normalize the sum of the denominators of all outputs to one. Then, a ratio is determined for each output. The sum will keep making it 1, so if one powerful feature appears, this value will also affect the rest of the values in the course of convergence to 1, converging to 0, thus accelerating learning.

In the present invention, the performance of the output (soft) regression is known to have better performance than the conventional sigmoid (sigmoid) function is applied to this.

4.2 Convolutional Neural Network (CNN)

CNN is a type of artificial neural network designed to be easily applied to images. CNN was originally proposed in 1998 by Lecun and consists of a convolutional layer and a pooling layer, unlike the structure used in a typical multilayer perceptron.

3 is a structural diagram of a CNN showing the structure of a CNN published by Lecun.

Such a CNN generally consists of several layers and basically three different layers.

Convolutional layer: A layer for extracting convolutional features, which means a layer for extracting significant features.

Pooling layer: In general, CNN is applied to images. Due to the image characteristics, the number of pixels is so large that subsampling (sub-sampling) to reduce the quality is called pooling.

Fully Connected Layer: Lastly applied and used to classify using features from the convolutional and pooling layers. Act like a normal artificial neural network

The general CNN structure has a convolution layer → a pooling layer → a convolution layer → a pooling layer →…. → It is composed of Fully Connected layer. In other words, the feature is extracted by alternating the convolution layer and the pooling layer, and finally, classification is performed through the Fully-Connected layer.

There are three main reasons why CNNs have recently performed better than other algorithms for image classification and object detection.

The first is the introduction of an activation function called Rectified Linear Unit (ReLU), which eliminates the problem of gradient vanishing, a problem that appeared in functions in previous activation functions such as sigmoid and tanh. Gradient vanishing is a phenomenon in which learning does not occur because the gradient change is almost eliminated due to the less amount of error propagating toward the lower layer in the error backpropagation algorithm, which is a representative algorithm for learning neural networks. Because of this problem, it was difficult to learn deep artificial neural networks, but the introduction of ReLU solved this problem, and even deep artificial neural networks were able to learn even lower layers.

The second reason is the emergence of large databases such as ImageNet. Advances in hardware have made mass storage more common and crowdsourcing with Amazon Mechanical Turk has made it possible to manually label the correct answer for large amounts of training data. The overfitting problem could be solved by learning the multi-layer CNN based on more than 1 million large image databases.

In the case of general artificial neural networks, the number of variables to be learned is so large that overfitting easily occurs with a small amount of training data. With the advent of large databases, deep artificial neural networks can be trained without overfitting.

The last reason is regularization with dropout. Dropouts are trained to disable a random percentage of neurons in a learning algorithm to prevent overfitting the artificial neural network. Each integration trained different neurons that did not work to prevent each neuron from learning the same information or learning any information.

For this reason, CNN is reported to perform image classification and object detection effectively when there is a large amount of image data and to show the best performance among existing algorithms.

CNN, originally designed for computer vision, has recently been shown to be effective in natural language processing, and has shown excellent results in semantic parsing, search query retrieval, sentence modeling, and other traditional natural language processing. Therefore, in the present invention, a test is performed using a classifier for classifying documents by using the LSTM and a composite model, which will be described later, together with the CNN.

4.3 Long Short Term Memory (LSTM)

LSTML is the RNN architecture proposed by Hochreiter & Schmidhuber in 1997 and is still the most important RNN. LSTM is a form of replacing the hidden layer units with LSTM blocks in the structure of the conventional RNN structure. 4 is a diagram illustrating an LSTM block structure. As shown in FIG. 4, the LSTM blocks have a recursive structure like the conventional hidden units, and each LSTM block has a recursive structure with a memory cell and an input gate. ), The Forget Gate, and the Output Gate are comprised of three types of gate units. Like conventional RNN, LSTM calculates the final output value through hidden variable, but in the calculation of hidden variable, it uses the gate units mentioned earlier to control the flow of information.

The process of deriving each hidden variable is as follows: First, the forget gate determines how much of the existing cell state stored in the memory device is lost.

LSTM is designed to avoid long-term dependency issues. Remembering information for a long time is actually the basic behavior of the LSTM. Every cyclic neural network has repeating neural network modules in chain form. In a standard cyclic neural network, this repeating module will have a very simple structure like one tanh layer.

5 is a diagram illustrating an LSTM network structure. As shown in FIG. 5, the LSTM has a chain-like structure. However, repeating modules have a different structure. This module has four layers that interact in a very special way instead of one neural network layer.

6 shows an LSTM block diagram. In the diagram shown in FIG. 6, the yellow box is a learned neural network layer. Pink circles represent element-wise operations, such as vector addition. Each arrow passes a whole vector from one node output to the other node's input. The merged arrows indicate concatenates. A forked arrow indicates that its contents are copied and sent elsewhere. At the heart of the LSTM is the cell state, the horizontal line passing through the top of the diagram. The cell state is a kind of conveyor belt. The cell state causes only a slight linear interaction and just passes straight through the entire horizontal line. The information can flow just unchanged. The LSTM may add or remove information to the cell state. Structures called gates control this process. The gate allows information to pass selectively. The gate consists of a sigmoid neural network layer and element-by-element multiplications.

○ Experimental data composition and classification model

1. Data Collection and Experimental Data Set Configuration

From October 2013 to December 2014, 172 e-government sites (95 informational websites, 43 government representative websites, 31 civil complaint handling websites, and public participation) collected through the National Customs Service Monitor Group The post corpus was constructed by collecting posts on the inconveniences and ideas of 3 websites).

[Table 7] Information and classification categories for experiment

As shown in [Table 7], the body information of the post categorizes the inconvenience / improvement and ideas of the site into three categories: "service", "information", and "system". Examples of text documents for each category are as shown in Table 8 for "Service", Table 9 for "Information", and Table 10 for "System".

Table 8 Example of Service Data

Table 9 Example of Information Data

Table 10 Example of System Data

Since the CNN and LSTM models used in this study are based on supervised learning to classify categories, pre-categorized category information is required for each post as a target class. Therefore, category information classified by handwriting of all collected posts was used. In addition, when the training data set is small in model training, the possibility of overfitting is high, so the number of posts by category is shown in [Table 11] below.

[Table 11] Post count and label by category

2. Data Preprocessing

The preprocessing of the experimental data is shown in FIG. 7.

The preprocessing is performed in five steps as shown in FIG. 7. The preprocessing can be largely divided into a process of extracting information, processing a string, and generating a vector representation of a document and a word by tokenizing the document.

2.1 Extracting Information and Organizing Data

For the experimental data set constructed above, first, the data created through the activities of the monitor group was converted into Excel format, and the text and category information were extracted with the elements of [Table 7] by parsing each set. After the string preprocessing was performed on the extracted information as shown in [Table 12], based on the label of [Table 11], the information was divided and stored in three files for each category.

[Table 12] String Preprocessing History

2.2 Select Document Tokenization Method

The CNN-LSTM model used in the present invention cannot input a string of text as it is for training, but requires that the input be digitized to a fixed length so that the computer can recognize it. Therefore, there is a need for a preprocessing process for expressing a document or words included in the document as a fixed size vector with respect to the string-processed data. The library used to convert documents or words to vectors in the present invention is gensim's doc2vec library. Gensim's doc2vec library takes a document's token as input and generates a document and a vector of words in the document, so the tokenization of the document should take precedence. Therefore, an experiment was conducted to select a tokenization method that showed better performance in document classification.

In order to find a better tokenization method for document classification, basic document classification was performed using the vector representation of the document generated using the doc2vec library as an input feature. This is because the vector representation of the document generated using the relatively high performance tokenization method shows the difference between the documents in each category and better classifies the documents in each category. This is because we think that it can contribute to the improvement of performance.

Therefore, the posts were tokenized in three ways: word units , auto-justification, and WPM, and the classification rate was calculated by passing vector representations of the documents created using the doc2vec library to the LR classifier. In consideration of resource usage and performance, the size of each document vector was generated in 300 dimensions.

In experiments looking for a tokenization method, the dataset was divided into 8: 1: 1, where 90% of the total dataset was used for training and the remaining 10% was tested. As shown in [Table 13], the result of applying WPM was the highest as the classification rate of 66%.

[Table 13] Comparison of tokenization methods

[Table 13] shows that when WPM is applied, the maximum number of word units and total tokens is generated, but the number of unique tokens is generated less. In fact, it can be estimated that the number of unique tokens is the smallest when applying WPM. As a result, applying WPM generates a small number of unique tokens, and it can be empirically confirmed that the vector representation of the document using the generated token is useful for the classification of the document.

In the following comparison experiment for each model, vector representations of documents and words were generated using the tokens generated by applying WPM as input to the doc2vec library.

2.3 Vector Generation with Word2Vec

As a final step of the preprocessing, we generated a vector representation of the document and words to be used as input to the CNN-LSTM model. Based on the experiment of selecting tokenization method useful for document classification, the token generated by applying WPM was input to gensim's doc2vec library to generate vector representation of documents and words. gensim's doc2vec library is an extension of the word2vec library. When the vector representation of a document is generated using the doc2vec library, a vector representation of the words contained in the document is also generated. Considering resource usage and performance, the vector size of each document and word is set to 300 dimensions. The parameter settings of the doc2vec class constructor are shown in [Table 14].

Table 14. Parameter settings for the doc2vec class constructor

○ Experiment and performance analysis

1. CNN Model (Base Model) using Word2Vec

CNN models, originally designed for computer vision, are known to be effective in natural language processing and have shown excellent results in semantic parsing, search query retrieval, sentence modeling, and other traditional natural language processing. Sentence level classification in a study by Yoon Kim ("Convolutional Neural Network for Sentence Classification," Proceedings of the 2014 Conference on Classification, "Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.) The CNN, which uses vector representations of pre-trained words through the word2vec algorithm, has been proposed for its work, and has been evaluated as a high-performance machine learning tool in many areas of natural language processing and text mining as well as document classification. The performance of the corpus was improved by using 10-fold cross validation for each corpus. Allocated.

8 is a diagram illustrating an example of a CNN model using word2vec.

The model of FIG. 8 begins by tokenizing a sentence. Tokenized sentences are converted to sentence matrices. The row of the matrix is the word vector representation of each token. In FIG. 8, the sentence is "National Police Agency Homepage, Phishing Crime Me" sentence is divided into 8 tokens, such as Police Agency, homepage, nowadays, blood, singh, crime, me, each token is a vector using the word2vec library Is expressed. In FIG. 8, each token is converted into a vector having 100 dimensions. If the dimension of the word vector is d and the length of the given sentence (number of tokens in the sentence) is s, the dimension of the sentence matrix is s × d, and in FIG. 8 has a shape of 8 × 300. It becomes a sentence matrix When converting to a sentence matrix, the CNN input length is adjusted using the zero-padding method. Now that we've converted the sentence to a matrix, we can treat the sentence like an image, just like in a normal CNN. Next, we perform convolution through the filter. Because the rows of the sentence matrix represent a single token, a word, the width of the filter will use the same width 300 as the dimension of the word vector. Filter width will be 300).

Therefore, the width of the filter is fixed and only the height of the filter can be set differently. The height of the filter is the size of the context window as the number of adjacent words that can be considered together. In the above-mentioned study by Yoon Kim, the height h of the filter is expressed as the region size of the filter. Multiple filters can be applied for the same region size to learn complementary qualities from the same region. In FIG. 8, three (3,4,5) region sizes are obtained, namely, three filters of 3 × 5, 4 × 5, and 5 × 5 each have 150 convolutions of 450 filters. I am adding 450 bias to the output and applying the activation function to create 450 new feature maps. Since only sliding the window from top to bottom for a given sentence, the feature map generated by each filter becomes a vector with a size of s-h + 1 dimension, and the filter region size h and the length of the sentence s Therefore, it will be diversified.

Next, a 1-max pooling operation is performed on each feature map. The 1-max pooling operation selects one of the largest values in each feature map to obtain the most important features in each feature map. The output values of the 1-max pulling operation applied to each feature map are concatenated, and this vector becomes a top-level feature vector. The size of the top-level feature vector is not affected by the length of the sentence, but only by the size of the region and the number of filters, and is equal to the size of the region size × the number of filters. Therefore, in FIG. 8, the size of the top-level feature vector has a fixed length of 3 × 150 = 450 regardless of the length of a sentence input to the model.

Finally, the top-level feature vectors are passed to the Fully-Connected softmax layer (hereafter referred to as the FC layer) for final classification, and the output of the FC layer is the probability distribution for the label. do. The label with the highest output will be the predicted label for the given sentence. In FIG. 8, a sentence is classified into three labels. Dropout may be applied as a means of regularization in the FC layer.

9 is a diagram schematically illustrating a CNN model using word2vec described as an example so far.

2. CNN-LSTM Model Using Word2Vec (Example Model)

The model proposed by Yoon Kim's research described earlier is a one-layer CNN consisting of only one convolution layer and 1-max pooling layer with multiple filters of various domain sizes, and the FC layer is soft without any hidden layer. 9 is a simple structure having only a softmax output layer.

The input layer of the present invention is the same as the base model described above, and the height of a matrix composed of the number of words inputted to the CNN, that is, a vector representation of the words is fixed. The reason for this is that, as described above, the length of the document in terms of the number of words does not decrease the performance instead of inputting the entire word contained in one document to compensate for inefficiencies such as waste of resource usage due to dummy words and increased training time. This is because it is assumed that the input is limited.

Therefore, in the following experiment, we select the number of input words (tokens) that do not degrade the model performance. In the base model, add the LSTM layer after the convolution layer and the MaxPooling layer. In this case, the LSTM model is composed of three LSTM cell layers and a MultiRnnCell. To compensate for the shortcomings of each model, we used the CNN-LSTM model, which utilizes two types of neural networks together, for text classification of posts.

The CNN model is used to extract a vector representing the features of the text, and the input is input to the LSTM model. The classification model is trained to reflect the long-term dependency in the context of the post content.

10 illustrates an example of a composite model in which a LSNN model is added to a CNN model using word2vec according to an embodiment of the present invention. The model of FIG. 10 begins by tokenizing a sentence. Tokenized sentences are converted to sentence matrices. The row of the matrix is the word vector representation of each token. For example, in FIG. 10, the sentence "The Police Agency homepage these days phishing crime I" sentence is divided into eight tokens of "Police Office, homepage, but nowadays, blood, Singh, crime, I", each token is a word2vec library It is represented as a vector using.

In FIG. 10, each token is converted into a 300-dimensional vector. If the dimension of the word vector is d and the length of the given sentence (number of tokens in the sentence) is s, the dimension of the sentence matrix is s × d, and in FIG. 10 has a shape of 8 × 300. It becomes a sentence matrix. When converting to a sentence matrix, the CNN input length is adjusted using a zero-padding method. Next, we perform convolution through a filter. Since the rows of the sentence matrix represent a single token, that is, the word, the width of the filter will use the same width 300 as the dimension of the word vector. Filter width is 300). Therefore, the width of the filter is fixed and only the height of the filter can be set differently. The height of the filter is the size of the context window as the number of adjacent words that can be considered together. Yoon Kim's research described earlier expresses the height h of the filter as the region size of the filter. Multiple filters can be applied to the same region size to learn complementary features from the same region. In FIG. 10, convolution is performed through a total of 450 filters having three (3,4,5) three region sizes, that is, three filters of 3 × 300, 4 × 300, and 5 × 300, respectively. I am creating a new feature map of 300 * 150 * 3 by adding a bias to the output and applying the activation function. Since window sliding is performed only from top to bottom for a given sentence, the feature map generated by each filter becomes a vector with a dimension of s-h + 1 dimension, and the filter region size h and the length of the sentence s Therefore, it will be diversified.

Next, a 1-max pooling operation is performed on each feature map. Although there are various pooling methods for compressing information, in the present invention, the most important feature in each feature map can be obtained by selecting one of the largest values in each feature map through the max_pool_size-max pooling operation using the MAX pooling method. In the max_pool_size-max pooling operation applied to each feature ㅁ map, the value of max_pool_size was selected by experiment 5, and ksize [1,5,1,1] and strides [1,5,1,1] were performed by 20 * Concatenation of 150 * 3 outputs and this vector becomes a top-level feature vector. The size of the top-level feature vector is not affected by the length of the sentence, but only by the size of the region and the number of filters, and is equal to the size of the region size × the number of filters. Therefore, in FIG. 10, the size of the top-level vector has a fixed length of 3 × 150 = 450 regardless of the length of the sentence input to the model. In this study, 20 × 450 top-levels pass through the max pooling layer. Feature vectors are generated.

Finally, the top-level feature vector is composed of 3 layers of BasicLSTMCell for final classification through the LSTM model, which constitutes a MultiRNNCell, which shows slightly better performance than the basic BasicLSTMCell. Passed to the full connection + softmax layer (hereafter referred to as the FC layer), the output of the FC layer is the probability distribution for the label. The label with the highest output value becomes the prediction label of the given sentence.

3. Classifier Performance Analysis

11 is a diagram illustrating a configuration diagram of a model classifier. In the present invention, after applying the automatic spacing and WPM to the post corpus, and generates a corpus. In order to determine the accuracy of the single classifier and the multiple classifier, we measured the accuracy of each classifier and generated a model through the Doc2Vec library of Python and Gensim.

After model generation, 10-fold cross-validation was applied to increase the statistical reliability of classifier performance measurement, and it was trained 20 ~ 500 times according to classification model, and applied to CNN and LSTM machine learning classifier as single model classifier. The accuracy was calculated and compared with the CNN-LSTM model using the proposed composite model classifier.

The classifier experiment environment of the present invention used TikiFlow-learn (sklearn), one of libraries for deep learning of TensorFlow and Python provided by Google. TensorFlow uses multiple GPUs to process the 300-channel array used in this experiment. Scikit-learn is a library mainly used for data mining and analysis.

Then, as a basic classifier, a softmax regression model was constructed as shown in FIG. 12. The training data was trained by inputting a Word2Vec 300-dimensional representation into an array, and the machine data model was generated after comparing the test data with 10-fold cross-validation. The accuracy rate was calculated.

The softmax regression result selected as the default classifier in the present invention will be described in more detail below, but as shown in Table 16, when the corpus for a post is applied to the Softmax regression classifier, the post corpus is classified as 658,579. The highest score was 73.7% when using CNN-LSTM, and 69.7% for LSTM and 68.6% for CNN.

4. Performance experiment and analysis by model

Performance accuracy in the present invention is evaluated with the values of Precision, Recall, F1 for each class. Table 15 below shows the experimental environment and Table 16 shows the final performance results for each model.

[Table 15] Performance Evaluation Experiment Environment

[Table 16] Classification performance by model

4.1 CNN-based Classifier Performance Analysis

13 shows performance according to the number of learning in the CNN based classifier. As shown in FIG. 13, the CNN performed tests by the number of learning (10, 30, 50, 100, 200). As the number of learning increases, performance decreases.

4.2 LSTM Based Classifier Performance Analysis

14 shows the performance according to the number of learning in the LSTM based classifier. LSTM performed the same test by learning count (10, 30, 50, 100, 200) as before. Similar to CNN, the performance was decreased as the number of learning increases.

4.3 CNN-LSTM Based Classifier Performance Analysis

First of all, the performance difference was shown according to the parameter setting value in CNN-LSTM configuration. The present invention basically uses 300-dimensional Word2Vec for word embedding, and the other parameters were tested for performance by adjusting the values variably. Table 17 below shows parameters that are variably changed.

[Table 17] Variable Parameter Types

A) Performance test by maximum sentence length (Max_Sentence_Length)

The minimum number of tokens in a sentence used in the present invention is 2 and the maximum number of tokens is 463, ranging from 2 to 4632.

[Table 18] Performance measurement by maximum sentence length

B) Performance test by filter

Filter-specific performance tests were conducted with two to five filter types. Table 19 below shows the performance of each filter.

[Table 19] Performance Measurement by Filter

C) Performance test by LSTM Hidden Size

The LSTM concealment size was changed in units of 50, 100, 200, 300, 400 and tested. Table 20 below shows the performance of LSTM concealment sizes.

[Table 20] Performance Test by LSTM Hidden Size

D) Performance comparison between BasicLSTM Cell and MultiRNNCell

When using a single layer when setting a cell to receive the value input from the CNN model to the LSTM, when the BasicLSTMCell is set to multi-layer, the number of layers of the cell can be set by using MultiRNNCell. Table 21 below shows the performance of each cell type. Also, in this regard, FIG. 15 shows a performance measurement result for each cell type. As can be seen from Table 21 and FIG. 15, the 50-per-run average 1.7% improved accuracy.

[Table 21] Performance Measurement by Cell Type

5) LSTM MultiRnnCell Layer Performance Test

The LSTM hidden layer settings were tested in 1, 2, and 3 increments. Table 22 below shows the performance of LSTM MultiRnnCell layers.

[Table 22] LSTM MultiRnnCell Layer Performance

4.4 CNN-LSTM Final Performance Measurement and Comparison

Performance measurements were made by combining the test results for each CNN-LSTM configuration method and the test results for each parameter. The configuration method and parameters of CNN-LSTM used in the final performance measurement were based on the performance measured in the unit test. Table 23 shows the parameters used in the final test.

[Table 23] Final Test Parameters

FIG. 16 shows the performance analysis of the CNN-LSTM based classifier using the final parameters of Table 23. FIG. 17 shows the accuracy of each model by performing 10, 30, 50, 100, and 200 training times, respectively. In case of single model of CNN and LSTM, performance decreases as the number of execution increases. However, CNN-LSTM showed more stable performance than single model.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and changes without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not for limiting the technical spirit of the present invention but for the description, and the scope of the technical idea of the present invention is not limited by these embodiments.

Therefore, the protection scope of the present invention should be construed by the claims below, rather than being limited by the above-described embodiment, and all technical ideas within the equivalent scope will be construed as being included in the scope of the present invention.

Claims (5)

In the method of automatically classifying the category of unstructured data,
Collecting a string of unstructured data from a post posted on an internet bulletin board;
Performing preprocessing on the collected strings of unstructured data, including information extraction and string processing, and character tokenization processing, wherein the string is converted into a vector representation by performing preprocessing;
Inputting the preprocessed vector representation into a CNN-LSTM classifier;
And automatically classifying the category of the post by the CNN-LSTM classifier.
How to automatically classify categories of unstructured data.
The method of claim 1,
The information extraction and string processing,
Parsing an Excel file to extract body and category information;
Processing newline characters and special characters in the string;
Automatic spacing of the character string; And
Applying WPM to the auto spaced string,
The character tokenization process is
Converting the WPM-applied string into a vector representation by using the Word2Vec library;
How to automatically classify categories of unstructured data.
The method of claim 2,
The vector representation input to the CNN-LSTM classifier has a predetermined input length for the CNN through zero-padding, and the height of the matrix of vectors (corresponding to the number of tokens input to the CNN) is fixed. there is
How to automatically classify categories of unstructured data.
The method of claim 3,
The step of automatically classifying the category of the post by the CNN-LSTM classifier,
Performing convolution through a predetermined filter to generate a plurality of feature maps;
Performing one max pulling operation on each of the generated feature maps to obtain one feature from each feature map;
Concatenating all output values to produce a top level feature vector having a fixed length;
Configuring the generated top-level feature vector with a MultiRNNCell consisting of three layers of BasicLSTM Cell;
Outputting through a Full Connection + Softmax layer; Containing
How to automatically classify categories of unstructured data.
The method of claim 4, wherein
The output through the Full Connection + Softmax layer is the probability distribution for the label, with the label with the highest output value classified as the predictive label of the given sentence.
How to automatically classify categories of unstructured data.

Images