JP2008084064A - Text classification processing method, text classification processing device and text classification processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a text classification processing method for determining whether a text belongs to a certain category or not. <P>SOLUTION: The text classification processing method comprises extracting a character string of a fixed length or less from the text, calculating a characteristic quantity of the character string, and generating a characteristic vector originated from the characteristic quantity. When a training data group with a label related to whether the text thereof belongs to a certain category or not being preliminarily assigned is given, the text of the training data group is converted to the characteristic vector, the characteristic vector is applied to a support vector machine together with the label to perform learning, and a text classifier by the support vector machine is generated. When a text which is unknow whether it belongs to a certain category or not is given, the characteristic vector of the unknown text is generated, and whether the text belongs to the category or not is determined by use of the generated text classifier. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention realizes, for example, a text classification processing method for performing text classification processing for determining whether or not the text belongs to a certain category for text including a character string that is difficult to perform morphological analysis, and realizes such classification. It relates to a method for constructing a classifier from a training data set of pre-labeled text sets as to whether they belong to a category.

  A lot of research and development has been done on text classification. For example, it was developed for the purpose of removing inappropriate e-mail (e-mail that is unnecessary for users called junk mail or spam), or automatically inquiring mail to the person in charge at the customer center. Much research and development has been done for various applications, such as organizing articles by topic.

  One method for realizing text classification is a method of classifying by a classifier prepared in advance. Typically, this is a method of classifying texts based on pre-defined IF / THEN rules. For example, it is a method of classifying texts using a rule such that if the words “TSE” and “closing price” and numbers appear, the text is related to “stock market”. Unfortunately, however, text classification by such a method has a problem that it becomes difficult to maintain and manage the rules for classification while maintaining consistency as the number of texts and the number of vocabularies increases.

  Therefore, in recent years, instead of such a method using a static classifier, a method for dynamically learning a classifier from data as necessary has become mainstream. For this type of classifier, various representation formats such as IF / THEN rules, neural networks, decision trees, probability models, separation hyperplanes, etc. are used, but various learning algorithms have been proposed for each representation format. . Since the learning method of these classifiers is generally known, and the description thereof is well known to those skilled in the art, the description thereof is omitted here.

  Here, an outline of a support vector machine (Non-Patent Document 1) used in the present invention will be described. It is assumed that the data is expressed as points on the n-dimensional space, and further, one of two types of labels +1 and −1 indicating whether or not the data belongs to a certain category is given to these points. At this time, the support vector machine is a method of calculating an n-dimensional space hyperplane that optimally separates the labeled data under a certain standard and classifying the data by this hyperplane. In the data classification, when data with an unknown label is given, the label of this data can be predicted by examining which side of the hyperplane this data is. In other words, if the text is expressed as a point (called a feature vector) in an n-dimensional space by some method, a classifier that determines whether the text belongs to a certain category is used using a support vector machine. The hyperplane in the n-dimensional space can be calculated and learned from the data, and a classifier can be generated accordingly.

  One of the advantages of using a support vector machine for learning a text classifier is that even if the dimension of the hyperplane is very high, using a method called kernel tricks requires an amount of computation that does not depend on this dimension. The point is that the hyperplane can be learned. Another advantage is that even if the dimension of the feature vector is very high, the risk that the classifier will overfit the training data and lose generality (called overfitting) It is known from experience that it is small compared to.

  For example, in Non-Patent Document 2, when a text is expressed as a feature vector that uses the feature amount of an appearing word as a component (feature), a support vector machine is used, which is more accurate than using other learning methods. It is shown that the classifier can be learned. Moreover, it has been shown that even if more than 10,000 words are used, the learning performance is improved without causing overfitting. Note that Non-Patent Document 4 can be referred to for more detailed information and theoretical analysis regarding the application of the support vector machine to text classification.

  In Patent Document 1, a word is extracted from Japanese text by morphological analysis, a word is analyzed based on part-of-speech information, etc., and a feature vector having the word as a feature is generated, and then a support vector machine is applied. The technology to do is disclosed.

  In Patent Document 2, a feature vector is calculated using tokens separated by blanks or punctuation marks as basic features, a hyperplane is learned using a support vector machine, and a weight vector of the obtained hyperplane is obtained. Disclosed is a text classification technique that uses a certain monotonic function as a classifier.

  As can be seen from these documents, feature vectors that identify words are often used in text classification. However, when the word sequence is important, the word N In some cases, the connection (called an N-gram) is used as a feature. Also, as can be seen in Patent Document 5, Patent Document 6, Patent Document 18, etc., words, phrases, dependency structures, etc. to be used as features are prepared in advance in the feature dictionary, and this dictionary is used. A method of extracting features and generating feature vectors is also commonly used.

  Also, in the case of Japanese text, unlike English text, words are not separated, so words are usually extracted by morphological analysis, but at this time, a feature (morpheme) dictionary is also used. The problem with using a feature dictionary is that it is not possible to extract features that are not registered in the dictionary, but unregistered words can often be detected as unknown words due to the relationship between before and after. In the text classification, the use of a feature dictionary is rarely a problem, and in fact, all the patent documents related to Japanese text classification cited below use some kind of feature dictionary.

  On the other hand, there is also known a method for analyzing an arbitrary character string in text without using a feature dictionary. For example, in Patent Document 4, whether a character string is a general expression or a specialized expression by extracting a character string of length N appearing in a text (referred to as an N-gram of a character) and calculating the importance. A technique for judging the above is disclosed. While the use of such character N-grams has the advantage that it is not restricted by the feature dictionary, there is a problem that linguistically meaningless character strings are extracted, and how to eliminate such character strings. Is one of the main points of the technique disclosed in Patent Document 4. However, when applying to text category classification, it does not matter whether a certain character string is linguistically meaningful, but it does not matter whether it contributes to classification. It is an interesting and unknown problem whether or not a technique that automatically learns a character string that does not contribute to classification in the learning process by learning a text classifier using the feature vector is effective.

  Non-patent document 3 is cited as a few studies on such a problem. Here, text classification performance when a feature vector having three consecutive characters as one feature is used for an English text in which words are separated, and the text classification performance of a feature vector generated using language knowledge Compare with According to Non-Patent Document 3, when character N-grams are used, the performance starts to deteriorate only by using more than 1000 features, and feature selection based on part of speech and stemming for removing word utilization parts, etc. It is reported that better performance can be obtained by using language knowledge. However, if you use a support vector machine that does not cause overfitting even if there are many features, or if the words are not shared and the linguistic knowledge mentioned above does not work effectively In the case of text, whether or not the technique using the character N-gram is effective is an interesting and unknown problem, and is a problem to be solved by the present invention.

The following documents can be referred to as documents of the prior art related to this type of text classification.
JP 2001-22727 A JP-T-2002-519766 JP 2004-348239 A Japanese Patent Laid-Open No. 11-272702 Japanese Patent Laid-Open No. 2005-247331 JP 2005-190284 A JP 2004-240517 A Japanese Patent Laid-Open No. 2004-234051 JP 2003-271616 A JP 2002-7433 A JP 2002-304401 A JP 2001-312501 A JP 2000-172691 A JP-A-11-328211 JP 11-296552 A Japanese Patent Application Laid-Open No. 11-167581 Japanese Patent Laid-Open No. 11-161671 JP-A-9-26963 JP 2003-256801 A V. Vapnik: The Nature of Statistical Learning Theory, Springer-Verlag, 1995. Vapnik: The Nature of Statistical Learning Theory, Springer-Verlag, 1995. T.A. Joachims: Text Categorization with Support Vector Machines: Learning with Many Relevant Features, Proc. of European Conference on Machine Learning, pp. 137-142, 1998. G. Neumann and S.M. Schmeier: Combining shallow text processing and machine learning in real world applications, Proc. of Machine Learning for Information Filtering, 1999. T.A. Joachims: Learning to classy text using supporting vector machines: methods, theories, and algorithms, Kluwer Academic Publishers, 2002. N. Christianiani and J.M. Shawe-Taylor: An Induction to Support Vector Machines, Cambridge University Press, 2002. Shaw-Taylor: An Introduction to Support Vector Machines, Cambridge University Press, 2002. I. Guyon et al. : Gene selection for cancer classification using support vector machines, Machine Learning, pp. 389-422, Vol. 46, 2002.

By the way, for example, text that includes character strings that are difficult to analyze morphologically, that is, when the text characteristics are different, the text classification method of the prior art is not used as it is when classifying the text. Not applicable. In particular, in a Japanese bulletin board site on the Internet, it is not possible to obtain sufficient classification accuracy with the conventional text classification method to be applied when identifying and removing inappropriate written text. The text written on the bulletin board site has the following characteristics that are different from those found in news articles and electronic mail.
(A) Many unregistered words appear in general dictionaries such as proper nouns such as personal names and product names, jargon, fuzzy characters, and pictograms.
(B) Writing is often grammatically incorrect.
(C) One entry is short. On the other hand, the number of writings is very large.

  Unlike English, in which words are shared, language processing such as morphological analysis is usually performed to extract words from Japanese text. For such language processing, linguistic knowledge such as dictionary and grammar is used explicitly or implicitly. In the case of written text on the bulletin board site, for example, the effectiveness of the language knowledge decreases due to the above-mentioned characteristics (a) and (b), and the division into incorrect morphemes and unregistered words as unknown words If it cannot be recognized frequently, the performance of morphological analysis will deteriorate significantly. Therefore, the performance of text classification using feature vectors generated from wrong morphemes is also deteriorated.

  In addition, when a support vector machine is used for learning a text classifier that determines whether writing on a bulletin board site is inappropriate, there is a problem of calculation cost required for learning. In general, the computation time required to train a classifier using a support vector machine tends to be longer than that of other learning methods. From experience, the number of training data is increased from the second to the third power. Proportional time is required. Therefore, when the number of writings on the bulletin board site increases, there arises a problem that the time required for learning increases rapidly. In order to improve the efficiency of learning, a matrix that stores the inner product between feature vectors of training data called a kernel matrix is often used. In this case, the number of training data is used to store the matrix. A memory area proportional to the square is required, and there is a problem that the memory consumption rapidly increases as the number of writings on the bulletin board site increases.

  The present invention has been made to solve the various problems as described above, and an object of the present invention is to generate a feature vector using only text without using language knowledge such as a feature dictionary. Even when a large number of feature vectors with categories are given as training data, the text classifier can be learned efficiently in terms of computation time and area, and if text with unknown categories is given, To provide a text classification processing method, a text classification processing apparatus, and a text classification processing program capable of generating a corresponding feature vector and determining a category of text using a learned text classifier.

  In order to achieve the above object, the present invention provides, as a first aspect, a text classification processing method according to the present invention, for example, a data storage device that stores text including a character string that is difficult to analyze morpheme, A text classification processing method for classifying the text, comprising a data processing device for performing text classification processing for determining whether or not the text belongs to a category, and extracting an arbitrary character string having a predetermined length or less from the text , A feature vector generation process of calculating a feature amount of the character string and generating a feature vector having the feature amount as a feature, and a training data set to which a label relating to whether the text belongs to a certain category is given in advance When given, the text of the training data set is converted into a feature vector by the feature vector generation process, and a label and Applying the feature vector to a support vector machine, learning to the support vector machine, generating a text classifier by the support vector machine, and text that is unknown whether it belongs to a certain category Is generated by the feature vector generation process, and the text classifier generated by the classifier generation process is used to determine whether the text belongs to the category. The processing with the category determination process is configured to be executed by the data processing device.

  In this case, the classifier generation process divides the training data set into a plurality of subsets, applies a support vector machine to each subset in turn, and trains a temporary classifier, A support vector is extracted from the classifier, the extracted support vector and the next subset are mixed, and the process of inputting the support vector machine again as an input to the support vector machine is repeated.

  As a second aspect of the present invention, the text classification processing apparatus according to the present invention includes, for example, a data storage device that stores text including a character string that is difficult to analyze morpheme, and whether the text belongs to a certain category. A text classification processing device for classifying the text, comprising a data processing device that performs text classification processing for determining, extracting an arbitrary character string having a predetermined length or less from the text, and calculating a feature amount of the character string When the training data set to which the feature vector generating means for generating the feature vector having the feature quantity as the feature and the training data set to which the text relating to whether or not the text belongs to a certain category is given in advance is given. The text is converted into a feature vector by the feature vector generation means, and the feature vector is supported along with the label. A classifier generating means that applies to a tol machine and performs learning on the support vector machine to generate a text classifier by the support vector machine, and when the text that is unknown whether it belongs to a certain category is given, A feature vector generating unit that generates a feature vector of the text, and a category determination unit that determines whether the text belongs to the category using the text classifier generated by the classifier generating unit. Configured.

  In this case, the classifier generation unit divides the training data set into a plurality of subsets, applies a support vector machine to each subset in turn, and learns a temporary classifier, A support vector is extracted from the classifier, the extracted support vector and the next subset are mixed, and the process of inputting the support vector machine again and inputting the support vector machine is repeated.

  As a third aspect of the present invention, the text classification processing program according to the present invention performs, for example, a text classification process for determining whether or not a text including a character string difficult to analyze morpheme belongs to a certain category by a computer. A text classification processing program to be executed, wherein an arbitrary character string having a predetermined length or less is extracted from the text, a feature amount of the character string is calculated, and a feature vector having the feature amount as a feature is generated And a training data set to which a label relating to whether or not the text belongs to a certain category is given in advance, the text of the training data set is converted into a feature vector by the feature vector generation unit, and a label And applying the feature vector to a support vector machine, A classifier generating means for learning a thin and generating a text classifier by a support vector machine, and when a text whose unknown whether or not belonging to a certain category is given, features of the text by the feature vector generating means The computer functions as a category determination unit that generates a vector and determines whether the text belongs to the category using the text classifier generated by the classifier generation unit.

  In this case, the classifier generation unit divides the training data set into a plurality of subsets, applies a support vector machine to each subset in turn, and learns a temporary classifier, A support vector is extracted from the classifier, the extracted support vector and the next subset are mixed, and the process of inputting the support vector machine again as an input to the support vector machine is repeated.

  According to the text classification processing method, the text classification processing device, and the text classification processing program of the present invention configured as described above, without using linguistic knowledge such as a feature dictionary, for text with poor language knowledge, Generate a feature vector with an arbitrary character string of a certain length or less as a feature, and let a support vector machine learn a text classifier from a feature vector set with a label indicating whether or not it falls into a category . This can be done with a small number of computational resources. Moreover, when a text whose category is unknown is given, it becomes possible to determine (predict) the category of this text with high accuracy using a learned classifier.

  Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a flowchart showing an example of a processing flow of a text classification processing method according to the present invention. In the text classification process of the present invention, as a basic process, as shown in FIG. 1, an arbitrary character string having a length that appears in text is extracted to calculate a feature amount, and these feature amounts are calculated. A feature vector generation process (P1) for generating a feature vector as a feature, a classifier generation process (P2) for generating a classifier using a support vector machine with a feature vector set to which a category is assigned as an input, Each process of the category determination process (P3) in which the classification text is converted into the feature vector by the feature vector generation process (P1) and then the category is determined by using the classifier learned by the classifier generation process (P2). I do.

  That is, in the text classification processing according to the present invention, for example, a data storage device that stores text including a character string that is difficult to perform morphological analysis, and data processing that performs text classification processing for determining whether the text belongs to a certain category. The data processing apparatus performs text classification processing for classifying the text. In this case, a feature vector generation process (P1) for extracting an arbitrary character string of a certain length or less from the text, calculating a feature amount of the character string, and generating a feature vector having the feature amount as a feature, and the text When a training data set with a label on whether or not it belongs to a certain category is given in advance, the text of the training data set is converted into a feature vector by the feature vector generation process, and the feature vector is supported together with the label Applying to a vector machine, learning to the support vector machine to generate a text classifier by the support vector machine (P2), and text that is unknown whether it belongs to a certain category or not In this case, a feature vector of the text is generated by the feature vector generation process (P1). , Using a text classifier generated by the classifier generation process, the text to perform a process of determining category determination process whether they belong to the category (P3).

  Further details will be described. When executing the text classification process, first, the given classified text is read into the main memory of the computer (step 101), and the length generated in the text by the feature generation vector process (P1) is less than a certain length. A feature vector having the character string as a feature is calculated (step 102).

FIG. 2 is a diagram showing a detailed processing flow of the feature vector generation process (P1). In the feature vector generation process (P1), it is first determined whether or not the text has been classified (step 121). If the text has been classified, an arbitrary character string having a length of N or less is determined from the text. s is extracted, and the appearance frequency TF (s, t _i ) in each text t _i of this character string s and the number of texts DF (s) in which this character string appears are calculated (step 122). The appearance frequency of an arbitrary character string having a length N or less appearing in the text can be efficiently calculated using a data structure called trie. The calculation using the data structure called trie is well known to those skilled in the art and will not be described here.

Next, thousands to tens of thousands of features are selected from all the features (step 123), and a vector having a feature quantity f (s _i , t) as a component is generated for an arbitrary t. (Step 124), this is a feature vector. Here, feature selection (step 123) is performed. This is because if there are too many features, the risk of overfitting may increase, or the calculation cost during classifier learning may become unacceptable. It is. Known features such as mutual information and chi-square test are used for feature selection.

Thus, using the selected features s ₁ ,..., S _d , each text t _i is converted into a d-dimensional feature vector (f (s ₁ , t _i ),..., F (s _d , t _i. )). f (s _j , t _i ) is a feature amount of the feature s _j , and for example, TF (s _j , t _i ) is simply used, or log (D / DF () is used for TF (s _j , t _i ). A standard index such as a TFIDF value multiplied by s _j )) (where D is the total number of texts) is used. These values can also be binarized depending on whether or not a certain threshold value is exceeded. In this case, the feature vector is a binary vector, and implementation that saves calculation resources is possible.

  Since the component of the feature vector is often zero, it is possible to use the sparse vector technique so that two small arrays are used instead of one large array. That is, the first array holds the index of the component having a non-zero value, and the second array holds the actual value of the non-zero component. In particular, when the feature vector is a binary vector, the second array is unnecessary.

  The classified text set is converted into a set of feature vectors to which a classification label is assigned through the processing described above, and then the classifier is learned in the classifier generation process (P2) using this as training data. (Step 103).

FIG. 3 is a diagram for explaining a detailed processing flow of the classifier generation process (P2). In the process of the classifier generation process (P2), first, the feature vector set to which the classification label is assigned is divided into _k subsets D ₁ ,..., D _k (step 131). This division processing is to prevent the time and memory required for training the support vector machine from becoming unacceptable because there is too much training data. It is desirable to divide the size into as large a size as possible within the allowable range of computing resources. Therefore, if learning using all training data is acceptable, k = 1 is desirable. Moreover, it is desirable to divide the divided training data so that the ratio of the data belonging to the category and the data not belonging to the category is almost equal.

Next, after the support vector set S is initialized to an empty set (step 132), the following processing is repeated for each 1 ≦ i ≦ k (steps 133 to 137). That is, let training data T be D _i ∪S (step 134), train a support vector machine using T to calculate a classifier (step 135), extract a support vector from the trained classifier, Is a new S. This process is repeated with this set of support vectors (step 136).

  The support vector machine training method is not an essential part of the present invention and will not be described. However, details can be referred to because they are described in Non-Patent Document 1, Non-Patent Document 4, and the like. Here, only the portions necessary for understanding the present invention are described below.

を分類境界として出力する。ただし、ｂ^＊は、α_ｉ ^＊＞０であるｉに対して、
ｙ_ｉｄ（ｘ_ｉ）＝１であるように決定する。 M training data sets {(y _i , x _i ) | y _i = + 1 or −1, x _i is a feature vector, 1 ≦ i ≦ M}
, The support vector machine solves a convex quadratic programming problem, and when the solution is α ₁ ^* ,..., Α _M ^* , the following hyperplane

Are output as classification boundaries. Where b ^* is i for α _i ^* > 0,
Determine that y _i d (x _i ) = 1.

Here, the training data x _{i in} which α _i ^* ≠ 0 is called a support vector, and is essential data for defining a hyperplane located closest to the classification boundary. In many cases, the number of support vectors is considerably smaller than the training data, and the classifier generation process (P2) process (step 134 to step 137) shown in FIG. Furthermore, the hyperplane obtained by using all the training data is approximated by passing fewer support vectors while having the same amount of information.

Further, K (x, x _i ) here is called a kernel function and is a function representing similarity between data. A known polynomial kernel, Gaussian kernel, or the like can be used, and a feature vector is 2 In the case of a value vector, the Boolean kernel disclosed in Patent Document 19 can also be used.

  Such a method of extracting support vectors and learning data can also be an incremental classifier learning. That is, in the initialization of S in FIG. 3 (step 132), if S is initialized as a support vector set extracted from a classifier that has already been learned, instead of making S an empty set, an existing classifier Incremental learning can be realized.

  Further, the feature selection method shown in Non-Patent Document 6 or Patent Document 19 is used to analyze the finally obtained hyperplane to remove features that do not contribute to classification, and to achieve superclassification with higher classification accuracy. It is also possible to relearn the plane.

  As described above, when unclassified text exists (step 104), the unclassified text is read (step 105), the feature vector generation process (P1) is performed (step 106), and the learned hyperplane is used. Then, the category determination process (P3) process (step 107) is performed to determine the category of the unclassified text. More specifically, in the feature vector generation process (P1), the unclassified text is converted into a feature vector x, and d (x) is calculated. If it is less than Dmin, it is classified into a category of −1, and if it is larger than Dmin and smaller than Dmax, processing for determining that the category is indefinite is performed.

  FIG. 4 is a block diagram showing an example of the hardware configuration of the text classification processing apparatus according to the present invention. As shown in FIG. 4, the text classification processing device 10 here has a hardware configuration that includes a ROM in which a system control program is incorporated, and a CPU (Central Processing Unit) 10a that performs data processing controls the entire system. Is made. A random access memory (RAM) 10b, a hard disk drive (HDD: Hard Disk Drive) 10c, a graphic processing device 10d, an input interface 10e, and a communication interface 10f are connected to the CPU 10a via a bus 10g.

  The RAM 10b temporarily stores at least a part of an OS (Operating System) program to be executed by the CPU 10a and a text classification processing program according to the present invention. The RAM 10b stores various data necessary for processing by the CPU 10a. The HDD 10c stores the OS, application programs, various data, and the like.

  A monitor 10h is connected to the graphic processing device 10d. The graphic processing device 10d displays an image for performing input / output processing on the display screen of the monitor 10h in accordance with an instruction from the CPU 10a. A keyboard 10i and a mouse 10j are connected to the input interface 10e. The input interface 10e transmits a signal sent from the keyboard 10i or the mouse 10j to the CPU 10a via the bus 10g.

  The communication interface 10f may be connected to the network 30 so that the text classification processing device according to the present invention is used as a server in the network system. Of course, it may be configured to operate as a single device. As shown in FIG. 5, when operating as a classification server 20 connected to a network, a client 21 used by a user accesses the classification server 20 via a network 30 and transmits a classified text set. The system can be utilized by requesting the construction of a classifier, sending an unclassified text set, requesting classification of text, and receiving classification categories. At this time, the text set to be used is stored in the file server 22, the client 21 designates the file name on the file server 22 to the classification server 20, and the classification server 20 It is also possible to read the specified file via the network.

  Next, each function of the processing module provided in the text classification processing apparatus 10 will be described. FIG. 6 is a functional block diagram of the text classification processing device. The text classification processing device 10 has feature vector generation means (B1), classifier generation means (B2), and category determination means (B3) as processing modules. is doing.

  When a classified text set is given, the feature vector generation means (B1) extracts an arbitrary character string of a certain length or less appearing in the text, and then sets the feature character string to several thousand to several tens of thousands Refine to. Then, a feature amount is calculated for each text and each feature, and a feature vector having this feature amount as a component is generated. Further, when an unclassified text set is given, the feature vector generation means (B1) calculates a feature amount for the already selected feature and generates a feature vector having this as a component.

  The classifier generation means (B2) constructs a classifier (B4) using a support vector machine from the feature vector set to which the classification label is assigned, and stores it in the RAM 10b or the HDD 10c. At this time, as described above, if the feature vector set is too large, it is divided into a plurality of subsets, and a support vector machine is applied to each subset in turn to learn a temporary classifier. After that, the support vector is extracted from the classifier, the extracted support vector and the next subset are mixed, and the process of inputting the support vector machine again can be repeated.

  The category determination means (B3) generates a feature vector of the text by the previous period feature vector generation means (B1) when an unclassified text is given, and the classifier generated by the classifier generation means (B2). Is used to determine whether this text belongs to a category.

  FIG. 7 is a flowchart of a program when the text classification process according to the present invention is executed on the classification server 20 as shown in FIG. In this process, as shown in FIG. 7, when there is a learning request from the client 21, the classified text is read (S1), a feature vector is generated from the text by a feature vector generation step (S2), and a classifier generation step ( A classifier is generated by S3). Then, after storing the generated classifier in the RAM 10b or the HDD 10c, the client 21 is notified of the end of processing (S5), and waits for the request again. If there is a classification request, unclassified text is read (S6), a feature vector is generated by a feature vector generation step (S3), a classifier stored in the RAM 10b or HDD 10c is read, and then a category determination step. The category of uncategorized text is determined (predicted) by (S8), transmitted to the client 21 (S9), and waits again.

  The classification server 20 in which this program is installed constitutes a text classification processing device that functions as a feature vector generation unit, a classifier generation unit, and a category determination unit by executing the processing of each step.

  As described above, the text classification processing device according to the present invention is inappropriate for a large amount of text that is difficult to apply general language knowledge, such as written text on an Internet bulletin board. It is used as a text filtering device that can identify and remove simple writing with high efficiency and high accuracy.

It is a flowchart which shows an example of the processing flow of the text classification | category processing method which concerns on this invention. It is a figure explaining the feature vector generation process of the text classification processing method by this invention. It is a figure explaining the classifier production | generation process of the text classification processing method by this invention. It is a block diagram which shows an example of the hardware constitutions of the text classification | category processing apparatus by this invention. 1 is a system configuration diagram when a text classification processing device according to the present invention is operated on a network as a classification server. FIG. It is a functional block block diagram of the text classification | category processing apparatus by this invention. It is a flowchart which shows an example of the processing flow in the case of operating the text classification processing program by this invention on a classification server.

Claims (6)

A text classification process for classifying the text, comprising a data storage device that stores text including a character string that is difficult to perform morphological analysis, and a data processing device that performs text classification processing for determining whether the text belongs to a certain category. A method,
Extracting an arbitrary character string having a predetermined length or less from the text, calculating a feature amount of the character string, and generating a feature vector having the feature amount as a feature;
When a training data set to which a label relating to whether or not the text belongs to a certain category is given in advance, the text of the training data set is converted into a feature vector by the feature vector generation process, and the feature together with the label Applying a vector to a support vector machine, learning to the support vector machine, and generating a text classifier by the support vector machine; and
When text that is unknown whether it belongs to a certain category or not is given, a feature vector of the text is generated by the feature vector generation process, and the text classifier generated by the classifier generation process is used to generate the text A text classification processing method, characterized in that a data processing apparatus executes a process with a category determination process for determining whether or not an image belongs to the category. The text classification processing method according to claim 1,
The classifier generation process includes:
Divide the training data set into multiple subsets, apply a support vector machine to each subset in turn, train a temporary classifier, then extract and extract support vectors from the classifier A text classification processing method characterized by repeating the process of mixing the supported support vector and the next subset and using it again as an input to the support vector machine. A text classification process for classifying the text, comprising a data storage device that stores text including a character string that is difficult to perform morphological analysis, and a data processing device that performs text classification processing for determining whether the text belongs to a certain category. A device,
Extracting an arbitrary character string having a predetermined length or less from the text, calculating a feature amount of the character string, and generating a feature vector having the feature amount as a feature;
When a training data set to which a label relating to whether or not the text belongs to a certain category is given in advance, the text of the training data set is converted into a feature vector by the feature vector generation means, and the feature is combined with the label. Classifier generating means for applying a vector to a support vector machine, learning the support vector machine, and generating a text classifier by the support vector machine;
When text that is unknown whether it belongs to a certain category or not is given, the feature vector generation unit generates a feature vector of the text, and the text classifier generated by the classifier generation unit uses the text A category determination means for determining whether or not the file belongs to the category;
A text classification processing apparatus comprising: The text classification processing device according to claim 1,
The classifier generating means includes:
Divide the training data set into multiple subsets, apply a support vector machine to each subset in turn, train a temporary classifier, then extract and extract support vectors from the classifier A text classification processing device that repeats the process of mixing the support vector and the next subset, which are input to the support vector machine again. A text classification processing program for executing text classification processing by a computer to determine whether or not text including a character string that is difficult to perform morphological analysis belongs to a category,
Extracting an arbitrary character string having a predetermined length or less from the text, calculating a feature amount of the character string, and generating a feature vector having the feature amount as a feature;
When a training data set to which a label relating to whether or not the text belongs to a certain category is given in advance, the text of the training data set is converted into a feature vector by the feature vector generation means, and the feature is combined with the label. Classifier generating means for applying a vector to a support vector machine, learning the support vector machine, and generating a text classifier by the support vector machine;
When text that is unknown whether it belongs to a certain category or not is given, the feature vector generation unit generates a feature vector of the text, and the text classifier generated by the classifier generation unit uses the text A category determination means for determining whether or not the file belongs to the category;
Text classification processing program that makes a computer function as a computer. The text classification processing program according to claim 1,
The classifier generating means includes:
Divide the training data set into multiple subsets, apply a support vector machine to each subset in turn, train a temporary classifier, then extract and extract support vectors from the classifier A text classification processing program that repeats the process of mixing the support vector and the next subset and using it as an input to the support vector machine again.

Images