JP4567025B2 - Text classification device, text classification method, text classification program, and recording medium recording the program - Google Patents

Description

  The present invention relates to a text classification apparatus and method and a program for classifying text into any one category of a predetermined category set, and a recording medium on which the program is recorded.

  As a conventional technique for calculating the degree of association between words, there is a method of Non-Patent Document 1. In this method, a co-occurrence frequency between words in a text is calculated to create an example of a co-occurrence frequency between words. Each row vector of the matrix represents a pattern of the corresponding word that co-occurs with other words. Since words with similar meanings tend to co-occur with common words, the corresponding patterns also tend to be similar. Therefore, the degree of association between words is calculated as the cosine of the corresponding vector.

  In the method of Non-Patent Document 2, a co-occurrence frequency between words and semantic attributes in a text is calculated using a dictionary in which semantic attributes are attached to words, and a co-occurrence frequency matrix between words and semantic attributes is created. Each row vector of the matrix represents a pattern that co-occurs with the semantic attribute of the corresponding word. Since words with similar meanings tend to co-occur with common semantic attributes, the corresponding patterns also tend to be similar. Therefore, the degree of association between words is calculated as the cosine of the corresponding vector.

  Both the method of Non-Patent Document 1 and the method of Non-Patent Document 2 raise the accuracy of the degree of association between vectors by converting the co-occurrence frequency matrix into a matrix with a singular value decomposition and a reduced number of columns.

Given a set of categories associated with a set of words, the task of classifying text into one of the categories uses the vectors generated above and includes categories and text vectors. The centroid of the word vector is calculated, the cosine between the centroid vectors is set as the relevance between the text and the category, and the category with the high relevance is set as the category to be classified.
H. Schutze, Dimensions of Meaning, Proc. of Supercomputing '92, pp. 786-796, 1992. Katsuhito Bessho, Toshiro Uchiyama, Ryoji Kataoka: Concept-based extension based on co-occurrence between words and semantic attributes, Information Processing Society of Japan Research Report, Vol. SIG-ICS 144, pp. 29-34, 2006.

  In the above prior art, a cosine between centroid vectors is used. In the cosine between vectors, as a word highly related to one word, not only a word corresponding to a higher / lower concept of the word but also a word having a sibling relationship having the same higher concept is derived. As a result, there is a problem that the text is misclassified into a category corresponding to the sibling concept of the content. For example, in cosine, a word having a common superordinate concept “disease” such as “heart disease” is also derived for the word “psychiatric disease”. As a result, the text related to “psychiatric disease” may be erroneously classified into the category related to “heart disease”.

  The present invention has been conceived in order to solve this problem, and an object of the present invention is to improve the accuracy of a task of classifying text into one of categories in a predetermined category set. A text classification device, a text classification method, a text classification program, and a recording medium on which the program is recorded.

  The text classification device according to claim 1 for solving the above-described problem is a text classification device for classifying a text into any category of a predetermined category set, wherein a corpus is input, and an arbitrary For each word A, each coordinate corresponds to a word or word semantic attribute, and the value of the coordinate is a relative value of the co-occurrence frequency of the word A and the word or word semantic attribute corresponding to the coordinate A word vector generating means for generating a vector, a word vector obtained by the word vector generating means for the word B for an arbitrary word B, a word vector obtained by the word vector generating means for an arbitrary word C, Calculating the relevance between the word B and the word C according to the ranking in the ranking. It is characterized in that a between relevance calculating means.

  The text classification method according to claim 7 is a text classification method for classifying text into any category of a predetermined category set, wherein the word vector generation means receives a corpus as an input, For each word A, each coordinate corresponds to a word or word semantic attribute, and the value of the coordinate is a relative value of the co-occurrence frequency of the word A and the word or word semantic attribute corresponding to the coordinate A word vector generation step for generating a vector, and an inter-word relevance calculation means for the arbitrary word B, the word vector obtained by the word vector generation means for the word B, and the word vector generation for the arbitrary word C The Cullback-Roller distance between the word vector obtained by the means is calculated, the word C is ranked in ascending order of the distance, and the word is determined according to the ranking in the ranking. It is characterized by an inter-word relevance calculating a degree of relevance between the words C and.

The text classification apparatus according to claim 3, wherein the word vector generation unit performs morphological analysis on the text and specifies a word necessary for processing, and an arbitrary word based on an analysis result of the morpheme analysis unit. For each pair of words, or a pair of any word and any word meaning attribute , the frequency of counting the events that the pair co-occurs in each of one or more predetermined ranges in the text is calculated over the entire text. A co-occurrence frequency matrix obtained by associating each word with a vector whose value of each coordinate is the frequency calculated for the word and the word or word semantic attribute pair corresponding to the coordinate. a co-occurrence matrix generating means for generating for the value of each coordinate of the row vector of the co-occurrence matrix is generated by the generating means co-occurrence frequency matrix, the relative frequency calculating means for converting the relative frequency, the It is characterized in that was example.

Further, in the text classification method according to claim 9, in the word vector generation step, the morpheme analysis unit performs a morpheme analysis on the text and specifies a word necessary for processing, and a co-occurrence matrix generation unit includes: From the analysis result of the morphological analysis step, the pair co-occurs in any one or a plurality of predetermined ranges in the text with respect to an arbitrary word pair or an arbitrary word and an arbitrary word semantic attribute pair. The frequency at which the event is counted over the entire text is calculated, and for each word, the value of each coordinate is the frequency calculated for the word and the word or word semantic attribute pair corresponding to the coordinate. co of the co-occurrence matrix generating step of generating a co-occurrence frequency matrix obtained by associating a vector, the relative frequency calculating means, produced by the co-occurrence matrix generating step The value of each coordinate of the row vector of frequency matrix, and the relative frequency calculating step of converting the relative frequency, comprising the.

  The text classification apparatus according to claim 4 is characterized in that the relevance in the inter-word relevance calculation means calculates the reciprocal of the ranking order.

  The text classification method according to claim 10 is characterized in that the relevance in the inter-word relevance calculation step is calculated by the reciprocal of the ranking order.

  The text classification apparatus according to claim 5 is characterized in that the relevance in the inter-word relevance calculation means is calculated based on a cosine between the word vectors of the words B and C.

  The text classification method according to claim 11 is characterized in that the relevance in the inter-word relevance calculation step is calculated based on a cosine between the word vectors of the words B and C.

  According to the above configuration, the word C is ranked in ascending order of the cullback / railer distance from the word B to the arbitrary word C with respect to the arbitrary word B. C is ranked. Although the magnitude of the Culbach-Riverer distance at the top of the ranking result for each word B is different, the top concept is always occupied at the top of the ranking regardless of this magnitude. Therefore, the degree of relevance calculated by the rank in the ranking accurately represents the degree of [C if B].

  Further, the text classification device according to claim 2 receives a set of categories associated with a certain word set and an arbitrary text as input to the text classification device according to claim 1, and for each category, Text-category relevance calculation for calculating the relevance between the text and the category based on the relevance calculated by the inter-word relevance calculation means between the word D and the word E in the category It is characterized by further providing means.

  The text classification apparatus according to claim 6, wherein the text-category relevance calculation unit performs morphological analysis on each character string Zpq of a category Cp associated with an OR-linked character string, and each character string Zpq is converted into a set of sets of different required word end forms and their frequencies in each character string, morphological analysis is performed on an arbitrary text U, and a set of different required word end forms in the text U and their set of frequencies And the word D in the text U and the category based on the relevance calculated by the inter-word relevance calculating means between the word D in the text U and the word E in the category Cp. The degree of association between the character string Zpq in Cp is calculated, and the degree of association between an arbitrary text U and an arbitrary category Cp is calculated.

  The text classification method according to claim 8 is the text classification method according to claim 7, wherein the text-category relevance calculation means inputs a set of categories associated with a set of words and arbitrary text. For each category, the degree of association between the text and the category is calculated based on the degree of association calculated by the inter-word association degree calculating means between the word D in the text and the word E in the category. The method further includes a step of calculating the degree of association between texts and categories.

  The text classification method according to claim 12, wherein the text-category relevance calculation step performs morphological analysis on each character string Zpq of the category Cp associated with an OR-linked character string, and Zpq is converted into a set of combinations of different required word end forms and their frequencies in each character string, morphological analysis is performed on an arbitrary text U, and a set of different required word end forms in the text U and their set of frequencies And the word D in the text U and the category based on the relevance calculated by the inter-word relevance calculating means between the word D in the text U and the word E in the category Cp. The degree of association between the character string Zpq in Cp is calculated, and the degree of association between an arbitrary text U and an arbitrary category Cp is calculated.

  According to the above configuration, since the relevance between text categories is calculated from the relevance representing the degree of [E if D] between the word D in the text and the word E in the category, this text category The degree of interrelationship accurately represents the degree of [the category if it is the text]. By classifying the text based on this degree of association, it is possible to prevent the text from being misclassified into a category corresponding to the sibling concept of the content, and the classification accuracy is improved.

  The text classification program according to claim 13 is a text classification program for causing a computer to function as the word vector generation means and the inter-word relevance calculation means according to any one of claims 1 to 6. It is said.

  A text classification program according to claim 14 is a text classification program for causing a computer to function as the text-category relevance calculating means according to any one of claims 2 to 6.

  A recording medium according to claim 15 is a computer-readable recording medium on which the text classification program according to claim 13 or 14 is recorded.

(1) According to the inventions described in claims 1 to 13 and 15, in order to rank the word C with respect to the arbitrary word B in ascending order of the Cullback-Lailer distance from the word B to the arbitrary word C, Word C is ranked in descending order of the degree of superordinate concept of word B. Although the magnitude of the Culbach-Riverer distance at the top of the ranking result for each word B is different, the top concept is always occupied at the top of the ranking regardless of this magnitude. Therefore, the degree of relevance calculated by the rank in the ranking accurately represents the degree of [C if B].
(2) According to the inventions of claims 2 to 15, the relationship between the text and the category is represented from the relevance indicating the degree of [E if D] between the word D in the text and the word E in the category. In order to calculate the degree, this text-category relevance accurately represents the degree of [the category if it is the text]. By classifying the text based on this degree of association, it is possible to prevent the text from being misclassified into a category corresponding to the sibling concept of the content, and the classification accuracy is improved.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments.

  FIG. 1 shows an example of the configuration of a text classification apparatus in the embodiments of claims 1 and 2 of the present invention. Claim 1 comprises word vector generation means 101 and word-to-word relevance calculation means 102, and claim 2 comprises text-category relevance calculation means 103.

  Each function to be described later included in the word vector generation unit 101, the inter-word relevance calculation unit 102, and the text / category relevance calculation unit 103 is achieved by a computer, for example.

  The word vector generation unit 101 receives a corpus, and for an arbitrary word A, each coordinate corresponds to a word or a word semantic attribute, and the value of the coordinate is the word A and the word or word corresponding to the coordinate A word vector that is a relative value of the co-occurrence frequency with the semantic attribute is generated. FIG. 2 is a detailed configuration example of the word vector generation unit 101, which includes a morpheme analysis unit 201, a co-occurrence matrix generation unit 202, a relative frequency calculation unit 203, and a word dictionary 204.

  In FIG. 2, a morphological analysis unit 201 performs morphological analysis on a text. This morphological analysis is performed with reference to the word dictionary 204, and FIG. 3 shows an example of the contents of the dictionary. In the figure, in the word dictionary, one record is information related to one word, and one record is composed of three items separated by commas. The first item is a word notation, and the second item is the part of speech information of the word. The third item is a semantic attribute of the word. In the word vector generation unit 101, when the co-occurrence between words / words is taken, the semantic attribute of the third item may be omitted. In addition, an end form may be registered for a use word.

  The word semantic attribute represents a semantic category to which the word belongs. A semantic category is generally a concept that abstracts things. This is generally obtained by a person examining the meaning of individual words. The set of semantic categories has a semantic system as shown in FIG. 4 as an example. In FIG. 4, each semantic category is expressed as a word, but the semantic category itself is not necessarily expressed as a word. Each semantic category is given an ID for specifying it. In this embodiment, this ID is identified with a semantic attribute for convenience.

  Although the semantic system itself represents the upper and lower conceptual systems, they are created manually, so the system is generally unsuccessful because of the great effort of creation and the arbitrary nature of the creator. Great integrity. For example, there are cases where words such as “psychiatric disease” and “depression” are classified in the semantic attribute of “disease”, and these words are not in a higher / lower relationship. A word belonging to a semantic attribute that is not in a higher / lower relationship may actually have a higher / lower relationship. As described above, since the semantic system is created manually and includes valuable information on the meaning of words, it does not accurately and comprehensively represent the upper / lower relationship.

  In the word dictionary of FIG. 3, one or more semantic attributes generally correspond to content words. In FIG. 3, a plurality of semantic attributes are separated by colons. When a single word has a plurality of semantic attributes, the semantic attributes may be arranged in the order in which they are frequently used. When a new word is registered in the word dictionary, it is generally given by a person who examines the part of speech and meaning of the word, which existing semantic attribute corresponds to the word.

  FIG. 5 is an example of text input to the morpheme analysis unit 201, and FIG. 6 is an example of a morpheme analysis result of the text of FIG. The morphemes are separated by “/”. Each morpheme is composed of a word notation, an end form, a part of speech information, a semantic attribute, and a necessary word flag, and each is delimited by “,”. When the final form is not registered in the word dictionary 204, the final form is derived from the word notation and the part of speech information after the morphological analysis. For words that do not have a closing form, the word notation is the closing form. The necessary word flag is null after the morphological analysis. In the word vector generation unit 101, when a word / word co-occurrence is taken, the semantic attribute column may not be included in the morphological analysis result.

  Also, in the case of co-occurrence between words and semantic attributes, a word dictionary for morphological analysis and a dictionary storing a set of pairs of words and their semantic attributes are separate from the processing process. When it is necessary to acquire a semantic attribute of a word, the latter dictionary may be searched for the word to acquire a corresponding semantic attribute. In the following description, the semantic attribute is also stored in the word dictionary for morphological analysis, and the semantic attribute is also output in the morphological analysis result.

  Next, by referring to the unnecessary word table and the unnecessary part-of-speech table, it is determined whether or not the morpheme in the morpheme analysis result is a word necessary for the subsequent processing. If it is a necessary word, the necessary word flag is set to 1. If it is not a necessary word, the necessary word flag is set to 0. FIG. 7 is an example of the unnecessary word table. The ending form of each unnecessary word is described as one record. FIG. 8 is an example of the unnecessary part-of-speech table. Each part of speech information that is unnecessary is described as one record. If the end form of the target morpheme matches a record in the unnecessary word table, or if the part of speech information of the target morpheme matches a record in the unnecessary part of speech table, it is determined that the morpheme is not a required word. By this processing, the morphological analysis result of FIG. 6 becomes as shown in FIG.

  The co-occurrence matrix generating unit 202 acquires different sets of necessary word termination forms from the morpheme analysis result obtained by the morpheme analyzing unit 201. Next, in the case of co-occurrence between words and semantic attributes, the co-occurrence between a set of different required forms of the required words (referred to simply as a word set) and a semantic attribute set as shown in FIG. Generate a frequency matrix. Each row in the co-occurrence frequency matrix corresponds to one word, and each column corresponds to one semantic attribute. Each row vector is a vector in which each coordinate of the corresponding word corresponds to a semantic attribute, and the value of the coordinate is a co-occurrence frequency between the word and the semantic attribute. Set all coordinate values of each row vector to 0.

  In addition, when taking the co-occurrence between words, the difference in the final form of the necessary word is associated with each column. At this time, in order to reduce the amount of calculation in subsequent processing, different sets of necessary word termination forms corresponding to each column may be limited to only high-frequency words in the input text.

  Next, in the case of co-occurrence between words and semantic attributes, a range in the text that is a target of processing for calculating the frequency of co-occurring words and semantic attributes is determined. The predetermined range includes a sentence, a paragraph, a string of a predetermined number of words, and the like. When the predetermined range is one sentence, the first sentence in the text is first processed. When the processing related to the processing target sentence is completed, the next sentence is set as the processing target. If the process related to the last sentence is completed, there is no sentence to be processed, and the process of the co-occurrence matrix generating unit 202 is ended. The same applies to the case where the predetermined range is other.

  Processing in the range of each text to be processed is performed as follows.

  First, the frequency of semantic attributes of the end forms of necessary words in the range (if there are multiple identical words, they are different) is counted. From the morphological analysis result of one range of text displayed in FIG. 9, the frequency hash that is a set of the semantic attribute and its frequency in FIG. 11 is obtained.

  Here, in the word dictionary 204, when a plurality of semantic attributes in one word are arranged in a frequently used order, and the order of semantic attributes in the morpheme in the morpheme analysis result also inherits it, the meaning in the morpheme Only the number of semantic attributes specified from the beginning in the attribute column may be counted.

  Next, the following processing is performed on the final form of each necessary word in the range (if there are multiple identical words, they are different). The frequency of the semantic attribute in the frequency hash is added to the coordinate value corresponding to each semantic attribute in the frequency hash of the row vector corresponding to the end form of the required word in the co-occurrence frequency matrix.

  Here, the co-occurrence of the end form of the necessary word and the semantic attribute in the morpheme may not be counted. That is, a temporary frequency hash obtained by subtracting 1 from the coordinate value corresponding to the semantic attribute of the count target held by the end form of the necessary word is generated from the frequency hash, and this temporary frequency is generated. You may make it add the frequency of each semantic attribute in a hash. For example, in the processing in the morphological analysis result of one range of the text displayed in FIG. 9, the semantic attribute 11, 91 is held for the necessary word termination type “US”, so the frequency hash of FIG. The temporary frequency hash of FIG. 12 is obtained. When the co-occurrence frequency matrix is in the state of FIG. 10 and the morphological analysis result of one range of text displayed in FIG. 9 is processed, the co-occurrence frequency matrix of FIG. 13 is obtained.

  When the processing for all the ranges is completed, a matrix in which the co-occurrence frequency between an arbitrary word and an arbitrary semantic attribute in the input text is recorded, and each row vector in this matrix is obtained by the co-occurrence matrix generating unit 202. The vector of the corresponding word to be obtained.

  The above-described calculation of the co-occurrence frequency between a word and a semantic attribute in one range can be executed without any process of counting the frequency of words. Therefore, the generation of the co-occurrence frequency matrix between words and semantic attributes can be performed without intervening the process of generating the inter-word co-occurrence frequency matrix.

  Here, in one range, when the end form M of a certain required word appears m times and the semantic attribute N appears n times, the co-occurrence frequency of M and N in the range is expressed as m × n or m × ( Although n-1), the co-occurrence frequency in the range can be set to 1 even if M and N appear multiple times in the same range.

  The co-occurrence frequency calculation of the co-occurrence matrix generation unit 202 in the case of taking the co-occurrence between words / words is described in the above description in the case of taking the co-occurrence between words / semantic attributes (the end form of the necessary word). The “semantic attribute” is performed as a difference in the final form of the necessary word corresponding to the column of the co-occurrence frequency matrix.

  The relative frequency calculation unit 203 converts the value of each coordinate of the vector generated by the co-occurrence matrix generation unit 202 into a relative frequency.

It is assumed that the vector of a certain word generated by the co-occurrence matrix generating unit 202 is (a ₁ , a ₂ , ... , _An ). By the relative frequency calculation means 203, this word vector is

Is converted to Each x _i (1 ≦ i ≦ n) is a relative frequency with respect to the sum of the values of the corresponding coordinates in the vector before conversion.

  Therefore, the converted vector can be regarded as a probability distribution in which each coordinate is a random variable and the coordinate value is a probability value.

  The word-to-word relevance calculating unit 102 is configured to, for an arbitrary word X, the word vector obtained by the word vector generating unit 101 for the word X and the word vector obtained by the word vector generating unit 101 for the arbitrary word Y. Kullback-Riverer distance (distance between two probability distributions) between and the word Y is ranked in ascending order of the distance, and the degree of association between the word X and the word Y is calculated based on the ranking in the ranking. . FIG. 14 is a detailed flowchart of the inter-word relevance calculation means 102.

Hereinafter, in the description of the inter-word relevance calculation means 102, it is assumed that each coordinate of the vector is based on a configuration corresponding to a semantic attribute. The same applies to a configuration in which each coordinate of the vector corresponds to a different end form of the required word.
(Step S1)
Among the words not processed so far, one word X to be processed is determined. If there is, the process proceeds to step S2, and if not, the processing of the inter-word relevance calculation unit 102 is terminated.
(Step S2)
Among words that have not been processed so far, one word Y to be processed is determined. If there is, the process proceeds to step S3, and if not, the process proceeds to step S4.
(Step S3)
For a pair of words X and Y, X and Y vectors v (X) and v (Y)
v (X): = (x ₁ , x ₂ , ... , x _n )
v (Y): = (y ₁ , y ₂ , ... , y _n )
When the Cullback-Lailer distance P (X, Y) from X to Y is

  Calculate as

  here,

It is defined as However, in this definition, if there is at least one coordinate i such that x _i ≠ 0 and y _i = 0, P (X, Y) = ∞, so a word pair in a higher / lower relationship On the other hand, there are many cases where the distance is ∞, and the recall rate of the word pairs in the upper / lower relationship is lowered.

  To always make the distance value finite, do as follows.

  P (X, Y) represents an expected value of information loss when v (X) is an actual distribution and approximated by v (Y).

Is particularly large when y _i is 0 or close to 0 and x _i is much larger than y _i . When Y is a superordinate concept of X, generally, if X co-occurs with the semantic attribute of coordinate i, Y tends to co-occur with the semantic attribute of coordinate i. Therefore, in the case where Y is a superordinate concept of X, the Cullback-Lailer distance P (X, Y) tends to be smaller than in the case where Y is not a superordinate concept of X.

  The Euclidean distance when the vector length is normalized to 1 is equivalent to the cosine similarity. In this distance scale, since the distance is small when the difference between the vector values is small, not only the words related to the upper and lower levels but also the words related to the siblings are determined to be relatively small in distance.

  If the Cullback-Liber distance is used, when the word X is designated, it becomes possible to detect a word Y having a relatively small Cullback-Roller distance P (X, Y) as a higher concept word of X. .

After the process of step S3 is complete | finished, it progresses to step S2.
(Step S4)
The word Y is ranked in ascending order of P (X, Y). This is the order in which the degree that is a superordinate concept of the word X is large. FIG. 15 is a ranking of the word Y when X is “psychiatric”.

  In order to reduce the calculation amount of the process of calculating P (X, Y) for an arbitrary word pair X, Y, a set of words Y, for example, a word set group associated with a classification destination category, You may limit to what merged the high frequency word set in a corpus.

After the process of step S4 is complete | finished, it progresses to step S5.
(Step S5)
For an arbitrary word Y, when the rank of Y is m, the degree of association E (X, Y) between X and Y representing the degree of [Y if X] is taken as an example.

  Calculate as

  The magnitude of the Cullback-Ribler distance at the top of the ranking results for each word X is different. However, the Cullback-Roller distance P (X, Y ′) for the word Y ′, which is a superordinate concept of X, is larger than the Cullback-Riverer distance P (X, Y ″), for the word Y ″, which is not a superordinate concept of X. Since it is small, the higher ranking is always occupied by the higher concept regardless of the difference in distance for each X. Therefore, the rank in the ranking represents the degree that is a superordinate concept of X. Therefore, the degree of relevance calculated based on this rank accurately represents the degree of [Y if X].

  Here, for example, a cosine between X and Y vectors v ′ (X) and v ′ (Y) obtained by the methods of Non-Patent Document 1 and Non-Patent Document 2.

In combination with CO ′ (X, Y) = 0.5 · CO (X, Y) +0.5, which is converted to a value of 0 or more and 1 or less,
E ′ (X, Y) = α · E (X, Y) + (1−α) · CO ′ (X, Y) (0 ≦ α ≦ 1)
In other words, E ′ (X, Y) is a relevance degree that has the tendency of associations of E (X, Y) and CO (X, Y). This degree of association may be the final degree of association derived by the inter-word degree of association calculation means 102.

  After the process of step S5 is complete | finished, it progresses to step S1.

  The text-category relevance calculating means 103 receives a set of categories associated with a set of words and an arbitrary text as input, and for each category, a word X in the text and a word in the category The degree of association between the text and the category is calculated based on the degree of association calculated by the inter-word association degree calculating means 102 with Y.

It is assumed that the following sp character string Z _pq (1 ≦ q ≦ sp) OR combination is associated with the category C _p (1 ≦ p ≦ h).

  As an example of the above, there is a category “TV program OR radio program”.

Each character string Z _pq (1 ≦ q ≦ sp) is subjected to morphological analysis.

Each character string Z _pq is converted into a set of sets of different required word termination forms Y _pqr (1 ≦ r ≦ t _pq ) and its frequency TF _pqr (1 ≦ r ≦ t _pq ), as follows: Expressed in

  From the category “TV program OR radio program”, “(TV: 1, program: 1) OR (radio: 1, program: 1)” is obtained.

A morphological analysis is performed on an arbitrary text U. Different from the necessary word termination form, it is converted into a set of a set of X _ug (1 ≦ g ≦ tu) and its frequency TFug (1 ≦ g ≦ tu) and expressed as follows.

The morpheme analysis for each character string Z _pq of the category C _p and the text U is performed by the morpheme analysis unit 201 of the word vector generation unit 101.

When an arbitrary text U is fixed, the relation E (U, C _p ) between U and C _p representing the degree of [Category C _p if text U] for an arbitrary category C _p is shown in FIG. Calculate by flow.
(Step S11)
Among the categories not processed so far, one category C _p to be processed is determined. If there is, the process proceeds to step S12, and if not, the process ends.
(Step S12)
One character string Z _pq to be processed is determined from among character strings in category C _p that have not been processed so far. If there is, the process proceeds to step S13, and if not, the process proceeds to step S16.
(Step S13)
One word X _ug to be processed is determined from words in the text U that have not been processed so far. If there is, the process proceeds to step S14, and if not, the process proceeds to step S15.
(Step S14)
The degree of relevance E (X _ug , Z _pq ) between X _ug and Z _pq representing the degree of [Z _pq if X _ug ] is calculated by the following equation.

Here, when E (X _ug , Y _pqr ) does not exist, TF _pqr = 0, and when E (X _ug , Z _pq ) has 0 denominator, E (X _ug , Z _pq ) = 0.

The relation E (X _ug , Y _pqr ) indicating the degree of [Y _pqr if X _ug ] between the word X _ug in the text U and the word Y _pqr in the category C _p The relevance calculation means 102 performs calculation processing.

After the process of step S14 is complete | finished, it progresses to step S13.
(Step S15)
The degree of association E (U, Z _pq ) between U and Z _pq representing the degree of [Z _pq if U] is calculated by the following equation.

Here, when the denominator of E (U, Z _pq ) is 0, E (U, Z _pq ) = 0.

After the process of step S15 is complete | finished, it progresses to step S12.
(Step S16)
The degree of association E (U, C _p ) between U and C _p representing the degree of [C _p if U] is calculated by the following equation.

  After the process of step S16 is complete | finished, it progresses to step S11.

The calculation in step S14 is performed by calculating the relevance E (X _ug , Y _pqr between the word X _ug in the text U and the word Y _pqr in the category C _p , which is [Y _pqr if X _ug ]. ) (That is, the relevance calculated by the inter-word relevance calculation means 102), and E (U, Cp) is composed of E (X _ug , Y _pqr ). C _p ] is accurately expressed.

In the above, instead of the frequency TF _i corresponding to each different word Wi, the TFIDF _i of the word represented by the following expression may be used.

Here, DN is the number of documents in a certain corpus, and ON _i is the number of appearing documents in the corpus of the different word W _i . In step S14, if the TFIDF _pqr is absent and TFIDF _pqr = 0, in step S15, if there is no TFIDF _ug and TFIDF _ug = 0.

In this way, E (U, C _p ) is obtained for each category Cp. Text U can be classified into C _p with the largest E (U, C _p ). Further, E (U, C _p ) can be classified into C _p having a value equal to or greater than a certain threshold. Since E (U, C _p ) is a degree of [C _p if U,] the text is prevented from being misclassified into the category corresponding to the sibling concept of the content, and the accuracy of classification is improved.

  It is possible to construct the processing in the above-described embodiment as a program, install the program from a communication line or a storage medium, and implement it by means such as a CPU.

  That is, it is also possible to supply a recording medium recording the program to a system or apparatus, and to read and execute the program stored in the recording medium by a CPU (MPU) of the system or apparatus. In this case, the program itself read from the recording medium realizes the functions of the above-described embodiment, and examples of the recording medium on which the program is recorded include CD-ROM, DVD-ROM, CD-R, CD- There are RW, MO, and HDD.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to language processing technology.

The lineblock diagram of the text classification device of the example of one embodiment of the present invention. The block diagram of the word vector production | generation means in the text classification device of the example of 1 embodiment of this invention. Explanatory drawing which shows an example of the content of the word dictionary in the word vector production | generation means of one embodiment of this invention. Explanatory drawing which shows the collection of the semantic category used by one embodiment of this invention. Explanatory drawing which shows an example of the text input into the morphological-analysis means in one embodiment of this invention. Explanatory drawing which shows an example of the result in the middle of the morphological analysis in one embodiment of this invention. Explanatory drawing which shows an example of the unnecessary word table in one embodiment of this invention. Explanatory drawing which shows an example of the unnecessary part of speech table in one embodiment of this invention. Explanatory drawing which shows an example of the final result of the morphological analysis in one embodiment of this invention. Explanatory drawing which shows an example of the co-occurrence frequency matrix between the word set and semantic attribute set in the example embodiment of the present invention. Explanatory drawing which shows an example of the frequency hash in one embodiment of this invention. Explanatory drawing which shows the other example of the frequency hash in one embodiment of this invention. 9 illustrates a co-occurrence frequency matrix between a word set and a semantic attribute set according to an exemplary embodiment of the present invention, and illustrates a co-occurrence frequency matrix when the morpheme analysis result of FIG. 9 is processed and the frequencies of frequency hashes are added. Illustration. The flowchart showing the process sequence of the related degree calculation means between words in the example of 1 embodiment of this invention. Explanatory drawing showing the ranking of the word Y when the word X is made into "psychiatric disease" in the word related degree calculation means of one example of this invention. The flowchart showing the process sequence of the text-category relation degree calculation means in one embodiment of the present invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 101 ... Word vector generation means, 102 ... Inter-word relevance calculation means, 103 ... Text-category relevance calculation means, 201 ... Morphological analysis means, 202 ... Co-occurrence matrix generation means, 203 ... Relative frequency calculation means, 204 ... Word dictionary.

Claims (15)

A text classification device for classifying text into any category of a predetermined category set,
With a corpus as input, for any word A, each coordinate corresponds to a word or word semantic attribute, and the value of the coordinate is the co-occurrence frequency of the word A and the word or word semantic attribute corresponding to the coordinate A word vector generation means for generating a word vector that is a relative value of
For an arbitrary word B, a Cullback-Roller distance between the word vector obtained by the word vector generation means for the word B and the word vector obtained by the word vector generation means for the arbitrary word C is A text classification device comprising: an inter-word relevance calculating unit that calculates and ranks words C in order of increasing distance, and calculates the relevance between words B and C according to the ranking in the ranking .   A set of categories associated with a set of words and an arbitrary text are input, and for each category, the degree of association between the words D in the text and the word E in the category is calculated. 2. The text classification apparatus according to claim 1, further comprising a text / category relevance calculating means for calculating a relevance between the text and the category based on the relevance calculated by the means. The word vector generation means includes
A morphological analysis means for performing morphological analysis on the text and identifying words necessary for processing;
From the analysis result of the morphological analysis means, the pair co-occurs in any one or a plurality of predetermined ranges in the text with respect to any word pair or any word and any word semantic attribute pair. The frequency at which the event is counted over the entire text is calculated, and for each word, the value of each coordinate is the frequency calculated for the word and the word or word semantic attribute pair corresponding to the coordinate. Co-occurrence matrix generating means for generating a co- occurrence frequency matrix obtained by associating vectors ;
A relative frequency calculation means for converting each coordinate value of the row vector of the co-occurrence frequency matrix generated by the co-occurrence matrix generation means into a relative frequency;
The text classification apparatus according to claim 1, further comprising:   4. The text classification apparatus according to claim 1, wherein the degree of association in the inter-word relation degree calculation means is calculated by an inverse number of the ranking order.   4. The text classification device according to claim 1, wherein the degree of association in the word-to-word degree-of-word calculation means is calculated based on a cosine between the word vectors of the words B and C. The text-category relevance calculating means is:
Morphological analysis is performed on each character string Zpq of the category Cp associated with the OR-connected character strings, and each character string Zpq is converted into a set of sets of different required word termination forms and their frequencies in each character string. And
Arbitrary text U is morphologically analyzed and converted to a set of sets of different required word end forms in the text U and their frequencies,
Based on the relevance calculated by the inter-word relevance calculating means between the word D in the text U and the word E in the category Cp, the character string Zpq in the text U and the category Cp. To calculate the relevance between
The text classification device according to claim 2, wherein a degree of association between an arbitrary text U and an arbitrary category Cp is calculated. A text classification method for classifying text into any category of a predetermined category set,
The word vector generation means receives a corpus, and for any word A, each coordinate corresponds to a word or word semantic attribute, and the value of the coordinate is word A and the word or word meaning corresponding to the coordinate. A word vector generation step of generating a word vector that is a relative value of the co-occurrence frequency with the attribute;
The inter-word relevance calculation means, for any word B, between the word vector obtained by the word vector generation means for the word B and the word vector obtained by the word vector generation means for the arbitrary word C And calculating a relevance level between words, ranking the word C in ascending order of the distance, and calculating the relevance between the word B and the word C according to the ranking in the ranking. A text classification method characterized by that.   The text-category relevance calculating means receives a set of categories associated with a set of words and an arbitrary text as input, and for each category, a word D in the text and a word E in the category A text-category relevance calculating step for calculating a relevance between the text and the category based on a relevance calculated by the inter-word relevance calculating means between the text and the category. Item 8. The text classification method according to Item 7. The word vector generation step includes
A morpheme analyzing unit morphologically analyzes the text and identifies words necessary for processing;
The co-occurrence matrix generating means, based on the analysis result of the morphological analysis step, each of one or a plurality of predetermined ranges in the text with respect to an arbitrary word pair or an arbitrary word and an arbitrary word meaning attribute pair The frequency of counting the events in which the pair co-occurs over the entire text, and for each word, the value of each coordinate is the word and the word or word semantic attribute pair corresponding to the coordinate . A co-occurrence matrix generating step for generating a co- occurrence frequency matrix obtained by associating the calculated vector with the frequency ;
A relative frequency calculating means for converting each coordinate value of the row vector of the co-occurrence frequency matrix generated by the co- occurrence matrix generating step into a relative frequency;
The text classification method according to claim 7 or 8, further comprising:   The text classification method according to claim 7, wherein the relevance in the inter-word relevance calculation step is calculated by the reciprocal of the ranking.   10. The text classification method according to claim 7, wherein the relevance in the inter-word relevance calculation step is calculated based on a cosine between the word vectors of the words B and C. The text-category relevance calculation step includes:
Morphological analysis is performed on each character string Zpq of the category Cp associated with the OR-connected character strings, and each character string Zpq is converted into a set of sets of different required word termination forms and their frequencies in each character string. And
Arbitrary text U is morphologically analyzed and converted to a set of sets of different required word end forms in the text U and their frequencies,
Based on the relevance calculated by the inter-word relevance calculating means between the word D in the text U and the word E in the category Cp, the character string Zpq in the text U and the category Cp. To calculate the relevance between
The text classification method according to any one of claims 8 to 11, wherein a degree of association between an arbitrary text U and an arbitrary category Cp is calculated.   A text classification program for causing a computer to function as the word vector generation means and the inter-word relevance calculation means according to any one of claims 1 to 6.   A text classification program for causing a computer to function as the text-category relevance calculating means according to any one of claims 2 to 6.   A computer-readable recording medium on which the text classification program according to claim 13 or 14 is recorded.

Images