JP3471253B2 - Document classification method, document classification device, and recording medium recording document classification program - Google Patents

Description

BACKGROUND OF THE INVENTION [0001] [Technical Field of the Invention The present invention relates to an apparatus for classifying electronic document won accompanied document structure meaning attributes from the electronic document. 2. Description of the Related Art In recent years, computer systems using natural language processing techniques, such as a Japanese word processor and a machine translation system, have advanced, and demands for improvements in natural language processing techniques have been increasing. [0003] In the natural language processing technology, a conventional method of obtaining a sentence structure is, for example, 14 of natural language processing (Iwanami Koza Software Science 15, edited by Makoto Nagao et al., Published by Iwanami Shoten).
The syntactic analysis and the semantic analysis were performed as described on pages 0 to 229. Further, the conventional classification method uses the frequency of appearance of nouns. [0004] In the above-mentioned conventional methods of syntactic analysis and semantic analysis, it takes time or erroneous analysis for a long sentence or a sentence having a complicated structure. There are many. In particular, there is a problem that the accuracy of analysis is low for documents that have not been proofread, such as e-mails and homepage documents that have been increasing in recent years. Also,
The classification method using the frequency of appearance of nouns has a problem in that classification cannot be performed in consideration of the detailed intention of the document creator. SUMMARY OF THE INVENTION In view of the above, it is an object of the present invention to correctly assign a sentence structure with semantic attributes to uncorrected sentences and low-quality sentences, and to classify the sentence including the intention of the document creator. The object of the present invention is to provide a document classification device capable of performing the following. A document classification device according to the present invention includes a document input unit for inputting a document, and a basic form and a part of speech by analyzing the input document. Morphological analysis means for dividing the morpheme into morphemes having morpheme information including inflected forms, and for each morpheme constituting the input document, generating a feature vector from the frequency of morpheme information of morphemes appearing before and after the morpheme, A feature in which the semantic attribute appears, obtained from morpheme information of morphemes appearing before and after the morpheme to which the semantic attribute is assigned from a corpus in which a plurality of documents composed of morphemes to which the semantic attribute is assigned are stored in advance. Calculates the inner product of the vector and the generated feature vector, and finds the semantic genus of the feature vector with the smallest distance found from the corpus. To the morpheme, and for the input document, the morpheme to which a semantic attribute is assigned uses a semantic attribute, and the morpheme without a semantic attribute is analyzed by analyzing the input document instead of the semantic attribute. A sequence of morphemes is created using the obtained basic form, a feature vector having a frequency of the created sequence of morphemes as a component is created, and for the documents stored in the corpus, morphemes to which a semantic attribute is assigned are: Using semantic attributes, morphemes to which no semantic attributes are assigned create a sequence of morphemes using the basic form stored in the corpus, and create a feature vector having the frequency of the created morpheme sequence as a component, Similarity calculating means for calculating a distance using an inner product of a feature vector of an input document and a feature vector of a document stored in a corpus;
Classification means for classifying the input document into categories of documents having the smallest distance and stored in the corpus,
An output unit that outputs the classification result to the outside is provided. The present invention will be described by taking as an example the case where received e-mails are classified. When a new received mail is input by the input means, the received mail is divided into morphemes by the morphological analysis means. Next, by means of giving meaning,
Before and after each morpheme composing the input document
The feature vector is calculated from the frequency of the morpheme information
Generated morphemes with morpheme information and semantic attributes
Corpus in which multiple documents consisting of
Appearing before and after morphemes with semantic attributes
The semantic attribute found from the morpheme information of the morpheme appears
And the inner product of the generated feature vector
Calculated from the corpus, the feature with the smallest distance
The semantic attribute of the vector is assigned to the morpheme. Next,
The similarity calculating means calculates a semantic attribute of the input document.
Morphemes are given semantic attributes, and the semantic attributes are
Unassigned morphemes are input documents instead of semantic attributes
Of morphemes using the basic form obtained by analyzing
And the frequency of the generated morphemes
Create a signature vector and add it to the document stored in the corpus.
Therefore, a morpheme to which a semantic attribute is assigned
Morphemes without semantic attributes
Create a sequence of morphemes using the stored basic forms,
The feature vector whose frequency is the frequency of the created morphemes
Create a vector and convert it to a feature vector and corpus of the input document
The distance using the inner product with the feature vector of the stored document
calculate. Then, the input means converts the input document into the distance
It is classified but to the category of the smallest corpus being stored document
You . Embodiments of the present invention will now be described with reference to the drawings. FIG. 1 is a schematic block diagram of the document classification device according to the first embodiment of the present invention.
FIG. 2 is a flowchart illustrating a document classification method according to the first embodiment of this invention. The document classification device 10 includes an input unit 11 for inputting a document, a morphological analysis unit 12, a corpus learning unit 13, a meaning providing unit 14, a similarity calculation unit 15,
Classification means 16, output means 1 for outputting the classification result of the document
7, a corpus 18 and a control unit 19 for controlling the processing of each unit. A corpus 18 is a group of linguistic data collected in large amounts (see Natural Language Processing, Iwanami Koza Software Science 15, Makoto Nagao et al., Published by Iwanami Shoten, page 253). It is assumed that a large number of documents composed of morphemes to which are added are stored in a predetermined format in advance. FIG. 3 shows an example of a document in the corpus 18. FIG.
Is an example of a corpus-tagged stored document used in the document classification method according to the first embodiment. Two documents, "I cannot communicate with my mobile phone" and "No effective solution is written" It is described as an example. The morphological analysis unit 12 divides the sentence of the input document input by the input unit 11 into morphemes having morphological information including basic forms, parts of speech, and inflected forms. Corpus learning means 14
Is a morpheme of a morpheme appearing in the vicinity of the morpheme to which the semantic attribute tag is attached, which is extracted from the corpus 18 in a predetermined form, and which is composed of the morphemes stored in the corpus 18 and to which the semantic attribute tag is attached. The frequency of the information is acquired, and the feature in which the semantic attribute appears is learned from the acquired frequency. The meaning providing means 13 compares the feature of the semantic attribute obtained as a result of learning with the feature of the frequency of the morpheme information of the morpheme located near the morpheme constituting the input document, and finds the most similar feature. Is assigned as a semantic attribute of a morpheme constituting the input document. The similarity calculating means 15 compares the input document having the morpheme with the semantic attribute and the stored document stored in the corpus with respect to the frequency of the morphemes obtained by adding the semantic attribute to the morpheme information. The similarity of the document is calculated based on the arrangement frequency. When the similarity between the input document and the storage document to be compared with the corpus 18 exceeds the threshold, the classification unit 16 classifies the input document into the category of the storage document. The operation of the document classification method according to the first embodiment of the present invention will be described with reference to FIG. When the document classification process is started (S11), a document is input by the input unit 11 (S12), and the input document is divided by the morphological analysis unit 12 into morphemes having morpheme information including basic forms, parts of speech, and inflected forms (S13). ). One of the morphemes to which the semantic attribute is assigned is selected by the meaning assigning unit 14 (S14), and the feature is calculated from the frequency of the morpheme information of the n morphemes before and after the morpheme (S15). The feature of the learned semantic attribute is compared (S16). If there is another candidate semantic attribute (S17N), the next candidate semantic attribute is selected (S18), and the process returns to step S16. Repeat the process. If no other candidate semantic attribute exists (S17N), a semantic attribute having the most similar feature is assigned (S19). If the assignment of the entire semantic attribute of the morpheme of the input document is not completed (S20N), the morpheme to which the next semantic attribute of the input document is assigned is designated (S21), and step S21 is performed.
Returning to step 14, the process is repeated. If the assignment of the entire semantic attributes of the morphemes of the input document is completed (S20Y), one of the stored documents is selected by the similarity calculating means 15 (S2).
2) comparing the feature consisting of the frequency of morpheme arrangement including the semantic attribute of the input document with the feature consisting of the frequency of morpheme arrangement including the semantic attribute of the selected stored document (S23), and determining the similarity to a predetermined threshold value; If not (S24N), the next storage document is designated (S25), and the process returns to step S22 to repeat the processing. If the similarity is equal to or more than a predetermined threshold (S24
Y) Classifying means 17 classifies the input document into stored document categories whose similarity is equal to or greater than a predetermined threshold (S26),
The document classification process ends (S27). Next, step S1
The procedure for learning the semantic attributes described in 4 to S21 will be described in detail with an example. First, Corpus 1
The morpheme information of the sentence of the stored document stored in 8 is based on the part of speech, basic type, and inflected form of a word obtained by morphological analysis as shown in FIG. A feature vector is constructed based on these pieces of information. Corpus
The document stored in No. 18 has the meaning before and after each semantic attribute.
Morphological information of n morphemes (basic form, part of speech, inflection
Form), so-called n-gram (natural language processing Iwanami)
Course Software Science 15, Makoto Nagao et al., Iwanami Shoten
Line 15), and each of the entire corpus
Weight morpheme information of co-occurring words using a predetermined method.
No, the characteristics of each semantic attribute from the weight of all morpheme information
Vector is composed. For example, a certain sentence of a document stored in the corpus 18 “ Quality of contents spoken in a lecture
As for “ question ”, morpheme information as shown in Table 1 is given. [Table 1] Here, when the document “there was a confirmation of its operation” is input and morphological analysis is performed, morphological information as shown in Table 2 is obtained. [Table 2] In Table 1, the basic form “re” of “re” has semantic attributes of “respect”, “passive”, “possible”, and “self-motivated”, but here is “respect”. The purpose of this step is to learn the semantic attributes of the basic form "re" of Table 2 "re". A feature vector of morphological information of n pieces before and after “re” is created for an input document. In this case, the weights of the co-occurring morpheme information are all 1, and the other weights are 0. The inner product of the feature vector of each semantic attribute of "re" that can be taken by the corpus 18 and the feature vector of "re" of the input document is calculated, and the distance (the angle of the vector) is the smallest. The semantic attribute is obtained and adopted as the semantic attribute of the morpheme of the sentence of the input document. As a result, the semantic attribute of the morpheme used in the similar context is selected as the semantic attribute of the morpheme of the input document. Next, the method of calculating the similarity between the document stored in the corpus 18 and the input document, which is described in steps S22 to S26, will be described in detail with reference to an example. Here, a morpheme sequence of the document is used, a morpheme having a semantic attribute is a semantic attribute, and a morpheme having no semantic attribute is created using a basic morpheme type. Create a vector. A document composed of morpheme information shown in Table 3 and a morpheme having a semantic attribute will be described as an example. [Table 3] In this document, when n-gram is created with n = 3, (there, "restricted", operation, confirmation) ("restricted, operation, confirmation") (operation, confirmation, performed) (confirmation, performed, "respect") (performed) , "Respect", "assertion") ("respect", "assertion", "call") ("assertion", "call", "confirmation"), and these n-grams for documents stored in the corpus. The feature vector of the document using the frequency of is created. Similarly, for the input document having acquired the semantic attribute, a feature vector of the document is created using the frequency of n-gram, and a distance is calculated using an inner product with the feature vector of the stored document stored in the corpus. The input document is classified into the category of the stored document or the stored document group having the smallest distance (angle of the vector). [0031] The sequence of morphemes of interest in the similarity calculation unit 15, shown in FIG. 4 is an example of a sequence of morphemes sentence of the document to be inputted to the morpheme of arrangement and new in the document example of corpus shown in FIG. The sequence of morphemes uses a semantic attribute when a semantic attribute exists, and uses a basic morpheme when no semantic attribute exists. Next, a document classification method and a document classification device according to a second embodiment of the present invention will be described with reference to the drawings. FIG. 5 is a schematic block diagram of a document classification device according to the second embodiment of this invention. FIG. 5 shows a document classifying apparatus 100 according to the present invention.
It is shown as a computer that constitutes the device,
The computer includes an input unit 110 such as a modem, a keyboard, and a pointing device, an output unit 120 such as a modem, a printer, and a display, a data processing device 130, a storage unit 140, and a recording medium 150. Recording medium 150
Records a document classification system control program of the present invention capable of controlling the operation of each unit.
M, a semiconductor memory or the like is used. The configuration of the document classifying apparatus and the document classifying method are the first.
The description is omitted because it is the same as that of the embodiment. The feature of the semantic attribute is learned from the feature of the frequency of the morpheme information of the morpheme appearing near the morpheme to which the tag of the semantic attribute of the stored document stored in the corpus is added, and a newly input document is learned. The control program for acquiring the semantic attributes of the morpheme obtained for the input document and further classifying the input document into a group of stored documents in the corpus similar to the input document is read from the recording medium 150 into the data processing device 130 and read from the data processing device 130. The operation of 130 is controlled. The data processing device 130 executes the following processing under the control of the control program. [0036] That is, each constituting a process of inputting the document, the basic form by analyzing the input document, part of speech, the process of dividing into morphemes with the morpheme information including inflected forms, the input document
For morphemes, the morpheme forms that appear before and after the morpheme
A feature vector is generated from the frequency of elementary information, and morpheme information and
Documents composed of morphemes with semantic attributes
A semantic attribute is assigned from a corpus stored in advance.
From the morpheme information of morphemes that appear before and after the morpheme
The obtained feature vector in which the semantic attribute appears and the raw
Calculate the inner product of the generated feature vectors and calculate
Genus of the feature vector with the smallest distance found
A process of applying to the morphological sexual, with the input document
Morphemes with semantic attributes use semantic attributes.
Morphemes without a semantic attribute are
Instead, form using the basic form obtained by analyzing the input document
Create a sequence of morphemes and configure the frequency of the
Create a feature vector as a component and store it in the corpus.
Form in which semantic attributes are assigned to the delivered document
The element uses semantic attributes and no semantic attributes are assigned
A morpheme is derived from a morpheme using the basic form stored in the corpus.
Create a list and configure the frequency of the morphemes created
Create a feature vector to be used as the base
Product of the file and the feature vector of the document stored in the corpus
And calculating the distance using the input document,
And processing for classifying the document into the category of the document stored in the corpus having the shortest distance . As described above, according to the present invention, the semantic attributes of morphemes are complemented from the corpus, and the similarity with the stored document stored in the corpus is calculated based on the arrangement of the morphemes including the semantic attributes. Thus, there is an effect that it is possible to classify the document more finely in consideration of the intention.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic block configuration diagram of a document classification device according to a first embodiment of the present invention. FIG. 2 is a flowchart illustrating a document classification method according to the first embodiment of this invention. FIG. 3 is an example of a corpus-tagged stored document used in the document classification method according to the first embodiment; FIG. 4 is an example of an arrangement of morphemes to be targeted by the similarity calculation means. FIG. 5 is a schematic block diagram of a document classification device according to a second embodiment of the present invention. [Description of Signs] 10, 100 Document Classification Device 11, 111 Input Unit 12, 112 Morphological Analysis Unit 13, 113 Corpus Learning Unit 14, 114 Meaning Assignment Unit 15, 115 Similarity Calculation Unit 16, 116 Classification Unit 17, 117 Output Means 18, 118 Corpus 19, 119 Control unit 110 Input unit 120 Output unit 130 Data processing device 140 Storage unit 150 Recording medium S11 to S17 Step

Claims (1)

(57) [Claim 1] An apparatus for acquiring a sentence structure with a semantic attribute from an electronic document and classifying the electronic document, wherein a document input means for inputting the document, A morphological analysis unit that analyzes the document and divides the morpheme into morphemes having morpheme information including a basic form, a part of speech, and an inflected form; and, for each morpheme constituting the input document, A feature vector is generated, and is obtained from morpheme information of morphemes appearing before and after a morpheme having a semantic attribute from a corpus in which a plurality of documents composed of morpheme information and a morpheme having a semantic attribute are stored in advance. Further, the inner product of the feature vector in which the semantic attribute appears and the generated feature vector is calculated, and the meaning of the feature vector having the smallest distance obtained from the corpus is calculated. Means for assigning attributes to the morpheme; and for the input document, morphemes to which semantic attributes are assigned use semantic attributes, and morphemes to which no semantic attributes are assigned analyze the input document instead of semantic attributes. A sequence of morphemes is created using the basic form obtained as a result, a feature vector having a frequency of the created sequence of morphemes as a constituent element is created, and a semantic attribute is given to a document stored in the corpus. For morphemes, semantic attributes are used. For morphemes to which no semantic attributes are assigned, a morpheme sequence is created using the basic form stored in the corpus, and a feature vector having a frequency of the created morpheme sequence as a component is used. Similarity calculating means for calculating and calculating a distance using an inner product of a feature vector of the input document and a feature vector of a document stored in the corpus; A document classification device, comprising: a classification unit that classifies the input document into a category of a document stored in the corpus with the shortest distance; and an output unit that outputs a classification result to the outside.