JPH08287097A - Method and device for sorting document - Google Patents

Abstract

PURPOSE: To sort documents based on the contents of a sentence by considering not only the probability of a word but also the relation between adjacent words at the time of obtaining the probability of sorting documents, thereby allowing each of them to obtain the probability of sorting the pertinent document into some document group. CONSTITUTION: When data for calculating sorting probability is accumulated by sorting probability accumulation processing, a document is newly inputted to a new document input part 6. A morphemic analyzing part 7 divides the inputted document to be the units of sentences and then morphemic-analyzes it to extract words. A sorting item extracting part 8 extracts the word, adjacent two words and three words to be sorting items from the obtained words. A sorting probability extracting part 9 selects one sorting group to be the candidate of sorting from among all the sorts. Then a sorting probability calculation part 10 extracts, with respect to the extracted sorting items, the probability of making each of them in the sorting group from a probability accumulation part 14. In addition, the probability of sorting the document to be in some sort is individually calculated through the use of the sorting probability of the sorting items.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document classification method and apparatus, and determines the most appropriate classification from a plurality of document classification candidates prepared in advance for a document input by a user. Document classifier In particular, the present invention relates to a document classification method and apparatus for calculating a probability that a word classifies a document in advance at the time of classifying the document and using the probability to classify the document stochastically.

[0002]

2. Description of the Related Art In recent years, research has been conducted on a document classification method for selecting an appropriate classification from preset classification candidates for a given document. Furthermore, among various document classification methods, a method called a probability-based document classification method is drawing attention. The probability-based document classification method is to input a document whose classification destination (there may be multiple) has been determined in advance as training data, and to determine how many documents the words in it classify. When a document to be stored and newly classified is input, it refers to a method that comprehensively determines which classification the document belongs to by using the classification probability of the stored word and the document. .

The document classification method based on probability in the prior art is represented by the following four methods. 1. First method: Probablistic Relevance Weighting
(PRW) As an example of this method, `` Relevance weighting of searc
h terms; Journal of the American Society for Inform
ation Science, 27: pp. 129-146, 1976 ”.

In this method, if the probability that a document d is classified into a certain classification c is represented by P (c | d), this P (c | d) is used as the probability that the document d is classified into other than the classification c. The basic idea is to express it approximately. represents that c * is classified other than classification c, and P (c * | d)
, The probability P (c | d) can be calculated by finding g (c | d) expressed below.

[0005]

[Equation 1]

Using Bayes' theorem, the above equation (1)
Can be transformed as follows.

[0007]

[Equation 2]

In the equation (2), P (c) is obtained from the training data, and the randomly selected document d
Represents the probability of being classified into a certain classification c. Therefore, P (d
This probability g (c | d) can be obtained by obtaining | c). This P (d | c) can be obtained from the appearance information of the words in the document. That is, assuming that each word appears independently in the document,

[0009]

(Equation 3)

It can be modified as follows. Here, cd is a set of words existing in a document other than the document d classified into the classification c, although it does not appear in the document d whose classification is to be determined in the previous training data. is there. Further, T _i represents a word, and T = 1,0 represents whether or not t _i exists in the document d. Finally, g (c | d) is expressed as follows.

[0011]

[Equation 4]

These can be calculated from the training data and the classification probabilities are determined. See reference p above for details.
p. 130-135. 2. Second method: Component Theory (CT) As an example of this method, "Experiments with a component"
theory of probablistic information retrieval base
d on single terms as document components; ACM Trans
actions on Information Systems, 8 (4): pp. 363-38
6, 1990 ”.

This method is designed to improve the problems of the first method. It is assumed that a document consists of words, and the first method uses the probability that each word classifies the document when calculating the probability, whereas the second method uses one or more arbitrary words. Utilizes the probability of determining the classification destination because they are related to each other. G (c | d) in the equation (1) of the first method is geometrically transformed as follows.

[0014]

(Equation 5)

Here, P (T = t | d) and P (T = t |
c) represents the probability of the word t included in the document d and the probability of the word t in the document classified into the classification c, respectively. These can be obtained from training data. For details, refer to the pp.
368-375.

3. Third method: Retrieval with Proba
blistic Indexing (RPI) As an example of this method, `` Models for retrieval with pr
obablistic indexing; Information Processing & Retri
eval, 25 (1); pp. 55-72, 1989 ". This method is a method in which a word is expanded and vectorized when the probability of classification is obtained, in addition to the second method. That is,
A word vector X representing the feature of the document d is represented by X = (T ₁ , T
₂ , ... T _N ). This T _i is an element that becomes 1 when the document d includes the word t _i and 0 otherwise. P (c | d) at this time can be expressed by the following equation.

[0017]

(Equation 6)

If the Bayes' theorem is applied to this equation by assuming that each word is independent, the above equation (9) can be transformed as follows.

[0019]

(Equation 7)

This can be calculated from the training data. For details, refer to pp.
56-63. 4. Fourth method: Single Random variable with Multi
ple Values (SVMV) As an example of this method, `` A Probablistic Model for Tex
t Categorization Based on a Single Random Variable
with Multiple Values; ANLP '94, pp. 162-167, 199
There is 4 ".

The fourth method is an extension of the third method described above. The fourth method is new in that instead of vectorizing the words in the third method, the frequency information of the words themselves existing in the document is considered. In this method, the classification probability P (c | d) is obtained by modifying the equation (9),

[0022]

(Equation 8)

It can be represented by Each given word t
_{If the} probability that the document d is classified into the classification c with respect to _i is independent, the above equation (12) can be modified as in the third method. That is,

[0024]

[Equation 9]

P (T = t _i | c), P (T = t _i |
d), P (T = t _i ), P (c), etc. can be calculated from the training data. For details, refer to pp. 164-165.

[0026]

Problems of the conventional probability-based document classification method will be described. The problems can be roughly divided into four. 1. First Method: Probablistic Relevance Weighting
The problems with (PRW) are as follows.

Problem 1 Word frequency information in a document is not taken into consideration. Since the probability is considered to be 1 when a word is present in a document and 0 when it is not, it is not possible to include frequency information of words considered to be effective for document classification.

Problem 2 Word weights for classification are not considered. There is no information about how many of the categories a word falls into. Therefore, the weight when a word is included in a certain classification is not considered. For example, the word w1 exists in the classification c1, and the word w2 exists in the classification c.
Assuming they are present in 1, c2, c3, c4, c5, the word w1 is easier to characterize the classification c1 than w2. This property must be taken into account for accurate classification.

Problem 3 It is impossible to deal with the case where the training data is small. If the training data is small, the denominator of the probability calculation formula may be 0, and the probability may not be calculated. A well-known solution to this problem is to add a small number to the numerator and denominator of the stochastic equation. However, this method cannot calculate the exact probability.

Problem 4 The relationship between words cannot be considered. The word for which the probability is calculated simply considers the word in the sentence, so the semantic relationship between the words cannot be considered. 2. Second method: There are the following problems regarding Component Theory (CT).

Problem 3 It is impossible to deal with the case where the training data is small. If the training data is small, the denominator of the probability calculation formula may be 0, and the probability may not be calculated. A well-known solution to this problem is to add a small number to the numerator and denominator of the stochastic equation. However, this method cannot calculate the exact probability.

Problem 4 The relationship between words cannot be considered. The word for which the probability is calculated simply considers the word in the sentence, so the semantic relationship between the words cannot be considered. 3. Third method: Retrieval with Probablistic Indexi
There are the following problems regarding ng (RPI).

Problem 1 The word frequency information in the document is not taken into consideration. Since the probability is considered to be 1 when a word is present in a document and 0 when it is not, it is not possible to include frequency information of words considered to be effective for document classification.

Problem 3 It is impossible to deal with the case where the training data is small. If the training data is small, the denominator of the probability calculation formula may be 0, and the probability may not be calculated. A well-known solution to this problem is to add a small number to the numerator and denominator of the stochastic equation. However, this method cannot calculate the exact probability.

Problem 4 The relationship between words cannot be considered. The word for which the probability is calculated simply considers the word in the sentence, so the semantic relationship between the words cannot be considered. 4. Fourth method: Single Random Variable with Multip
There are the following problems regarding le Values (SVMV).

Problem 4 The relationship between words cannot be considered. The word for which the probability is calculated simply considers the word in the sentence, so the semantic relationship between the words cannot be considered. As described above, in the above-mentioned conventional technique, the frequency information of words is not used, it is impossible to cope with a small amount of training data, and the word included in the document at the time of classifying the document is not included in the probability of being included in the document. Since the document group is determined based on the document based on that, the station between words in the document is not reflected in the document classification result, which causes a problem that accurate classification cannot be performed.

The present invention has been made in view of the above points, and provides a document classification method and apparatus capable of solving the above-mentioned conventional problems and classifying newly input documents more accurately. The purpose is to

[0038]

SUMMARY OF THE INVENTION The present invention is a document classification method for selecting an appropriately appropriate document input by a user from among document classification candidates stored in advance. Is classified into a pre-classified document group,
When determining the document classification probability that a word included in the document is included in the document when classifying the document, not only the probability of the word,
The relationship between adjacent words is acquired and the document classification probability is calculated.

FIG. 1 is a diagram for explaining the principle of the present invention. According to the present invention, when a document is classified, a document classified in advance is read (step 1), a sentence in the read document is subjected to morphological analysis (step 2) for document classification probability calculation, and the document is divided by morphological analysis. It is extracted as a classification item of a document of continuous n (n is a natural number) words used in calculating the classification probability from the extracted word (step 3), and the document is included in all classifications for each extracted classification item. The frequency of classification into a certain classification is calculated (step 4), the probability that each classification item classifies a document from the calculated frequency is calculated (step 5), and the calculated probability is stored for document classification ( In step 6), when a document to be classified is newly input (step 7), the sentence in the input document is morphologically analyzed (step 8), and the words subjected to morphological analysis are used.
A continuous n word string in the classification item is extracted (step 9), and the extracted word string is classified into a class in which there is a matching document among the n consecutive words stored in advance. The classification probabilities to which the documents belong are extracted (step 10), and the classification probabilities of the extracted classification items are used to individually calculate the probabilities that documents are classified into a certain classification (step 1).
1) Among the calculated classification probabilities, the highest classification probability is determined as the document classification result in order (step 1
2) Then, the determined document is displayed as a classification result (step 13).

Further, when determining the classification result of the document in step 12, the document exceeding the preset threshold value among the calculated classification probabilities is determined as the classification result. According to the present invention, in a document classification device that selects an appropriate document from among the document classification candidates stored in advance, a document input by a user is classified into a pre-classified document group. Means for obtaining the document classification probability that a word included in the document is included in the document when classifying the document, the relationship between adjacent words is acquired in addition to the probability of the word, and the document classification probability is calculated. It has a means for calculating.

FIG. 2 is a block diagram showing the principle of the present invention. According to the present invention, when classifying an input document, a first document input unit 101 that reads a document into a memory in advance for calculating a document classification probability and a sentence in the document read from the first document input unit 101 A first morpheme analyzing unit 102 that divides each word, and n consecutive words (n is a natural number) used when calculating a classification probability are extracted as document classification items from the words that are divided by the first morpheme analyzing unit 102. First classification item extraction means 103 and first classification item extraction means 10
For each classification item extracted by 3, the classification frequency calculation means 104 for calculating the frequency of classification of a document into a certain classification among all classifications, and the frequency calculated by the classification frequency calculation means 104, A document classification probability calculating means 105 for calculating a probability that the classification item classifies a document, and a classification probability accumulating means 106 for accumulating the probability calculated by the document classification probability calculating means 105 in the probability accumulating device 107 for document classification. Classification probability calculation / accumulation means 100 using
When a document to be newly classified is input, the second document input unit 201 that reads the document to be newly classified and the document in the document input from the first document input unit are divided into words. A second morpheme analysis means 202 and a second classification item extraction means 20 for extracting a continuous n word string using the words divided by the second morpheme analysis means 202.
3, a classification probability extracting unit 204 that refers to the classification probability accumulating unit 107, and extracts a classification probability belonging to a certain category of a document having a matching n consecutive words; and a category extracted by the classification probability extracting unit 204. A classification probability calculating unit 205 and a classification probability calculating unit 2 which individually calculate the probabilities that a document is classified for a certain classification by using the classification probabilities for each item.
Of the classification probabilities calculated in 05, the classification determination unit 206 that determines the document classification result in order from the highest probability, and the classification result output unit 207 that displays the documents determined by the classification determination unit 206 as the classification result. And a classification probability reference / calculation means 200 to be used.

When determining the classification result of the document, the classification judging means 206 determines the document which exceeds the preset threshold value among the calculated classification probabilities as the classification result.

[0043]

According to the present invention, the inputted documents are classified into a document group which is classified in advance, and in order to solve the above-mentioned problems 1 and 2, not only whether or not a word exists, but also the frequency of the existence of the word. To deal with.

For Problem 3, when the training data is small and the probability cannot be calculated, not only a single word is considered but the frequency information of adjacent words is considered. It becomes possible to ask. Further, for Problem 4, when obtaining the probability of document classification,
Since not only the probability of a word but also the number of adjacent n words are extracted at the same time, and the probability of classifying a document into a certain category is obtained, the co-occurrence relation exists between these words. Considering this, it becomes possible to classify the documents according to the content of the sentence as compared with the conventional technique.

[0045]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 3 shows the configuration of the document classification apparatus according to the embodiment of the present invention, and FIG. 4 is a flowchart showing the general operation of the document classification apparatus according to the embodiment of the present invention.

In FIG. 3, the document classifying apparatus, when accumulating classification probabilities, the document input unit 1 for inputting a training document for calculating probabilities used at the time of document classification, and morphological analysis for classifying an input document in units of parts of speech. Part 2, a classification item extraction part 3 for extracting items necessary for calculating classification probability, an item-wise probability calculation part 4 for calculating a classification probability of a document for each probability item,
A system including a classification probability accumulating unit 5 for calculating the classification probability of the document as a whole, a Japanese dictionary 13 used for morphological analysis, and a probability accumulating unit 14 for accumulating the classification probability calculated for each item is used. Also, the item-based probability calculation unit 4
Is a classification frequency calculation unit 41 that obtains a frequency for which a document is classified into a certain classification among all classifications for each classification item,
It has a classification frequency calculation unit 42 that calculates the probability that each classification item classifies a document from the calculated classification frequency.

When the classification probability is calculated, the document input unit 6 for inputting a new document to be classified, the morphological analysis unit 7 for dividing the input document into units of parts of speech, and the classification for extracting the corresponding items for calculating the classification probability. An item extraction unit 8, a classification probability extraction unit 9 that extracts a classification probability for each classification item accumulated in advance,
A classification probability calculation unit 10 that calculates the probability that the input document belongs to, a classification determination unit 11 that determines which classification the input document belongs to, and a classification result output unit 1 that outputs the classification result.
2. A system including a Japanese dictionary 13 used for morphological analysis and a probability storage device 14 for storing the classification probabilities calculated for each item is used. Further, in this embodiment, the classification result output unit 12 outputs the classification result to the display device.

The operation of the embodiment of the present invention will be described below with reference to the block diagram of the document classification device of FIG. 3 and the processing flow of the document classification device of FIG. In the present embodiment, an example will be shown in which document classification is performed by focusing on three words adjacent to a certain word. The processing flow is the same even when the number of adjacent words increases.

First, in the process of FIG. 4, a process of accumulating in the probability accumulating device 14 the probabilities for each item required for giving the classification probability of a document will be described. Further, in FIG. 5, a process at the time of calculating the classification probability will be described. In FIG. 6, detailed processing when calculating the classification probability will be described. Step 101) The document input unit 1 inputs a large number of training documents shown in FIG. In the example of FIG. 7, a plurality of correct classification destinations of the document are input together with the body of the document. Examples of classifications range from subdivided ones such as “airplane” as shown in FIG. 7 to large classifications such as “crime”. There are generally a plurality of document classification destination candidates.

Step 102) When the document is input,
The morphological analysis unit 2 divides the input document into sentence units with reference to the Japanese dictionary 8 and performs morphological analysis. For example, if we morphologically analyze the text of Fig. 7, "A plane from Cape Town in South Africa to London is about to take off.
Hijacked by a group of people. ”,“ There are 359 passengers and passengers left on the plane, which is a tense situation that cannot be foreseen. ”Divide into sentences and use the Japanese dictionary 8 for each. Perform morphological analysis. FIG. 8 shows the result. In FIG. 8, the sentence is divided into words, and a reading and a part of speech are given to each. For example, for the word "South Africa", the reading is "Minami Africa" and the part of speech is "proper noun".
Is given.

Step 103) After the words are cut out from all the training sentences, the classification item extracting unit 3 extracts the classification items for calculating the classification probability in the following steps,
Further, the item-wise probability calculation unit 4 calculates the category probability for each item. In order to classify documents, not the individual words but the adjacent words and the adjacent three words are focused. FIG. 9 shows an example of adjacent words and three adjacent words. The table in FIG. 9 shows some examples of adjacent words. In this figure, the words in FIG. 7 are grouped by the designated number in the order of appearance. Similarly, the lower part of FIG. 9 also shows an example of three adjacent words. Using these words and phrases, the probability that the document d is classified into a certain classification c is obtained as follows.

Basically, a document d is classified into the classification c.
Probability is not the word existing in the document but the adjacent 3 words
Pay attention to the calculation. Now any adjacent word
Row t _i-2, T_i-1, T_i(T_i, T_i-1, T
_i-2). In the example of FIG. 9, adjacent
The word string is (of South Africa) and the adjacent 3 word strings are
(Cape Town, South Africa). this
Then, the classification probability P (c | d) is

[0053]

[Equation 10]

It can be represented by Assuming that the probability that the document d is classified into the classification c is independent for each given t _i , the above equation can be transformed as follows in the same manner as the method of 3 described above.

[0055]

[Equation 11]

Some values in the above equation will be described. P (T = (t _i , t _i-1 , t _{i- 2} | c)) is a time classification c in which three adjacent words t _i , t _i-1 , and t _i-2 appear consecutively.
Probability (A) of being classified into P (T = (t _i , t
_i-1 , t _i-2 | d) indicates the probability (B) that the document d exists and three adjacent words appear therein. For example, the document of FIG. 7 is classified into the categories of “crime” and “airplane”, and further, this document has three adjacent words (South Africa, Cape Town) and the like as its classification items as shown in the upper diagram of FIG. keeping. The classification item extraction unit 3 extracts the documents having the same word strings from all the documents having the same classification destination,
The classification frequency calculation unit 41 of the item-based probability calculation unit 4 calculates the frequency, and the document classification probability calculation unit 42 uses the frequency to calculate the conditional probability. For example, assuming that a document different from that shown in FIG. 7 exists and one of the classification destinations is “crime”, the above-mentioned three adjacent words (South Africa, Cape Town) are present in this document. Then, they are regarded as the same information, and the probability is calculated by adding them to the frequency information.
P (c) is the probability (C) that the randomly selected document d is classified into the classification c. P (T = (t _i |
t _i-1 , t _i-2 ) is the probability (D) that t _i appears when _two adjacent words t _i-1 , t _i-2 appear. The values from (A) to (C) can be calculated by checking the frequency of appearance of three adjacent words and the classification destination of the document to which they are related.
The last value (D) cannot be calculated correctly when the number of occurrences of three adjacent words in all the training documents is small, so the following measures are taken to increase the frequency of probability calculation. The frequency of occurrence of a variable t in the document is f (t)
It is represented by.

P (T = (t _i | t _i-1 , t _i-2 )) (18) = q ₁ (f (T = t _i | t _i-1 , t _i-2 ) + q ₂ f ( _{T = t i | t i-} 1) + q 3 f (T = t i) (19) f in the equation _{(T = t i | t i} -1) is t _i when word t _i-1 appeared , F (T = (t _i ) is t
_The frequency at which _i appears and q ₁ , q ₂ , and q ₃ are positive numbers less than or equal to 1 that can be determined by training data, or constants that take 0. Therefore, in order to calculate this value, f (T
= T _i | t _i-1 , t _i-2 ), f (T = (t _i |
The following calculation may be performed to obtain t _i−1 ), f (T = (t _i )). First, the classification frequency for each word is calculated. Therefore, the same notation is extracted from the words extracted from the entire training sentence.

Step 104) The classification item extracting section 3
For the extracted word, mark where the document containing the word is classified. If the marked classification destinations are the same, the classification frequency calculation unit 41 of the item-by-item probability calculation unit 4 regards them as one set and calculates the frequency. This frequency f is used as the word t ₁
Is represented by f (t ₁ ). Next, the frequency of adjacent words is calculated. Similar to the case of a single word, first, adjacent words are extracted from the whole training sentence, and similarly, a set is created based on the classification destination and the frequency is calculated. This frequency f is set to the adjacent word t ₁ ,
with t ₂ represents the _{f (t 1, t 2)} . Furthermore, adjacent 3
The word frequency is calculated in the same way. This frequency is also f
It is expressed as (t ₁ , t ₂ , t ₃ ). Value (D) from these values
Can be calculated. As an example, consider an example of collecting frequency information of three adjacent words in FIG. In FIG. 10, for example, if three words (South Africa, Cape Town) are consecutive,
It indicates that the documents including them are classified into "crime", "airplane", "Africa", "travel", and the like. Frequency information is given to each.

Step 105) As described above, if the frequency information is obtained for all the documents, the process proceeds to step 106, and if there are remaining documents, the process proceeds to step 101. Step 106) When a document d is given, the document classification probability calculation unit 42 of the item-wise probability calculation unit 4 uses the classification items t _i , t _i-1 , t _i-2, etc., in the document to determine the document. The training data of which category corresponds to and what probability each category has for classifying the document is calculated.

Step 107) The classification probability accumulating unit 5
These are stored in the probability storage device 11. By using the equation (19), the equation (17) can be finally transformed into the following form.

[0061]

(Equation 12)

Next, the processing for calculating the classification probability will be described with reference to FIG. Step 201) When the data for classification probability calculation is accumulated by the classification probability accumulation process of FIG. 4, a new document is input to the new document input unit 6. Step 202) The morphological analysis unit 7 divides the input document into sentence units and then performs morphological analysis to extract words.

Step 203) The classification item extracting section 8
A single word that becomes a classification item from the words thus obtained,
Two adjacent words and three adjacent words are extracted. (Step 204) The classification probability extraction unit 9 selects one classification group as a classification candidate from all the classifications.

Step 205) Classification probability calculation unit 10
For the extracted classification items, the probability accumulating unit 14 extracts the probabilities that the respective classification items will belong to the classification group. Further, using the classification probabilities of the classification items, the probabilities of documents being classified for a certain classification are individually calculated. Detailed operation will be described later with reference to FIG.

Step 206) If all the classification groups have been confirmed, the procedure proceeds to step 207, and if there is an unconfirmed classification group, the procedure proceeds to step 204. Step 207) The classification probability calculation unit 10 arranges the classification groups in order from the highest calculated probability.

Step 208) Classification probability calculator 10
Indicates the classification determination unit 11 from the one having the highest classification probability in the memory.
Transfer to. Step 209) The classification determination unit 11 checks whether or not the threshold for extraction is exceeded. If the result of the inspection is exceeded, go to step 210, and if not,
The process ends.

Step 210) The classification judgment section 11 passes the threshold value and passes the inspection to the classification result output section 12. The classification result output unit 12 determines only the documents that have passed the inspection, the document classification destination, and finally outputs the documents in order. Step 211) In this way, the inspection is completed for all the classification groups, and all the processes are completed.

FIG. 6 is a flow chart for explaining the detailed operation of the classification probability calculating section according to the embodiment of the present invention. Step 301) Set an initial value 1 to the count i. (Step 302) First, i words adjacent to a single word are extracted.

Step 303) Extract classification probabilities for adjacent i words. Step 304) If i is 4 or less, Step 3
If i is greater than 4 in step 05, step 306
Move to (Step 305) Next, the word is replaced with two adjacent words, and the classification probability is similarly extracted.

Step 306) In this way, the three adjacent words and the necessary probability values such as P (c) are extracted from the probability accumulator 14. Step 307) The classification probability calculation unit 10 substitutes the above value into the equation (21) which is the probability calculation equation.

Step 308) As a result, the classification probability calculation section 10 calculates the classification probability of the document for the selected classification group. For example, it is assumed that the input document is given a probability of being classified into each classification as shown in FIG. 5.0x threshold
In the case of 10 ⁻⁸ , if the items exceeding this threshold are displayed in order from the one with the highest probability in FIG.
It becomes 8, 3, 2. These classifications are determined as classification destinations of the input document.

As described above, by using the present invention, it is possible to solve the problems of the conventional probability-based document classification method, and it becomes possible to classify documents with higher accuracy than the conventional method. In the above embodiment, the document input unit 1,
6, the morpheme analysis units 2 and 7, and the classification item extraction units 3 and 8 are described separately in FIG. 3 and are given different reference numerals, but they have the same configuration, and the classification probability accumulation time and the classification probability accumulation time It is divided for convenience of explanation at the time of probability calculation.

The present invention is not limited to the above embodiments, but various modifications and applications are possible within the scope of the claims.

[0074]

As described above, according to the present invention, the problem of not using the frequency of words that the probability-based document classification method in the prior art does not use (problems 1 and 2) and the case where the training data is small are addressed. For the problem that can not be done (Problem 3),
It can be seen that this can be solved by referring to all the words contained in the document.

Furthermore, in the conventional technique, the relationship between words, which is not used in the probability-based document classification method, is
By considering that three adjacent words appear together, the relationship between them is incorporated into the classification probability to calculate the classification probability of the document, so that the document can be classified with higher accuracy.

As described above, according to the present invention, when the probability of document classification is obtained, not only the probability of words but also the relationship between adjacent words are taken into consideration, and these documents are classified into a certain document group. Since the probability of being processed is obtained, it is possible to classify the document according to the content of the sentence.

[Brief description of drawings]

FIG. 1 is a diagram for explaining the principle of the present invention.

FIG. 2 is a principle configuration diagram of the present invention.

FIG. 3 is a configuration diagram of a document classification device according to an embodiment of the present invention.

FIG. 4 is a flowchart showing a process at the time of accumulating classification probabilities according to an embodiment of the present invention.

FIG. 5 is a flowchart showing a process when calculating a classification probability according to an embodiment of the present invention.

FIG. 6 is a flowchart illustrating a detailed operation of a classification probability calculation unit according to an exemplary embodiment of the present invention.

FIG. 7 is a diagram showing an example of a document input to the document input unit according to the embodiment of the present invention.

FIG. 8 is a diagram showing a morphological analysis result according to an embodiment of the present invention.

FIG. 9 is a diagram showing an example of an adjacent word and three adjacent words according to an embodiment of the present invention.

FIG. 10 is a diagram showing an example of collecting frequency information of adjacent three words according to an embodiment of the present invention.

FIG. 11 is a diagram showing an example of assigning a probability to each document according to an embodiment of the present invention.

[Explanation of symbols] 1 document input unit 2 morphological analysis unit 3 classification item extraction unit 4 item-wise probability calculation unit 5 classification probability accumulation unit 6 document input unit 7 morphological analysis unit 8 classification item extraction unit 9 classification probability extraction unit 10 classification probability Calculation unit 11 Classification determination unit 12 Classification result output unit 13 Japanese dictionary 14 Probability accumulation device 41 Classification frequency calculation unit 42 Document classification probability calculation unit 100 Classification probability calculation / accumulation unit 101 First document input unit 102 First morpheme analysis Means 103 First classification item extraction means 104 Classification frequency calculation means 105 Document classification probability calculation means 106 Classification probability accumulation means 200 Classification probability reference / calculation means 201 Second document input means 202 Second morpheme analysis means 203 Second Classification item extraction means 204 Classification probability extraction means 205 Classification probability calculation means 206 Classification determination means 207 Classification results Output means

Claims (6)

[Claims] 1. A document classification method for selecting an appropriate document from among document candidates stored in advance for a document input by a user, wherein the input document is classified into a pre-classified document group. When categorizing and determining the document classification probability that a word included in the document is included in the document when classifying the document, not only the probability of the word but also the relationship between adjacent words of the word are acquired, and The document classification method according to claim 1, wherein a document classification probability is calculated. 2. When classifying the documents, in order to calculate the document classification probability, a document classified in advance is read, a sentence in the read document is morphologically analyzed, and a classification probability is calculated from words divided by the morphological analysis. Extracted as a classification item of a document of consecutive n (n is a natural number) words used at the time of calculation, and for each extracted classification item, calculate the frequency with which the document is classified into classifications among all classifications Then, from the calculated frequency, the probability that each classification item classifies the document is calculated, the calculated probability is accumulated for document classification, and when the document to be classified is newly input, Morphological analysis is performed on the sentence of, and n consecutive word strings in the classification items are extracted using the morphologically analyzed words, and the consecutive n words accumulated in advance for the extracted word string Match in The classification probabilities that belong to a certain category of each document are extracted, and the probabilities that the documents are classified to a certain category are individually calculated using the classification probabilities of the extracted classification items. Among the calculated classification probabilities, The document classification method according to claim 1, wherein the document having the highest probability is determined in order as the classification result of the document, and the determined document is displayed as the classification result. 3. The document classification method according to claim 2, wherein when the classification result of the document is determined, a document out of the calculated classification probabilities that exceeds a preset threshold value is determined as the classification result. 4. A document classification device for selecting an appropriate document from among document classification candidates stored in advance for a document input by a user, and converting the input document into a pre-classified document group. A means for classifying, when obtaining a document classification probability that a word included in a document is included in the document when classifying the document, not only the probability of the word but also the relationship between adjacent words of the word are acquired. A document classification device, comprising: means for calculating the document classification probability. 5. When classifying an input document, first reading the document into a memory for calculating a document classification probability.
Document input means, first morpheme analysis means for dividing the sentence in the document read from the first document input means into words, and when the classification probability is calculated from the words divided by the first morpheme analysis means. First classification item extracting means for extracting consecutive n words (n is a natural number) to be used as classification items of the document, and for each classification item extracted by the first classification item extracting means, Classification frequency calculation means for calculating the frequency of classification of a document into a certain classification among all classifications, and document classification probability calculation for calculating the probability that each classification item classifies a document from the frequencies calculated by the classification frequency calculation means Means and a classification probability accumulating means for accumulating the probabilities calculated by the document classification probability calculating means for document classification, and inputting a document to be newly classified. A second document input means for reading the document to be newly classified, and a second morphological analysis means for dividing the document in the document input from the first document input means into words. A second classification item extracting means for extracting a continuous word string of n words using the words divided by the second morpheme analyzing means; and a continuous n word referring to the classification probability accumulating means. A classification probability extraction unit that extracts a classification probability that a matching document belongs to a certain classification and a classification probability for each classification item extracted by the classification probability extraction unit are used to classify the document with respect to a certain classification. Among the classification probabilities calculated by the classification probability calculation means and the classification probability calculation means for individually calculating the probability
A classification probability reference unit that uses a classification determination unit that determines the classification result of the document in order from the highest probability and a classification result output unit that displays the documents determined by the classification determination unit as the classification result.
The document classification device according to claim 4, further comprising calculation means. 6. The document classification apparatus according to claim 5, wherein the classification determination unit determines, as a classification result, a document that exceeds a preset threshold value among the calculated classification probabilities when determining the classification result of the document. .