JP2013191194A - Document categorizing device, method thereof and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a document categorizing device capable of performing document classification close to human sensitivity.SOLUTION: A document of a portion corresponding to a certain topic of an input document containing a plurality of topics is defined as a reference document, a document vector of the reference document is extracted, and inter-vector similarity is determined between the document vector and a topic vector that is a center of gravity of a document vector included in a topic class obtained by cluster-classifying topics determined from a sample document. A category sample document vector correspondence table makes a sample document vector determined from a sample document whose topics are classified for each category correspondent to a topic. Inter-vector similarity is determined for each topic class between the topic vector corresponding to a topic class of the high inter-vector similarity and a sample document vector in the category sample document vector correspondence table. A value obtained by accumulating values each resulting from multiplying the similarity by a degree of importance of the topic is determined as document similarity, and the reference document is classified to a category having the highest document similarity.

Description

  The present invention relates to a document categorizing apparatus for classifying a plurality of document data, a method thereof, and a program thereof.

  In document categorization in which an entire document set is divided into arbitrary document groups, document vectors are widely used as feature amounts for calculating similarity between documents. The document vector is a column vector generated from a concept vector determined in terms of word appearance frequency or word. The similarity between documents can be expressed by the similarity between document vectors.

  Patent Document 1 discloses a method of categorizing a document (specimen document) to be collected using similarity between document vectors. In this method, a word concept vector determined for each word is obtained for a plurality of words included in a sample document set, and the vector centroid of each word is extracted as a feature amount represented by the document vector. Next, calculate the similarity between the feature quantity of the sample document set and the feature quantity of the reference document to be categorized, and collect the documents you want to collect in the same category as the sample document set with the smallest value. Categorize.

  In this method, the weight of the word concept vector in the feature amount of a certain document is determined based on the appearance frequency in a document such as TF-IDF or is given manually. At this time, it is general that the commonality of topics is not considered.

  As a method for classifying topics from the context included in each document, a document data classification device 1 disclosed in Patent Document 2 is known. The operation will be briefly described with reference to the functional configuration of the document data classification device 1 (FIG. 12). Here, the topic is a characteristic word or a set of words included in a group of documents or contexts, and is recalled from them.

  The document data classification device 1 includes a memory 10, a document input unit 20, a context extraction unit 30, a document vector generation unit 40, a context combination unit 50, a cluster classification unit 60, a result display unit 70, and a control unit 80. The document input unit 20 records an input document input from the outside in the memory 10. The context extraction unit 30 reads each input document from the memory 10, determines whether each word in the input document matches a main word recorded in the memory 10 in advance, and matches the word in the input document. A part of word strings in the input document that are determined based on the position of each are extracted as estimated contexts. The document vector generation unit 40 generates a first document vector that is a document vector of each estimated context. The context combining unit 50 integrates a plurality of estimated contexts in which some of the extracted plurality of estimated contexts overlap each other into one estimated context, and generates a second vector of a combined vector obtained by combining the first document vectors. To do. The cluster classification unit 60 performs the second clustering for all the second vectors and determines the final classification.

  The document data classification device 1 can classify topics at high speed even when a plurality of topics to be classified into a plurality of fields coexist in one document.

JP 2009-277099 A JP 2009-211177 A

  As disclosed in Patent Document 1, a document vector widely used as a feature amount for calculating similarity between documents in document categorization for classifying an entire document set into an arbitrary document group is a word appearance frequency or a word It is generated from a uniquely determined concept vector, and the document similarity between documents is represented by the similarity between document vectors. On the other hand, when the similarity between documents is determined by human subjectivity, it is common to determine important topics included in the document based on the commonality between the documents. However, the document vector method generally weights words based on importance, and does not consider the topical viewpoint. Therefore, since the commonality of each important topic cannot be reflected in the similarity, a difference from the subjective similarity determination is caused, and the classification accuracy in document categorization is lowered. In the method of Patent Document 2, for each topic included in a document, only those having a large feature amount similarity are classified into one category, and the document is not classified for each topic.

  In other words, there has conventionally been no device for classifying an arbitrary document group into a predetermined category in a form close to human intuition. The present invention has been made in view of such problems, and an object of the present invention is to provide a document categorizing apparatus, a method thereof, and a program thereof that can realize document classification closer to human intuition.

  The document categorizing apparatus of the present invention includes a feature amount extraction unit, a topic / topic vector / importance level correspondence table, a topic / topic vector / importance level output unit, a category-document vector correspondence table, a similarity comparison and classification unit, Are provided. The feature amount extraction unit uses an input document including a plurality of topics as a reference document, extracts a document vector of the reference document, and outputs the reference document and the document vector. The topic / topic vector / importance correspondence table includes a topic class obtained by clustering topics obtained from a sample document, a topic vector that is the center of gravity of the document vector included in the topic class, and all of the sample documents. And the importance level of the topic calculated from the co-occurrence relationship of the topics included in the reference document are recorded in association with each other. The topic / topic vector / importance output unit obtains the similarity between the vector of the document vector and the topic vector, the topic class corresponding to the topic vector having the large similarity, the importance, and the reference corresponding to the word vector Output document. The category-sample document vector correspondence table classifies the topic vectors of the topic / topic vector / importance correspondence table for each category, and records the centroid of the classified topic vector as a sample document vector corresponding to the category. . The similarity comparison classifying unit obtains a similarity between the topic vector and the sample document vector for each topic class, and accumulates a value obtained by multiplying the similarity by the importance of the topic class for each topic class. Is obtained as the document similarity, and the reference document is classified into the category having the largest document similarity.

  The importance used in the document categorizing apparatus of the present invention reflects the commonality of topics in the similarity based on a certain topic in all the documents of the sample document and the co-occurrence relationship of the topic in the reference document. By classifying reference documents (input documents) for each category using this importance, it is possible to perform document classification close to human senses.

The figure which shows the function structural example of the document categorizing apparatus 100 of this invention. The figure which shows the operation | movement flow of the document categorizing apparatus 100. FIG. The figure which shows the function structural example of a topic / topic vector / importance / correspondence table creation apparatus. The figure which shows the function structural example of the topic class extraction part 210. FIG. The figure which shows the operation | movement flow of the topic class extraction part 210. FIG. The figure which shows the production | generation process of a topic class typically. The figure which shows the difference in the importance of a document typically. The figure which shows the function structural example of the document categorizing apparatus 300 of this invention. The figure which shows the structure of the category-topic class comparison table. The figure which shows the operation | movement flow of the topic class selection part 310 which does not require similarity calculation. The figure which shows the structure of a document-topic class comparison table. The figure which shows the function structure of the conventional document data classification device 1. FIG.

  Embodiments of the present invention will be described below with reference to the drawings. The same reference numerals are given to the same components in a plurality of drawings, and the description will not be repeated.

  FIG. 1 shows a functional configuration example of a document categorizing apparatus 100 according to the present invention. The operation flow is shown in FIG. The document categorizing apparatus 100 includes a feature amount extraction unit 110, a topic / topic vector / importance output unit 120, a topic / topic vector / importance correspondence table 130, a similarity comparison / classification unit 140, and a category-sample document vector. A correspondence table 150 and a control unit 160 are provided. The document categorizing apparatus 100 is realized by, for example, reading a predetermined program into a computer constituted by a ROM, a RAM, a CPU, and the like, and executing the program by the CPU.

  The feature quantity extraction unit 110 extracts a document vector of the reference document as a topic vector using a document corresponding to a certain topic of an input document including a plurality of topics as a topic document, and extracts the topic, the reference document, and the topic. The vector is output (step S110).

  The topic / topic vector / importance correspondence table 130 includes a topic class obtained from a sample document, a topic vector that is the center of gravity of the document vector included in the topic class, and topics included in all documents and reference documents of the sample document. The importance level of the topic calculated from the co-occurrence relationship is recorded in advance in association with each other.

  The topic / topic vector / importance output unit 120 obtains the similarity between the topic vector and the document vector, the topic class, the topic vector, the importance corresponding to the document vector having the similar similarity, and the document vector. The reference document corresponding to is output (step S120). The category-sample document vector correspondence table 150 records a category and a sample document vector that is a centroid of a topic vector for each topic class included in the category in association with each other in advance.

  The similarity comparison and classification unit 140 obtains a similarity between the topic vector and the sample document vector for each topic class, and accumulates a value obtained by multiplying the similarity by the importance of the topic class for each topic class. The document similarity is obtained and the reference document is classified into the category having the largest document similarity (step S140). The reference document is stored in the categorization result storage unit 170 for each category. The control unit 160 controls the operation of each unit.

  The importance recorded in advance in the topic / topic vector / importance correspondence table 130 is calculated from the co-occurrence relationship between all the documents of the sample document and the topics included in the reference document. If the input document is classified for each topic using this importance, it is possible to perform document classification close to a human sense. Detailed explanation of the importance will be described later.

  Hereinafter, the operation of each functional unit will be described in more detail. First, the topic / topic vector / importance correspondence table 130 will be described. The topic / topic vector / importance level correspondence table 130 is generated by the topic / topic vector / importance / correspondence table creation apparatus 200.

[Topic / Topic Vector / Importance / Correspondence Table Creation Device]
FIG. 3 shows a functional configuration example of the topic / topic vector / importance / correspondence table creation apparatus 200. The topic / topic vector / importance / correspondence table creation apparatus 200 includes a topic class extraction unit 210 and an importance level acquisition unit 220. The importance level acquisition unit 220 includes a document number counting unit 221 and a Cullback / liver information amount calculating unit 222.

  The topic / topic vector / importance / correspondence table creation apparatus 200 is also realized by reading a predetermined program into a computer composed of, for example, a ROM, a RAM, a CPU, etc., and executing the program by the CPU. is there.

  The topic class extraction unit 210 receives a sample document group as an input, and extracts a topic class and a document vector of the topic class included in the sample document group. A sample document group is a set of documents in which documents including topics that are samples to be collected are selected for each category by human subjectivity. The topic class is a cluster generated by performing clustering on a document vector generated from a document including each topic. The topic vector of the cluster is the centroid of word vectors included in the topic class.

[Topic class extraction section]
FIG. 4 shows a functional configuration example of the topic class extraction unit 210. The operation flow is shown in FIG. The topic class extraction unit 210 is the same as the document data classification device 1 disclosed in Patent Document 2 of the prior art. Therefore, here, a description will be given for the purpose of clarifying document vectors and topic classes that are strongly related to the present invention.

  The topic class extraction unit 210 includes a document extraction unit 111, a first document vector generation unit 113, a context combination unit 114, a topic class classification unit 115, and a memory 112. The memory 112 corresponds to a RAM when the topic / topic vector / importance / correspondence table creation apparatus 200 is realized by a computer, for example.

  The context extraction unit 111 receives a sample document group as an input, refers to one or more main words for extracting the estimated context recorded in the memory 112, and uses the position of the word of the sample document that matches the main word as a reference. A word string in a predetermined range of the sample document to be determined is extracted as an estimated context (step S111). Each estimated context is stored in memory 112.

  The first document vector generation means 113 reads each estimated context from the memory 112, generates a first document vector that is a document vector of each estimated context, and outputs each first document vector to the memory 112. (Step S113). The document vector is represented by a column vector whose components are weights indicating the appearance tendencies of all the different words included in the document set (for example, reference: Kazuaki Kishida, “Document clustering technique: document review”). , Mita Library and Information Society, NO.49 (2003), p.33-75)). A known method may be used to generate the document vector. For example, a plurality of words may be selected from a document group using a TF-IDF scale, and the appearance frequency (TF) of each selected word in the estimated context may be used as an element of the document vector of the estimated context. Alternatively, a document vector with a reduced number of dimensions may be generated based on the inter-word co-occurrence frequency matrix (for example, reference: Takenobu Tokunaga, Junichi Sakurai, “Information Retrieval and Language Processing, Language and Calculation”, Tokyo University Press, 1999). In addition, an estimated context ID for identifying each corresponding estimated context is assigned to each generated first document vector. The first document vector and its ID are stored in the memory 112.

  The context combining unit 114 reads the first document vector stored in the memory 112 and performs first clustering. For a cluster to which a plurality of first document vectors belong, a synthesized vector obtained by synthesizing a plurality of first document vectors belonging to the cluster is output as a document vector, and for a cluster to which one first document vector belongs, The first document vector is output as a document vector (step S114). The document vector is stored in the memory 112.

  The topic class classification unit 115 reads the document vector from the memory 112, performs the second clustering for all the document vectors, outputs the result of the second clustering to the memory 112, and stores it therein (step S115). ). The process of step S115 can be realized using various clustering methods. For example, it can be realized by using a non-hierarchical clustering method (k-means method) by division optimization. The result of the second clustering is that the document vector and the document number from which the document vector is obtained are associated and classified, and constitutes a topic class.

FIG. 6 schematically shows a topic class generation process. The first line is a main word for extracting the estimated context, and shows how the main words are collected to form the documents D ₁ , D ₂ , and D ₃ . The second line shows the extracted estimated context containing the main word. The third line shows clustered topic classes.

The topic class i is an identifier representing a topic such as i = a (for example, weather), i = b (for example, economy), i = c (for example, sports), i = d (for example, politics), or the like. In the topic class i = a (weather), the document vector (a ₁ ) included in the document D ₁ and the document vector (a ₂ ) included in the document D ₂ are classified. The topic class i = b (economy) includes a document vector (b ₁ ) included in the document D ₁ , a document vector (b ₂ ) included in the document D ₂ , and a document vector included in the document D _3. (B ₃ ) is classified. In the topic class i = c (sports), the document vector (c ₂ ) included in the document D ₂ and the document vector (c ₃ ) included in the document D ₃ are classified. In the topic class i = d (politics), the document vector (d ₃ ) included in the document D ₃ is classified.

  The word vector of each topic class i is averaged for each topic class to be a topic vector that is the center of gravity of the vector representing the topic class.

[Importance acquisition part]
The importance level acquisition unit 220 includes a document counting unit 221 and a cullback / liver information amount calculating unit 222. The document counting means 221 counts the number of documents included in each topic class with the topic class i as an input. In the example shown in FIG. 6, the number of documents of topic class i = a is 2, the number of documents of topic class i = b is 3, the number of documents of topic class i = c is 2, and the number of documents of topic class i = d is 1. Therefore, the total number of documents is counted as eight.

  The cullback / liver information amount calculation means 222 obtains an appearance probability distribution P (i) including a certain topic i, and calculates a difference in distribution from the appearance probability distribution Q (i) of the topic i in all documents Q in the topic class. The importance is calculated by the following formula.

This scale reflects the difference between the co-occurrence relationship of a topic class i with all other topic classes and the co-occurrence relationship of the topic in all documents (Q) as the importance of the topic class i. It is. A specific example of the co-occurrence relationship will be described with reference to the example shown in FIG. In the topic class i = a, documents including a are documents D ₁ and D ₂ , so the co-occurrence number is 2. Since the documents including a in the topic class i = b are the documents D ₁ and D ₂ , the co-occurrence number is 2. Article containing a in topic class i = c co-occurrence count since only D ₂ is 1. Since there is no document including a in the topic class i = d, the co-occurrence number is 0. Since all co-occurrence numbers are 2 + 2 + 1 = 5, the occurrence probability distribution P (i) of topic class i includes P (i) = 2/5, and the appearance probability of topic i in all documents Q The distribution Q (i) is Q (i) = 2/8.

  Substituting this value into equation (1) to calculate the importance of topic class i = a yields 0.073. Similarly, the importance level of the topic class i = b is 0.0, the importance level of the topic class i = c is 0.016, and the importance level of the topic class i = d is 0.167. Table 1 shows the relationship between the co-occurrence numbers for which the importance is calculated.

重要度の順番は、この例では、ｄ（政治）＞ａ（天気）＞ｃ（スポーツ）＞ｂ（経済）となった。

In this example, the order of importance is d (politics)> a (weather)> c (sports)> b (economics).

  The reason why the importance of this embodiment becomes like this is schematically shown in FIG. 70 is an estimation context, and there are a plurality of them. Reference numerals are omitted. The dashed-dotted ellipse indicating the topic class Pe indicates that there is a co-occurrence relationship with the other topic classes Pf and Pg. Topics such as the topic class Pe that have many co-occurrence relationships with other topic classes are less important than the independent topic classes Pf and Pg. A topic with many co-occurrence relationships is relatively low in importance because it includes formal content, interjections such as interjections and impression verbs.

  The topic / document vector / importance / correspondence table 130 can be created by associating and recording the topic class, the topic vector, and the importance that are output by the topic / document vector / importance / correspondence table creation apparatus 200.

[Feature Extraction Unit]
The feature quantity extraction unit 110 extracts a document vector of the reference document by using a part of the input document including a plurality of topics corresponding to a certain topic as a reference document, and outputs the reference document and the document vector. The feature quantity extraction unit 110 is substantially the same as the topic class extraction unit 210 of the topic / topic vector / importance / correspondence table creation apparatus 200 described above. The feature quantity extraction unit 110 performs processing from the context extraction unit 111 to the context combination unit 114 of the topic class extraction unit 210 using an input document including a plurality of topics as an input. As a result of the processing, the document vector and the reference document corresponding to the vector are output to the topic / document vector / importance level output unit 120.

[Topic / Topic Vector / Importance Output Unit]
The topic / topic vector / importance output unit 120 obtains the similarity between the document vector output by the feature amount extraction unit 110 and the topic vector recorded in the topic / topic vector / importance correspondence table 130, A topic class, a document vector, an importance level corresponding to the topic vector having a high similarity, and a reference document of the topic vector are output.

  The similarity between vectors is evaluated using, for example, a cosine similarity expressed by the following equation.

a _n is a document vector, and P (i) is a topic vector.

[Category-document vector correspondence table]
The category-sample document vector correspondence table 150 is created from the topic / topic vector / importance level correspondence table 130 described above. A category-sample is obtained by classifying the topic vectors in the topic / topic vector / importance correspondence table 130 for each category of the sample document and recording the centroid of the classified topic vectors as the sample document vector corresponding to the category. This is a document vector correspondence table 150.

[Similarity comparison and classification unit 140]
The similarity comparison and classification unit 140 calculates the similarity between the word vector output by the topic / topic vector / importance output unit 120 for each topic class and the sample document vector recorded in the category-document vector correspondence table 150. A value obtained by multiplying the similarity by the importance of the topic class and a cumulative value for each topic class is obtained as a document similarity, and a topic / topic vector / importance output unit is assigned to the category having the highest document similarity. The reference documents output by 120 are classified.

  Various methods can be applied as the similarity calculation method. In this embodiment, a value obtained by normalizing the cosine similarity between the document vectors with the importance of each topic is added to calculate the document similarity of the document. To do.

In calculating the similarity between two documents, the cosine similarity cos (a ₁ , a ₂ ) between the topic vectors (a ₁ , a ₂ ) of the topic class a included in both is used as the topic similarity S (a ₁ , a ₂ , a ₂ ), and the Kullback-Roller information amount D _KL (A‖Q) between the topic probability distribution in all the documents Q and the topic probability distribution in the document set A including the topic class a is the topic importance I ( As a), the similarity between topics is weighted and added with the importance of each topic to obtain the document similarity between documents.

For example, in the case of one category and one document, when the sample document D ₁ and the reference document D ₂ have topics (a ₁ , b ₁ ) and (a ₂ , b ₂ ) of topic classes a and b, respectively, the document similarity Is represented by Formula (3).

Further, in the case of one category plural documents, the topic vector (a _c1 , b _c1 ) of the category C ₁ in the category C ₁ composed of the plurality of sample documents D ₁ and D ₂ is _converted into the sample documents D ₁ and D ₂ . The arithmetic average of topic vectors a _c1 = 1/2 · (a ₁ + a ₂ ), b _c1 = 1/2 · (b ₁ + b ₂ ), and the document similarity between the category C ₁ and the reference document D ₃ is It is represented by Formula (4).

  The importance level I (•) is calculated from the co-occurrence relationship of topics included in all the documents of the sample document and the input document (reference document), and the input document is determined using the importance level I (•). By classifying by topic, it is possible to prevent low-priority topics from becoming the target of document similarity determination, and to solve problems that differ from the results of conventional document categorization based on human subjectivity. . In addition, the reliability of the similarity that has been reduced due to the influence of words of low importance can be improved, and the task of manually selecting words and topics to eliminate topics of low importance is unnecessary. can do.

  The document categorizing apparatus 100 described above needs to calculate the document similarity from the co-occurrence relationship of topics included in all the documents of the sample document and the input document (reference document), and the calculation amount may increase. Therefore, a document categorizing apparatus 300 devised to reduce the amount of calculation will be described next.

  FIG. 8 shows a functional configuration example of the document categorizing apparatus 300. The document categorizing apparatus 300 differs from the document categorizing apparatus 100 in that it includes a topic class selection unit 310 that does not require similarity calculation and a category-topic class comparison table 320. Further, the control unit 160 is different from the configuration in that the control unit 160 becomes the control unit 160 ′. Other components are the same as those in the document categorizing apparatus 100 as is clear from the reference numerals.

[Category-topic class comparison table]
FIG. 9 illustrates the structure of the category-topic class comparison table 320. For example, categories are arranged in the row direction, and topic classes i are arranged in the column direction, whereby the category-topic class comparison table 320 is configured.

For example, category C ₁ is “politics”, category C ₂ is “sports”, and category C ₃ is “weather”. The topic classes for the category are, for example, “election”, “prime”, “soccer”, “baseball”, “low pressure”, and “typhoon”.

  The category-topic class comparison table 320 is created in advance according to the target reference document. The creation may be performed manually or may be generated using a computer from the result of morphological analysis of a large number of documents.

[Similarity calculation unnecessary topic class selection part]
FIG. 10 shows an operation flow of the topic level selection unit 310 that does not require similarity calculation. The operation will be described with reference to the operation flow.

The similarity calculation unnecessary topic class selection unit 310 refers to the category-topic class comparison table 320 to search for a category including a certain topic class in a cross-sectional manner, and to select a category including a certain topic class that is less than a predetermined number. Extract (step S310). In this process, for example, a category including a topic of topic class i = 3 is searched in the vertical direction in the example of FIG. 9, and a category whose number is smaller than a predetermined number (for example, 5) is extracted. 9, the category comprising topic class i = 3 illustrates the case where only the category C _2. The predetermined number is a number determined according to the number of categories of the reference document to be processed.

The category _{C 2} only Included set of topics class _{g e (2),} when the set of others was _{g o (2),} for example, if the topic class was i = 1~4, _{g e ( 2)} = {3}, go ₍₂₎ = {1, 2, 4}. Let the topic vector of each topic class be V (i) (i = 1, 2,..., K). FIG. 11 shows the correspondence between documents and topic classes as a document-topic class comparison table. For each document, topic vectors of topic classes are arranged in order. Article _{D 2} of the second row represents that does not include the topic classes i = 2, 3, 4. The relationship shown in FIG. 11 is the same for the reference document. The correspondence between the document and the topic class is included in the information output from the topic / topic vector / importance output unit 120.

  Next, the similarity calculation unnecessary topic class selection unit 310 obtains the maximum category similarity of documents having topic classes included in categories other than the extracted category j (formula (5)) (step S311).

Here, s ≠ j means a category other than the extracted category. _Wi is the importance (weight) of the topic class i, and 1 is the maximum value of the similarity of the topic vector. Therefore, the maximum value of the similarity of categories other than the extracted category can be obtained.

  Next, the minimum value of the category similarity of the document having the topic class included in the extracted category is obtained (formula (6)) (step S312).

  Here, ε is the minimum value of the similarity between any two topic vectors in the class when the topic class is divided by, for example, division optimization clustering. ε is a value considerably smaller than 1.

  Next, a topic class included in a category having the minimum value larger than the maximum value is associated with the category as a similarity calculation unnecessary topic class (step S313). Here, as is clear from the relationship of ε << 1, a topic class having a similarity that satisfies this relationship is a topic class strongly related to the category, and is classified into the category without calculating the document similarity. I can do it. The topic class i is associated as “topic class i” ∈ “a set of topic classes included only in a certain category”.

  Next, the similarity calculation unnecessary topic class selection unit 310 receives the reference document output from the topic / topic vector / importance output unit 120, and the topic class included in the reference document matches the similarity calculation unnecessary topic class. Then, the reference document is classified into the category and output to the outside (categorization result accumulation unit 170), and the reference document that does not include the similarity calculation unnecessary topic class is output to the similarity comparison classification unit 140 (step) S314).

  As described above, by including the topic level selection unit 310 that does not require similarity calculation and the category-topic class comparison table 320, it is possible to determine whether it is necessary to calculate the document similarity from the topic class of the reference document. Since it is possible, the amount of calculation can be reduced.

  It is assumed that the maximum value of the similarity obtained with ε cannot be obtained because the value of ε is small. In that case, the minimum value of the similarity may be obtained by Expression (7) when the reference document is input. As the maximum value at that time, the maximum value obtained by the equation (8) is used.

Here, | ge _(j) | is the number of topic classes included in the set of topic classes g _{e (j)} included only in a certain category, and V _r (i) is the topic vector of a reference document.

  Further, the similarity calculation in the similarity comparison and classification unit 140 may be calculated by the above formulas (3) and (4), or the similarity score RS shown in the formula (9) is calculated to determine the similarity. You may do it.

  Next, a document categorizing apparatus 400 capable of reducing the amount of calculation further than the document categorizing apparatus 300 will be described. The illustration of the document categorizing apparatus 400 is omitted. The document categorizing apparatus 400 is obtained by replacing the similarity calculation unnecessary topic class selecting unit 310 of the document categorizing apparatus 300 with a similarity calculation unnecessary topic class selecting unit 410.

  Similarity calculation unnecessary topic class selection unit 410 differs only in that, when a reference document is input, a process of specifying a category range including a topic class included in the reference document is performed. After the category range is specified, only the specified range of the category-topic class comparison table is the processing target.

When r is a set of topic classes included in the reference document, the amount of calculation can be reduced by narrowing down to a set of categories C _r (equation (10)) including part or all of r in the topic class. .

  The processing after narrowing down the specific range of the category-topic class comparison table 320 is the same as that of the topic class selection unit 310 that does not require similarity calculation.

  According to the document categorizing apparatus 400, since the range of the category to be calculated is narrowed down when the reference document is input, the amount of calculation can be further reduced. After a certain amount of reference document is input, whether the category-topic class comparison table in the latest specific range can be used from the topic class of the reference document without performing the calculation of equation (10). It is also possible to make a judgment.

  Further, the example in which the minimum value of the similarity is obtained by the equation (7) when the reference document is input when the maximum value of the similarity obtained by ε described in the second embodiment is not obtained has been described. Before calculating the minimum value, the range of the target category-topic class comparison table may be narrowed down using Expression (10), and then the minimum value may be calculated using Expression (7). By doing so, the amount of calculation can be reduced as compared with the method described in the second embodiment.

  The processes described in the above apparatuses and methods are not only executed in time series according to the order of description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. .

  Further, when the processing means in the above apparatus is realized by a computer, the processing contents of functions that each apparatus should have are described by a program. Then, by executing this program on the computer, the processing means in each apparatus is realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used. Specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape or the like, and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording medium, MO (Magneto Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a recording device of a server computer and transferring the program from the server computer to another computer via a network.

  Each means may be configured by executing a predetermined program on a computer, or at least a part of these processing contents may be realized by hardware.

Claims (8)

A feature quantity extraction unit that takes an input document including a plurality of topics as a reference document, extracts a document vector of the reference document, and outputs the reference document and the document vector;
The topic class obtained by clustering the topics obtained from the sample document, the topic vector that is the center of gravity of the document vector included in the topic class, the topic included in all the documents of the sample document and the reference document A topic / topic vector / importance correspondence table recorded in association with the importance of the topic calculated from the co-occurrence relationship of
A topic / topic that obtains a similarity between vectors of the document vector and the topic vector, and outputs a topic class and importance corresponding to the topic vector having a large similarity and a reference document corresponding to the word vector A vector / importance output section;
A category-sample document vector correspondence table in which the topic vectors of the topic / topic vector / importance correspondence table are classified for each category and the center of gravity of the classified topic vectors is recorded as a sample document vector corresponding to the category. When,
The similarity between the topic vector and the sample document vector is obtained for each topic class, and a value obtained by multiplying the similarity by the importance of the topic class is accumulated as the document similarity. A similarity comparison and classification unit that classifies the reference document into a category having the highest document similarity,
A document categorizing apparatus comprising: The document categorizing apparatus according to claim 1,
The above importance is
A document categorizing apparatus, characterized in that it is based on a difference between an appearance probability distribution of a certain topic in the reference document and an appearance probability distribution of the topic in all documents of the sample document. The document categorizing apparatus according to claim 1,
The above importance is
It is given by the amount of information of Cullback-Librer that calculates the difference between two probability distributions, and is defined by the following equation:

It is given by the difference in distribution between the appearance probability distribution P (i) of the topic class i in the set P of documents included in a certain topic class p and the appearance probability distribution Q (i) of the topic class i in all documents Q. Feature document categorizing device. The document categorizing apparatus according to any one of claims 1 to 3,
The above document similarity is
A value obtained by multiplying the cosine similarity between the topic vector for each topic and the sample document vector by the amount of the cullback / railer information representing the difference in probability distribution between the topic vector and the sample document vector for each topic. A document categorizing apparatus characterized by being an accumulated value for all topics. The document categorizing apparatus according to any one of claims 1 to 4,
Furthermore,
A category-topic class comparison table that records the correspondence between categories and topic classes included in the category;
With reference to the category-topic class comparison table, a category including a certain topic class is traversed to extract a category including a certain topic class less than a predetermined number,
Find the maximum category similarity of documents with topic classes included in categories other than the extracted categories,
Find the minimum category similarity of documents with topic classes included in the extracted category,
A topic class included in a category in which the minimum value is larger than the maximum value is associated with the category as a similarity calculation unnecessary topic class,
When the reference document is input and the topic class included in the reference document matches the similarity calculation unnecessary topic class, the reference document is classified into the category and output to the outside, and the similarity calculation unnecessary topic is output. A similarity calculation unnecessary topic class selection unit that outputs a reference document not including a class to the similarity comparison and classification unit;
A document categorizing apparatus comprising: The document categorizing apparatus according to any one of claims 1 to 4,
Furthermore,
A category-topic class comparison table that records the correspondence between categories and topic classes included in the category;
When the reference document is input, the range to be referred to in the category-topic class comparison table is specified according to the topic class included in the reference document.
With reference to the category-topic class comparison table in the specific range, a category including a certain topic class is traversed to extract a category including a certain topic class less than a predetermined number,
Find the maximum category similarity of documents with topic classes included in categories other than the extracted categories,
Find the minimum category similarity of documents with topic classes included in the extracted category,
A topic class included in a category in which the minimum value is larger than the maximum value is associated with the category as a similarity calculation unnecessary topic class,
When the reference document is input and the topic class included in the reference document matches the similarity calculation unnecessary topic class, the reference document is classified into the category and output to the outside, and the similarity calculation unnecessary topic is output. A similarity calculation unnecessary topic class selection unit that outputs a reference document not including a class to the similarity comparison and classification unit;
A document categorizing apparatus comprising: A feature amount extraction process in which an input document including a plurality of topics is used as a reference document, a document vector of the reference document is extracted, and the reference document and the document vector are output;
Topic class obtained by cluster classification of topics obtained from sample documents recorded in the above document vector, topic / topic vector / importance correspondence table, and topic vector that is the centroid of the document vector included in the topic class To the topic class corresponding to the topic vector having the large similarity and all the documents of the sample document recorded in the topic / topic vector / importance correspondence table and the reference document. A topic / topic vector / importance output process for outputting the importance of the topic calculated from the co-occurrence relationship of the included topics and a reference document corresponding to the word vector;
For each topic class, the topic vectors of the topic / topic vector / importance correspondence table recorded in the topic vector / category-sample document vector correspondence table are classified for each category, and the centroid of the classified topic vectors is used. A similarity between a sample document vector is obtained, a value obtained by multiplying the similarity by the importance of the topic class for each topic class is obtained as a document similarity, and the largest document similarity is obtained. A similarity comparison and classification process that classifies reference documents into categories;
A document categorizing method comprising:   A program for causing a computer to function as the document categorizing apparatus according to any one of claims 1 to 6.

Images