JP4940251B2 - Document processing program and document processing apparatus - Google Patents

Description

  The present invention relates to a document processing program and a document processing apparatus for classifying a large number of document groups.

  In recent years, document processing apparatuses that classify a large number of documents (sentences) into several similar document sets (clusters) are known.

  As a document classification method in this document processing apparatus, for example, there is an inter-document similarity calculation method using a vector space model composed of words appearing in a document.

  As a related technology, for example, call centers and mail centers, etc. Electronic texts gathered in companies and local governments, voices of customers and residents, and business reports created by salespeople are automatically classified with high accuracy. Techniques that can be performed (hereinafter referred to as prior art) are disclosed (see, for example, Patent Document 1).

  In the prior art, the idea of classifying the input data (data to be classified) based on the already classified data (correct answer set), and these data as text (text data) and non-text data It is divided into (data in a narrow sense), and a concept of calculating a reference value for judging similarity by mining (text mining and data mining) and classifying based on the reference value is adopted.

JP 2005-71229 A

  However, in the above-described prior art, no classification is performed in a document in consideration of the meaning of sentences included in the document. In other words, since “important description part” and “non-important part” in the document are handled with equal weight, the classification accuracy is low.

  By the way, as an evaluation method of classification accuracy, it is performed based on whether documents classified into the same cluster have the same meaning or whether “A (object) B (verb)” is the same. Is common.

  Therefore, in order to improve the classification accuracy, it is necessary to perform classification in consideration of a set of “object” and “verb”.

  Accordingly, an object of the present invention is to provide a document processing program and a document processing apparatus capable of improving the document classification accuracy.

According to one aspect of the present invention, a document configured by a document storage unit that stores a plurality of documents including sentences including character strings, an external storage device having a feature storage unit, and a computer that uses the external storage device. In a processing apparatus, a document processing program executed by the computer, wherein each document stored in the computer is stored in the computer based on the appearance frequency of the character string in the document. Extracting a character string as an important word, extracting a sentence including the extracted important word from a document from which the important word is extracted as a summary sentence, and characters included in the extracted summary sentence a step of analyzing the dependency between the columns, on the basis of key words and the analysis results are within the spirit sentence the extracted, the gist sentence By simplifying the representation of the non-important word included, composed of a plurality of character strings and generating a paraphrase sentence of the summary sentences including the key words, key words included in the generated paraphrase sentence A feature set to be extracted from the paraphrase sentence, a step of storing the extracted feature set in the feature storage unit, and a document stored in the document storage unit for each document Calculating a document vector component value based on the appearance frequency of the feature set stored in the feature storage means in the summary sentence extracted from the document, and based on the calculated document vector component value, the document storage means generating a document vector for each document that is stored in, based on the extracted important word and the analysis results, and object or verb the important word That template to execute the steps of generating, in the step of extracting a set of feature, wherein the relative generated paraphrase sentence, extracts a set of feature by matching the generated template A document processing program is provided.

  According to the present invention, it is possible to improve document classification accuracy.

1 is a block diagram showing a hardware configuration of a document processing apparatus according to a first embodiment of the present invention. FIG. 2 is a block diagram mainly showing a functional configuration of the document processing apparatus 30 shown in FIG. 1. 6 is a flowchart showing a processing procedure of the document processing apparatus 30 according to the present embodiment. The figure which shows an example of the dependency analysis result by the dependency analysis part 313. FIG. The figure for demonstrating the specific example of the paraphrase process in case there is one important word. The figure for demonstrating the specific example of the paraphrase process in case there are two important words. The block diagram which mainly shows a function structure of the document processing apparatus which concerns on the 2nd Embodiment of this invention. 6 is a flowchart showing a processing procedure of the document processing apparatus 40 according to the present embodiment. The figure which shows an example of the template produced | generated by the template production | generation part 41 when the important word extracted by the important word extraction part 311 is "analog". The figure which shows an example of the template produced | generated by the template production | generation part 41 when the important words extracted by the important word extraction part 311 are "analog" and "conversion". The figure for demonstrating the group of the features extracted when a paraphrase sentence does not match a template. The block diagram which mainly shows a function structure of the document processing apparatus which concerns on the 3rd Embodiment of this invention. 6 is a flowchart showing a processing procedure of the document processing apparatus 50 according to the present embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
First, a first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing a hardware configuration of the document processing apparatus according to the present embodiment. As shown in FIG. 1, the computer 10 is connected to an external storage device 20 such as a hard disk drive (HDD). The external storage device 20 stores a program 21 executed by the computer 10. The computer 10 and the external storage device 20 constitute a document processing device 30.

  FIG. 2 is a block diagram mainly showing a functional configuration of the document processing apparatus 30 shown in FIG. As shown in FIG. 2, the document processing apparatus 30 includes a summary sentence specifying unit 31, a paraphrase processing unit 32, and a clustering unit 33. In the present embodiment, these units 31 to 33 are realized by the computer 10 illustrated in FIG. 1 executing the program 21 stored in the external storage device 20. This program 21 can be stored in advance in a computer-readable storage medium and distributed. The program 21 may be downloaded to the computer 10 via, for example, a network.

  The document processing apparatus 30 includes a document storage unit 22, a synonym phrase dictionary storage unit 23, a paraphrase sentence storage unit 24, and a document cluster storage unit 25. In the present embodiment, the document storage unit 22, the synonym phrase dictionary storage unit 23, the paraphrase sentence storage unit 24, and the document cluster storage unit 25 are stored in, for example, the external storage device 20.

  The document storage unit 22 stores a plurality of document data (text data). This document data (hereinafter referred to as document) is composed of a sentence including a character string.

  The abstract sentence specifying unit 31 includes an important word extracting unit 311, a target sentence extracting unit 312, and a dependency analyzing unit 313.

  Based on the appearance frequency of the character string (word) in the document stored in the document storage unit 22, the important word extraction unit 311 extracts a character string that is important in the document as an important word.

  The target sentence extracting unit 312 extracts a sentence (target sentence) including the important word extracted by the important word extracting unit 311 from the document from which the important word is extracted. Since the target sentence includes the important word extracted by the important word extraction unit 311, it is an important description part in the document from which the target sentence is extracted. Therefore, the target sentence extracted by the target sentence extraction unit 312 is referred to as a summary sentence (of the document).

  The dependency analysis unit 313 analyzes the dependency between character strings included in the summary sentence extracted by the target sentence extraction unit 312 (dependency analysis). A specific example of this dependency analysis result will be described later. By executing this dependency analysis, for example, the part of speech of a character string (important word) included in the summary sentence extracted by the target sentence extraction unit 312 can be acquired.

  In the synonym phrase dictionary storage unit 23, synonymous expressions of words (character strings) are stored in advance. Specifically, for example, “video” is stored in the synonym phrase storage unit 23 as a synonymous expression of the word “image”. The synonym dictionary storage unit 23 is used, for example, in synonym processing for unifying the important words extracted by the important word extraction unit 311.

  The paraphrase processing unit 32 includes a number determination unit 321 and a paraphrase text generation unit 322. The number determination unit 321 determines whether the number (number) of important words included in the summary sentence extracted by the target sentence extraction unit 312 is one, or two or more.

  The paraphrase sentence generation unit 322 converts the key word extracted by the key word extraction unit 311, the dependency analysis result of the summary sentence by the dependency analysis unit 313, and the synonym expression stored in the synonym phrase dictionary storage unit 23. In other words, the summary sentence extracted by the target sentence extraction unit 312 is paraphrased. As a result, the paraphrase text generation unit 322 executes a process (paraphrase process) of generating a paraphrase text of the abstract sentence extracted by the target sentence extraction unit 312. The paraphrase text generated by the paraphrase text generation unit 322 includes the key word extracted by the key word extraction unit 311.

  In addition, the paraphrase text generation unit 322 executes paraphrase processing according to the determination result by the number determination unit 321. Details of the paraphrase process will be described later.

  The paraphrase text generation unit 322 stores the paraphrase text of the generated summary text in the paraphrase text storage unit 24. That is, the paraphrase text storage unit 24 stores the paraphrase text of the abstract sentence extracted from each document stored in the document storage section 22, that is, the paraphrase text for each document stored in the document storage section 22. The

  The clustering unit 51 classifies the paraphrase text stored in the paraphrase text storage unit 24. For example, the clustering unit 51 classifies the paraphrase text based on the appearance frequency of the character string included in the paraphrase text stored in the paraphrase text storage unit 24. The classification result by the clustering unit 51 is stored in the document cluster storage unit 27.

  Next, a processing procedure of the document processing apparatus 30 according to the present embodiment will be described with reference to the flowchart of FIG. Note that the processing described below is executed in response to an instruction (operation) to the user, for example.

  First, the important word extraction unit 311 included in the summary sentence specification unit 31 acquires one of the documents (group) stored in the document storage unit 22 from the document storage unit 22 (step S1).

  Next, the keyword extraction unit 311 extracts the keyword in the document based on the appearance frequency of the character string (word) in the acquired document (step S2). Specifically, the keyword extraction unit 311 extracts the keyword using a feature word extraction method such as determining a characteristic word based on, for example, a TF / IDF value (score). There may be a plurality of important words extracted by the important word extraction unit 311.

  The important word may be extracted from the entire document acquired by the important word extracting unit 311 or from a specific part such as an important paragraph (for example, the first paragraph) in the document. The configuration may be extracted. That is, the important word may be extracted based on the position in the document acquired by the important word extracting unit 311.

  The target sentence extraction unit 312 extracts a summary sentence (target sentence) including the important word extracted by the important word extraction unit 311 in the document acquired by the important word extraction unit 311 (step S3). There may be a plurality of abstract sentences extracted by the target sentence extraction unit 312.

  The abstract sentence may be extracted from the entire document acquired by the important word extracting unit 311 or may be extracted from a specific portion (important paragraph) in the document. I do not care. That is, the summary sentence may be extracted based on the position in the document acquired by the keyword extraction unit 311.

  The dependency analysis unit 313 performs dependency analysis on the summary sentence extracted by the target sentence extraction unit 312 (step S4).

  Here, FIG. 4 shows an example of a dependency analysis result by the dependency analysis unit 313. In FIG. 4, for example, the important words extracted by the important word extraction unit 311 are “analog” and “conversion”, and the abstract extracted by the target sentence extraction unit 312 is “input an analog image and execute conversion. ”Is a dependency analysis result by the dependency analysis unit 313.

  The dependency analysis unit 313 performs dependency analysis on the summary sentence to acquire a part of speech such as a character string (important word) “analog” or “conversion” included in the summary sentence.

  Next, the number determination unit 321 included in the paraphrase processing unit 32 determines whether the number of important words included in the summary sentence extracted by the target sentence extraction unit 312 is one or two or more. .

  The paraphrase sentence generation unit 322 is extracted by the target sentence extraction unit 312 based on the important words extracted by the important word extraction unit 311, the dependency analysis result by the dependency analysis unit 313 and the determination result by the number determination unit 321. A paraphrase process (paraphrase process) is performed. Thereby, the paraphrase sentence production | generation part 322 produces | generates the paraphrase sentence of the summary sentence extracted by the target sentence extraction part 312 (step S5).

  At this time, the paraphrase sentence generation unit 322 uses a synonym expression stored in the synonym phrase dictionary storage unit 23 to convert a character string such as an important word included in the abstract sentence extracted by the target sentence extraction unit 312. The synonym processing is executed for it.

  In addition, the paraphrase text generation unit 322 stores the generated paraphrase text in the paraphrase text storage unit 24.

  Here, the paraphrasing process described above is a process for paraphrasing the complicated expression of the abstract sentence extracted by the target sentence extracting unit 312 into a simpler expression. In this paraphrase process, for example, first to fourth paraphrase processes are performed.

  The first paraphrase process is simplification of noun phrases. Specifically, the first paraphrase process is a process in which the expression “B of A” is simply replaced with the expression “A”. In other words, in the first paraphrase process, the expression “B” in the expression “B of A” is omitted.

  The second paraphrase process is simplification of the functional verb phrase. Specifically, the second paraphrasing process is a process of simply paraphrasing the expression “execute A” to the expression “execute A”.

  The third paraphrase process is simplification of subordinate phrases. Specifically, the third paraphrasing process is a process of simply rephrasing the expression “do A when B is A” and simply “express B as A and C”.

  The fourth paraphrase process is simplification of the case sharing structure. Specifically, the fourth paraphrasing process is a process of simply paraphrasing the expression “A to B and C” into the expression “A to C”. That is, in the fourth paraphrase process, the expression “B” is omitted in the expression “A to B and C”.

  The paraphrase text generation unit 322 generates a paraphrase text of the abstract sentence extracted by the target sentence extraction unit 312 by the first to fourth paraphrase processes described above.

  In addition, although a paraphrase sentence is produced | generated by the above-mentioned 1st-4th paraphrase process, when the said paraphrase sentence is produced | generated, the important word contained in the said summary sentence is not abbreviate | omitted. In other words, not all the first to fourth paraphrasing processes are applied to all the abstract sentences.

  Specifically, for example, when the abstract sentence is “A to B and C”, according to the above fourth paraphrasing process, it can be rephrased as “A to C”, but “B” is an important word. In such a case, since the important word is omitted when the fourth paraphrase process is applied, the fourth paraphrase process is not applied to the summary sentence “A to B and C”.

The paraphrase processing by paraphrase sentence generating unit 322 paraphrase processing is performed by performing a pruning Ri or the like on the above-mentioned first to fourth paraphrase processing other than the example dependency analysis result (parse tree) It doesn't matter. It should be noted, it is a process of removing the branches cutting Ritowa, unnecessary representation from the syntax tree (a string).

  Here, with reference to FIG. 5, a specific example of the paraphrase process in the case where there is one important word included in the abstract will be described. Here, the important word extracted by the important word extraction unit 311 is “analog”, and the abstract extracted by the target sentence extraction unit 312 is “input an analog image and execute conversion”. .

  In FIG. 5, the abstract sentence 101 and the paraphrase sentences 102 to 104 of the abstract sentence 101 are shown in the form of the dependency analysis result as shown in FIG.

  First, when the first paraphrasing process described above is applied to the abstract sentence 101, the abstract sentence “input an analog image and execute conversion” 101 is changed to the paraphrase sentence “input analog and execute conversion” 102. (Step S11).

  Next, when the second paraphrase process described above is further applied to the paraphrase sentence 102, the paraphrase sentence “input analog and execute conversion” 102 is paraphrased as paraphrase sentence “input analog and convert” 103. (Step S12).

  From this paraphrase sentence "input and convert analog" 103, paraphrase sentences "input analog" and "convert analog" 104 are generated (step S13).

  In this way, the paraphrase sentence generation unit 322 performs the paraphrase process on the abstract sentence “input an analog image and execute conversion” 101 to thereby change the paraphrase sentences “input analog” and “convert analog”. “Yes” 104 is generated. The paraphrase text “input analog” and “convert analog” 104 generated by the paraphrase text generation unit 322 are stored in the paraphrase text storage unit 24.

  In the example shown in FIG. 5, the paraphrase sentence “convert analog” can be considered by applying the fourth paraphrase process described above to the paraphrase sentence “input and convert analog” 103. . However, in the paraphrase sentence "input and convert analog" 103, "input" and "conversion" are considered to be parallel to the important word "analog" and have the same weight, so only one is omitted. Such a paraphrase process is not performed.

  Moreover, depending on the result of the dependency analysis by the dependency analysis unit 313 for the summary sentence extracted by the target extraction unit 312 as described above, the number of paraphrase sentences generated from the summary sentence is not limited to one, As shown in FIG. 5, two or more paraphrase sentences may be generated.

  Next, a specific example of the paraphrasing process when there are two important words included in the abstract will be described with reference to FIG. Here, the important words extracted by the important word extraction unit 311 are “analog” and “conversion”, and the abstract extracted by the target sentence extraction unit 312 is “input analog image and execute conversion”. It shall be. That is, the abstract sentence 201 shown in FIG. 6 is the same as the abstract sentence 101 shown in FIG.

  6, similar to FIG. 5 described above, the abstract sentence 201 and the paraphrase sentences 202 to 204 of the abstract sentence 201 are shown in the form of dependency analysis results.

  First, when the first paraphrase process described above is applied to the abstract sentence 201, the abstract sentence “input an analog image and execute conversion” 201 is reworded as an alternative sentence “input analog and execute conversion” 202. (Step S21).

  Next, when the second paraphrase process described above is further applied to the paraphrase sentence 202, the paraphrase sentence “input analog and execute conversion” 202 is paraphrased as the paraphrase sentence “input analog and convert” 203. (Step S22).

  Here, in the example shown in FIG. 6, unlike the case where there is one important word included in the above-described abstract, “input” and “conversion” are parallel to the important word “analog”, but the “ “Conversion” is an important word, and it is considered that “import”, which is the important word, has a higher weight than “input”. Therefore, the above-described fourth paraphrase process is applied to the paraphrase sentence 203. Thereby, the paraphrase sentence “input analog and convert” 203 is paraphrased into the paraphrase sentence “convert analog” 204 (step S23).

  As described above, the paraphrase sentence generation unit 322 generates the paraphrase sentence “convert analog” 204 by executing the paraphrase process on the abstract sentence “input analog image and execute conversion” 201. The paraphrase text “convert analog” 204 generated by the paraphrase text generation unit 322 is stored in the paraphrase text storage unit 24.

  When there are three or more important words in the summary sentence, the above-described process shown in FIG. 6 is executed for each combination of two important words of the important words.

  Returning to FIG. 3 again, it is determined whether or not the processing in steps S1 to S5 described above has been executed for all documents stored in the document storage unit 22 (step S6).

  When it is determined that the processing in steps S1 to S5 described above has not been executed for all the documents stored in the document storage unit 22 (NO in step S6), the processing returns to the above step S1 and is repeated. . In this case, in step S <b> 1, a document for which the processing in steps S <b> 1 to S <b> 5 has not been executed is acquired from the document storage unit 22.

  On the other hand, when it is determined that the processing of step S1 to step S5 has been executed for all the documents stored in the document storage unit 22 (YES in step S6), the clustering unit 33 is stored in the paraphrase text storage unit 24. Are classified (clustered) (step S7). For example, the clustering unit 33 performs document classification based on the appearance frequency of a character string included in the paraphrase text. Here, the description has been made assuming that the classification process is executed based on the appearance frequency of the character string included in the paraphrase text, but various methods other than the method described here are conceivable as the paraphrase text classification method.

  The paraphrase text storage unit 24 stores a paraphrase text for each document stored in the document storage unit 22 as described above.

  That is, the clustering unit 51 classifies the document group stored in the document storage unit 22 by classifying the paraphrase text stored in the paraphrase text storage unit 24. The classification result of the document group stored in the document storage unit 22 by the clustering unit 33 is stored in the document cluster storage unit 27.

  As described above, in the present embodiment, for each document stored in the document storage unit 22, a summary sentence including the important word extracted by the important word extraction unit 311 is extracted. In the present embodiment, dependency analysis is performed on the extracted summary sentence, and paraphrase processing is performed on the summary sentence based on the important word and the dependency analysis result. Therefore, in the present embodiment, it is possible to classify the document group by classifying the paraphrase text for each document stored in the document storage unit 22.

  In the present embodiment, for example, classification processing such as dependency analysis is performed only on a paraphrase sentence, compared to the case where the document is classified based on the entire document stored in the document storage unit 22, for example. Since the classification process is not executed for sentences and the like that are not important in the document, the classification accuracy can be improved and the processing amount can be reduced.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIG. FIG. 7 is a block diagram mainly showing a functional configuration of the document processing apparatus according to the present embodiment. The same parts as those in FIG. 2 described above are denoted by the same reference numerals, and detailed description thereof is omitted. Here, parts different from FIG. 2 will be mainly described.

  The hardware configuration of the document processing apparatus according to this embodiment is the same as that of the first embodiment described above, and will be described with reference to FIG. 1 as appropriate. The same applies to the following embodiments.

  In the present embodiment, a feature set, which will be described later, is extracted from a sentence of a paraphrase text (paraphrase text generated by the paraphrase text generation unit 322) stored in the paraphrase text storage unit 24, and a document based on the feature set is extracted. The point which produces | generates a document vector for every document stored in the storage part 22 differs from 1st Embodiment mentioned above.

  As illustrated in FIG. 7, the document processing apparatus 40 includes a template generation unit 41, a feature extraction unit 42, a feature output unit 43, and a document vector processing unit 44. In the present embodiment, these units 41 to 44 are realized by the computer 10 shown in FIG. 1 executing the program 21 stored in the external storage device 20.

  The document processing apparatus 40 includes a feature storage unit 26 and a document vector storage unit 27. In the present embodiment, the feature storage unit 26 and the document vector storage unit 27 are stored in, for example, the external storage device 20.

  The template generation unit 41 generates a template composed of the key words based on the key word extracted by the key word extraction unit 311 and the dependency analysis result of the summary sentence by the dependency analysis unit 313. Details of the data structure of the template generated by the template generation unit 41 will be described later.

  The feature extraction unit 42 extracts a set of features from the sentence of the paraphrase text (paraphrase text generated by the paraphrase text generation unit 322) stored in the paraphrase text storage unit 24. The feature extraction unit 42 matches the template generated by the template generation unit 41 with the paraphrase text stored in the paraphrase text storage unit 24. As a result, the feature extraction unit 42 extracts a feature set including important words included in the paraphrase text stored in the paraphrase text storage unit 24 from the paraphrase text. The feature extraction unit 42 stores the extracted feature set in the feature storage unit 26.

  The feature set extracted by the feature extraction unit 42 includes, for example, a set of “object” and “verb” or a set of “object”, “tool case”, and “verb”.

  The feature output unit 43 outputs (displays) the feature set extracted by the feature extraction unit 42 to the user.

  The document vector processing unit 44 includes a document vector component value calculation unit 441 and a document vector generation unit 442.

  The document vector component value calculation unit 441 calculates a document vector component value for each document stored in the document storage unit 22. The document vector component value calculation unit 441 stores each feature stored in the feature storage unit 26 in a summary sentence extracted from a document stored in the document storage unit 22 (a summary sentence extracted by the target sentence extraction unit 312). The document vector component value is calculated based on the appearance frequency of the set. The document vector component value calculation unit 441 calculates the document vector component values of the number of feature pairs stored in the feature storage unit 26 for one document.

  The document vector component value may be calculated using a word weight calculation method such as a mutual information amount.

  The document vector generation unit 442 generates a document vector for each document stored in the document storage unit 22 based on the document vector component value calculated by the document vector component value calculation unit 441.

  The document vector generation unit 442 stores the document vector generated for each document stored in the document storage unit 22 in the document vector storage unit 27. This document vector is used, for example, when classifying documents (groups) stored in the document storage unit 22.

  Next, a processing procedure of the document processing apparatus 40 according to the present embodiment will be described with reference to the flowchart of FIG.

  First, the process of step S31-step S35 equivalent to the process of step S1-step S5 shown in FIG. 3 mentioned above is performed. The paraphrase text generated in step S35 is stored in the paraphrase text storage unit 24 as described above.

  Next, the template generation unit 41 generates a template composed of the important words based on the important words extracted by the important word extraction unit 311 and the dependency analysis result by the dependency analysis unit 313 (step S36). .

  Here, FIG. 9 shows an example of a template generated by the template generation unit 41 when the keyword extracted by the keyword extraction unit 311 is “analog”, for example. The template generation unit 41 acquires the part of speech (here, a noun) of the important word “analog” based on the dependency analysis result by the dependency analysis unit 313. As a result, the template generation unit 41 generates a template 301 having the keyword “analog” as the “object” as shown in FIG.

  FIG. 10 shows an example of a template generated by the template generation unit 41 when the important words extracted by the important word extraction unit 311 are “analog” and “conversion”, for example. The template generation unit 41 acquires parts of speech (here, nouns and verbs) of the important words “analog” and “conversion” based on the dependency analysis result by the dependency analysis unit 313. As a result, the template generation unit 41 generates a template 302 having the important word “analog” as “object” and the important word “conversion” as “verb” as shown in FIG.

  When there are three or more important words extracted by the important word extraction unit 311, for example, a noun and verb pair is generated as a template from the important words. Further, even when there are two important words extracted by the important word extraction unit 311, when the two important words are both nouns, each important word has been described with reference to FIG. 9 described above. Such a template (a template having the important word as the “object”) is generated. That is, a plurality of templates may be generated according to the important word extracted by the important word extraction unit 311.

  Returning again to FIG. 8, the feature extraction unit 42 is composed of key words or character strings (features) from the paraphrase text stored in the paraphrase text storage unit 24 using the template generated by the template generation unit 41. A set (feature group) is extracted (step S37). The feature extraction unit 42 extracts a set of features by matching the paraphrase text stored in the paraphrase text storage unit 24 with the template generated by the template generation unit 41. The feature set is composed of, for example, “object” and “verb”.

  Depending on the paraphrase text, the key word, and the template, a feature set composed of the above-mentioned “object”, “tool case”, and “verb” may be extracted. In addition, when a plurality of templates are generated as described above, feature set extraction processing is executed for each template.

  The feature set extracted by the feature extraction unit 42 is stored in the feature storage unit 26. At this time, in the feature storage unit 26, for example, the same feature set extracted from different paraphrase sentences is treated as one feature set.

  The feature output unit 43 outputs (displays) the feature set extracted by the feature extraction unit 42 to, for example, the user. Thereby, the user can check the feature set extracted by the feature extraction unit 42, that is, the expression (character string) included in the feature set.

  Here, the feature set extraction processing by the feature extraction unit 42 will be described in detail. For example, the important word is one of “analog”, the template of the important word “analog” is the template 301 shown in FIG. 9 described above, and the paraphrase text stored in the paraphrase text storage unit 24 is the above-described FIG. It is assumed that “analog input” and “analog conversion” 104 are performed as described in FIG. In this case, since the important word “analog” is used as an object in the paraphrased sentences “input analog” and “convert analog” 104, the feature extraction unit 42 selects “object” and “verb”. (Analog, input) and (analog, conversion) feature pairs are extracted.

  On the other hand, there are two important words “analog” and “conversion”, and the templates of the important words “analog” and “conversion” are the templates 302 shown in FIG. 10 described above, and are stored in the paraphrase sentence storage unit 24. It is assumed that the paraphrase text is “convert analog” 204 as described in FIG. In this case, in the paraphrase sentence “convert analog” 204, the important word “analog” is used as an object, and the important word “conversion” is used as a verb. Matches the template 302 shown. For this reason, the feature extraction unit 42 extracts a set of features (analog, conversion) composed of “object” and “verb”.

  Here, the case where the paraphrase text matches the template in the case where the number of important words is one or two has been described. However, referring to FIG. 11, a set of features extracted when the paraphrase text does not match the template is described below. Will be described.

  For example, there are two important words “analog” and “conversion”, and the templates of the important words “analog” and “conversion” are the templates 302 shown in FIG.

  Further, the abstract sentence extracted by the target sentence extraction unit 312 is assumed to be “converting characters into speech and outputting them in analog”.

  In FIG. 11, a summary sentence “convert characters into speech and output in analog” 402 is shown in the form of dependency analysis results.

As shown in FIG. 11, by benefit of this spirit sentence 402 for example pruning, "and outputs the character in analog and converted to speech" SUMMARY sentence 401 "outputs an analog converts" text paraphrase paraphrased to 402 Shall be. In other words, it is assumed that the paraphrase text “convert and output in analog” 402 is generated by the paraphrase text generation unit 322.

  In this case, in the paraphrase sentence “convert and output in analog” 402, the important word “conversion” is used as a verb, but since the important word “analog” is not used as an object, the paraphrase sentence 402 is It does not match the template 302 shown in FIG.

  In this case, the feature extraction unit 42 sets the key words “analog” and “conversion” as a set of features. That is, the feature extraction unit 42 extracts, for example, (conversion, analog) and (analog, conversion) as feature sets. In this case, unlike the case of matching with the template described above, a set of features in which “object” and “verb” are not considered is extracted.

  As described above, depending on the abstract sentence extracted by the target sentence extraction unit 312, the paraphrase sentence generated by the paraphrase sentence generation unit 322 does not match the template generated by the template generation unit 41, and the “object” and A set of features that does not consider “verb” is extracted.

  Note that if the paraphrase text includes one important word and the paraphrase text does not match the template, the key word cannot be made into a feature set, and the feature set is not extracted.

  Returning to FIG. 8 again, it is determined whether or not the processing in steps S31 to S37 described above has been executed for all the documents stored in the document storage unit 22 (step S38).

  If it is determined that the processing in steps S31 to S37 described above has not been executed for all the documents stored in the document storage unit 22 (NO in step S38), the processing returns to the above step S1 and is repeated. . In this case, in step S31, a document for which the processing in steps S31 to S37 has not been executed is acquired from the document storage unit 22.

  On the other hand, when it is determined that the processing in steps S31 to S37 has been executed for all the documents stored in the document storage unit 22 (YES in step S38), the document vector component value included in the document vector processing unit 44 is calculated. The unit 441 acquires one of the documents stored in the document storage unit 22 from the document storage unit 22 (step S39). Hereinafter, the document acquired by the document vector component value calculation unit 441 is referred to as a target document.

  Next, the document vector component value calculation unit 441 calculates the document vector component value of the target document based on the set of features stored in the summary sentence and the feature storage unit 26 extracted from the target document (step) S40). The document vector component value calculation unit 441 calculates a document vector component value indicating the appearance frequency of the feature set in the summary sentence extracted from the target document for each feature set stored in the feature storage unit 26. That is, the document vector component value calculation unit 441 calculates the document vector component value for each feature set stored in the feature storage unit 26 for one target document.

  A feature set appears in a summary sentence extracted from a target document means that a feature set, for example, a “object” and “verb” set appears in the summary sentence. Specifically, for example, assuming that the feature set is (analog, input), when “analog” is used as the object and “input” is used as the verb in the abstract extracted from the target document The appearance frequency of this feature set in the summary sentence is, for example, 1. In addition, when this feature group appears twice or more in one summary sentence, the value of the appearance frequency becomes large.

  Here, the document vector component value calculation unit 441 has been described as calculating the appearance frequency of the feature set in the abstract of the target document as the document vector component value. However, as described above, for example, a word such as a mutual information amount is calculated. The document vector component value may be calculated using a weight calculation method. The mutual information amount is an amount representing the degree of association between the words based on the degree of appearance of two words appearing in the document at the same time.

  The document vector generation unit 442 generates a document vector of the target document based on the document vector component value calculated by the document vector component value calculation unit 441 (step S41).

  The document vector generation unit 441 stores the generated document vector in the document vector storage unit 27. The document vector stored in the document vector storage unit 27 is used, for example, when a plurality of documents stored in the document storage unit 22 are classified.

  For example, it is assumed that a feature set (analog, input) and (analog, conversion) are stored in the feature storage unit 26. In this case, if the appearance frequency of the feature set (analog, input) in the abstract extracted from the target document is 1 and the appearance frequency of the feature set (analog, conversion) is 0, the document vector of the target document d is , D (1,0).

  Note that 1 in the document vector d (1, 0) is a document vector component value of a feature set (analog, input) in the target document. Similarly, 0 in the document vector d (1, 0) is a document vector component value of a feature set (analog, conversion) in the target document.

  As described above, the document vector is generated by combining the document vector component values calculated for each feature set.

  When the process of step S41 is executed as described above, it is determined whether or not the processes of step S39 to step S41 described above have been executed for all the documents stored in the document storage unit 22 (step S42). .

  If it is determined that the processes in steps S39 to S41 described above have not been executed for all the documents stored in the document storage unit 22 (NO in step S42), the process returns to the above step S39 and is repeated. . In this case, in step S39, a document for which the processes in steps S39 to S41 are not executed is acquired from the document storage unit 22.

  On the other hand, when it is determined that the processes in steps S39 to S41 have been executed for all the documents stored in the document storage unit 22 (NO in step S42), the process ends.

  As described above, a vector whose dimension is the number of feature pairs extracted from all the documents stored in the document storage unit 22 (different number) is a document vector, and the value of the document vector is set for each document. By calculating, a document vector corresponding to each document is generated.

  As described above, in the present embodiment, for each document stored in the document storage unit 22, a summary sentence including the important word extracted by the important word extraction unit 311 is extracted. In this embodiment, dependency analysis is performed on the extracted summary sentence, and the paraphrase process is performed on the summary sentence based on the important word and the dependency analysis result, so that it is appropriate for document classification. The feature set can be extracted. Therefore, in the present embodiment, since a document vector can be generated based on an appropriate feature set in document classification, the accuracy of document classification performed using the document vector can be improved.

  Further, in the present embodiment, since dependency analysis is performed only on the extracted summary sentence, useless dependency analysis processing can be reduced. In the present embodiment, the number of feature sets extracted is used as the number of dimensions of the document vector, so that the number of dimensions of the document vector can be reduced without reducing the document classification accuracy. Processing speed can be increased.

[Third Embodiment]
Next, a third embodiment of the present invention will be described with reference to FIG. FIG. 12 is a block diagram mainly showing a functional configuration of the document processing apparatus according to the present embodiment. The same parts as those in FIGS. 2 and 7 described above are denoted by the same reference numerals, and detailed description thereof is omitted. Here, parts different from FIGS. 2 and 7 will be mainly described.

  In the present embodiment, the document classification (clustering) of the document (group) stored in the document storage unit 22 using the document vector stored in the document vector storage unit 27 is the first and second described above. This is different from the embodiment.

  As shown in FIG. 12, the document processing apparatus 50 includes a document classification processing unit 51. In the present embodiment, the document classification processing unit 51 is realized by the computer 10 illustrated in FIG. 1 executing the program 21 stored in the external storage device 20.

  The document classification processing unit 51 includes a similarity calculation unit 511 and a clustering unit 512. The similarity calculation unit 511 calculates the similarity between the document vectors using the document vectors stored in the document vector storage unit 27.

  The clustering unit 512 performs clustering (classification of document groups) of document groups stored in the document storage unit 22 based on the similarity (value) calculated by the similarity calculation unit 511.

  The clustering unit 512 stores the document group classification result stored in the document storage unit 22 in the document cluster storage unit 25.

  Next, a processing procedure of the document processing apparatus 50 according to the present embodiment will be described with reference to a flowchart shown in FIG.

  First, the process of step S51-step S62 equivalent to the process of step S31-step S42 shown in FIG. 8 mentioned above is performed. The document vector generated in step S61 is stored in the document vector storage unit 27 as described above.

  Next, the similarity calculation unit 511 included in the document classification processing unit 51 calculates the similarity between the document vectors based on the document vectors for each document stored in the document vector storage unit 27 (step S63). . The similarity calculation unit 511 calculates the similarity between document vectors based on the document vector component value for each set of features in the document vector.

  The clustering unit 512 executes clustering processing (classification processing) of the document group stored in the document storage unit 22 based on the similarity calculated by the similarity calculation unit 511 (step S64).

  The clustering unit 512 clusters the documents stored in the document storage unit 22 by collecting documents having similar (similar) similarity values between documents calculated by the similarity calculation unit 511. )

  The clustering unit 512 stores the document group classification result stored in the document storage unit 22 in the document cluster storage unit 25.

  As described above, in this embodiment, similarly to the second embodiment described above, a document vector can be generated based on an appropriate feature set in document classification, and a document can be generated using the document vector. The classification process of the document group stored in the storage unit 22 can be executed. Therefore, in the present embodiment, the classification accuracy of the document group stored in the document storage unit 22 can be improved.

  In the present embodiment, it has been described that the process of step S63 is executed when it is determined that the process has been executed for all the documents in step S62 shown in FIG. 13, but for all the documents in step S62. When it is determined that the process has been executed, that is, after the document vectors are generated for all the documents stored in the document storage unit 22, the process of step S63 is executed when the user gives an instruction. It does not matter. In other words, the document vector generation process described in the second embodiment and the document group classification process performed using the document vector may be executed separately.

  Note that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in each embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 10 ... Computer, 20 ... External storage device, 22 ... Document storage part, 23 ... Synonym dictionary storage part, 24 ... Paraphrase sentence storage part, 25 ... Document cluster storage part, 26 ... Feature storage part, 27 ... Document vector storage , 30, 40, 50... Document processing apparatus, 31... Abstract sentence specifying part, 32... Paraphrase processing part, 33... Clustering part, 41. Document vector processing unit 51 ... Document classification processing unit, 311 ... Multiple term extraction unit, 312 ... Target sentence extraction unit, 313 ... Dependency analysis unit, 321 ... Number determination unit, 322 ... Paraphrase sentence generation unit, 441 ... Document vector Component value calculation unit, 442... Document vector generation unit, 511... Similarity calculation unit, 512.

Claims (4)

Executed by a computer in a document processing apparatus comprising a document storage means for storing a plurality of documents including sentences including character strings, an external storage device having a feature storage means, and a computer using the external storage device A document processing program,
In the computer,
For each document stored in the document storage means, based on the appearance frequency of the character string in the document, extracting a character string that is important in the document as an important word;
Extracting the sentence including the extracted important word from the document from which the important word is extracted as a summary sentence;
Analyzing a dependency between character strings included in the extracted summary sentence;
Based on the important words contained in the extracted summary sentences and the analysis results, the expression other than the important words contained in the summary sentences is simplified to generate a paraphrase text of the summary sentences including the important words. And steps to
Extracting a feature set composed of a plurality of character strings including important words included in the generated paraphrase text from the paraphrase text;
Storing the extracted feature sets in the feature storage means;
For each document stored in the document storage means, calculating a document vector component value based on the appearance frequency of the feature set stored in the feature storage means in the summary sentence extracted from the document;
Generating a document vector for each document stored in the document storage means based on the calculated document vector component value ;
Generating a template based on the extracted important word and the analysis result, and using the important word as an object or a verb ,
In the step of extracting the feature set, the feature set is extracted by matching the generated template with the generated paraphrase sentence.
A document processing program characterized by that . In the computer,
Calculating a similarity between the document vectors based on the document vectors for each document stored in the generated document storage means;
The document processing program according to claim 1, further comprising: classifying a plurality of documents stored in the document storage unit based on the calculated similarity. In a document processing apparatus comprising a document storage means for storing a plurality of documents composed of sentences including character strings and a paraphrase text storage means, and a computer using the external storage apparatus, the document processing apparatus is executed by the computer. A document processing program,
In the computer,
For each document stored in the document storage means, based on the appearance frequency of the character string in the document, extracting a character string that is important in the document as an important word;
Extracting the sentence including the extracted important word from the document from which the important word is extracted as a summary sentence;
Analyzing a dependency between character strings included in the extracted summary sentence;
Based on the extracted important words included in the extracted summary sentence and the analysis result, by simplifying the expression other than the important words included in the summary sentence , the document in which the important words are extracted Generating a corresponding paraphrase text;
Storing the generated paraphrase text in the paraphrase text storage means;
Classifying the paraphrase text based on a character string included in the paraphrase text stored in the paraphrase text storage means;
Classifying a document stored in the document storage unit corresponding to the paraphrase text based on a classification result of the paraphrase text. Document storage means for storing a plurality of documents consisting of sentences including character strings;
For each document stored in the document storage means, based on the appearance frequency of the character string in the document, an important word extraction means for extracting a character string that is important in the document as an important word;
A summary sentence extracting means for extracting a sentence including the important word extracted by the important word extracting means as a summary sentence from the document from which the important word is extracted;
Analyzing means for analyzing a dependency between character strings included in the abstract sentence extracted by the abstract sentence extracting means;
Based on the important words included in the abstract sentence extracted by the abstract sentence extraction means and the analysis result by the analysis means , the expression other than the important words included in the abstract sentence is simplified to include the important words. Paraphrase text generating means for generating a paraphrase text of the summary text ;
A feature extraction unit that extracts a set of features composed of a plurality of character strings including important words included in the paraphrase text generated by the paraphrase text generation unit, from the paraphrase text;
Feature storage means for storing a set of features extracted by the feature extraction means;
For each document stored in the document storage unit, a calculation unit that calculates a document vector component value based on the appearance frequency of the feature stored in the feature storage unit in the summary sentence extracted from the document;
Document vector generation means for generating a document vector for each document stored in the document storage means based on the document vector component value calculated by the calculation means ;
Generating a template with the important word as an object or a verb based on the extracted important word and the analysis result ,
The feature extraction unit extracts the feature set by matching the generated template with the generated paraphrase sentence.
A document processing apparatus characterized by that.

Images