JP5117590B2 - Document processing apparatus and program - Google Patents

Description

  Embodiments described herein relate generally to a document processing apparatus and program for extracting synonyms from a document.

  In general, in document processing, the handling of another term representing the same meaning as a certain term (word), that is, a synonym is an important issue.

  For example, “eraser” and “eraser” are considered to be synonyms representing the same meaning, but information indicating that “eraser” and “eraser” are synonyms (hereinafter referred to as synonym information). Is not prepared in advance, it is not easy to search or generate one from the other (for example, “eraser” to “eraser”).

  Specifically, in document search, it is difficult to search for a document including “eraser” from the search keyword “eraser” without using synonym information indicating that “eraser” and “eraser” are synonyms. is there. In addition, in the document classification, if synonym information indicating that "eraser" and "eraser" are synonyms is not prepared, the document including "eraser" and the document including "eraser" are essentially the same classification. Although it should belong to, it may be classified as a different category.

  Therefore, in such a case, a document search or document classification is executed by referring to a synonym dictionary in which terms (for example, “eraser” and “eraser”) related to the above-described synonyms are registered in advance. It is known to do.

  However, since the number of terms having a synonym relationship is enormous, it is difficult to prepare all the terms having the synonym relationship in advance manually and enormously.

  Thus, for example, it is considered that synonyms are automatically extracted from a document using the context similarity or the character string similarity of terms appearing in a given document (set). According to this, it is possible to register all terms having a synonym relationship in the synonym dictionary without manually preparing in advance.

  Note that the context similarity is a similarity calculated based on the distribution hypothesis that “semanically similar words appear in similar contexts”. For example, the terms of dependency terms or co-occurring terms Calculated as similarity. Further, the character string similarity is the similarity of the character strings themselves constituting the terms, and is calculated according to, for example, the number of characters common to the two terms.

JP-A-5-346938 JP 2000-222427 A

  However, the above-described context similarity and character string similarity are based on local similarity in the document.

  For this reason, when a set of terms (synonym sets) having a synonym relation is extracted from a document using only the context similarity or the string similarity, the influence of local information is strong and is not suitable as a synonym. A term (noise term) may be included in the synonym set. Specifically, when a synonym set is extracted simply using context similarity or character string similarity, in addition to the appropriate synonyms “eraser” and “eraser”, the synonym is included in the synonym set. For example, “cancellation” may be included.

  When a document search or document classification is performed with reference to a synonym dictionary that contains a set of synonyms (terms related to synonyms) that contain terms that are not suitable as synonyms, an appropriate result is obtained. You may not get it.

  Therefore, a problem to be solved by the present invention is to provide a document processing apparatus and program capable of extracting an appropriate term as a synonym from a document.

  The document processing apparatus according to the present embodiment includes a document storage unit, a term extraction unit, a similarity calculation unit, a cluster generation unit, a feature calculation unit, and a synonym extraction unit.

  The document storage means stores a plurality of documents including terms consisting of one or a plurality of words.

  The term extracting unit extracts the first and second terms from the terms included in the plurality of documents stored in the document storing unit.

  The cluster generation unit generates a cluster to which each of the plurality of documents stored in the document storage unit belongs.

  The feature degree calculating unit is configured to determine the first term for the cluster based on the appearance frequency of the extracted first term in the plurality of documents stored in the document storing unit and the document belonging to the generated cluster. A feature degree is calculated, and based on the appearance frequency of the extracted second term in the plurality of documents stored in the document storage unit and the document belonging to the generated cluster, the second term for the cluster is calculated. The feature degree is calculated.

  Based on the similarity calculated by the similarity calculation unit, the feature level of the first term and the feature level of the second term calculated by the feature level calculation unit, the first and second terms are synonymous. Extract as

1 is a block diagram showing a hardware configuration of a document processing apparatus according to a first embodiment. FIG. 2 is a block diagram mainly showing a functional configuration of the document processing apparatus 30 shown in FIG. 1. The figure which shows an example of the data structure of the document database 22 shown in FIG. 6 is a flowchart showing a processing procedure of the document processing apparatus 30 according to the present embodiment. The figure which shows an example of the data structure of the analysis result storage part. The figure which shows an example of the data structure of the analysis result storage part. The figure which shows an example of the data structure of the term totalization result storage part 24. FIG. The figure for demonstrating the cluster produced | generated by the cluster production | generation part 35. FIG. The flowchart which shows the process sequence of a similarity calculation process. The figure which shows an example of the data structure of the intermediate process result information produced | generated by the similarity calculation part. The figure which shows an example of the data structure of the similarity calculation result storage part 25 after the group of the term A and the term B is stored. The figure which shows an example of the data structure of the similarity calculation result storage part 25 after the appearance frequency of the term A is stored. The figure which shows an example of the data structure of the similarity calculation result storage part 25 after the context similarity of the term A and the term B is stored. The figure which shows an example of the data structure of the similarity calculation result storage part 25 after the character string similarity of the term A and the term B is stored. The flowchart which shows the process sequence of a feature degree calculation process. The figure which shows an example of the data structure of the characteristic calculation result storage part 26 after the cluster appearance frequency in the clusters 1-3 is stored for every term. The figure which shows an example of the data structure of the characteristic calculation result storage part 26 after the characteristic with respect to the clusters 1-3 of each term is stored. The figure for demonstrating the terminology with respect to the clusters 1-3. The flowchart which shows the process sequence of a synonym set extraction process. The figure which shows an example of the data structure of the synonym set storage part 27 after the similarity calculation result is stored. The figure which shows an example of the data structure of the synonym set storage part 27 after the characteristic degree calculation result is stored. The figure which shows the determination result by the synonym set extraction part 37. FIG. The figure for demonstrating the synonym set extraction process in the case of the cluster 1. FIG. The figure for demonstrating the synonym set extraction process in the case of the cluster 3. FIG. The figure which shows an example of the display screen in case the two terms extracted as a synonym by the synonym set extraction part 37 are displayed. 10 is a flowchart showing a processing procedure of the document processing apparatus 30 according to the second embodiment. The flowchart which shows the process sequence of a similarity calculation process. The figure which shows an example of the data structure of the intermediate process result information produced | generated by the similarity calculation part. The figure which shows an example of the data structure of the similarity calculation result storage part 25 after a process is performed about each of the clusters 1-3. The flowchart which shows the process sequence of a synonym set extraction process.

  Hereinafter, each embodiment will be described with reference to the drawings.

(First embodiment)
First, the first embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a block diagram showing a hardware configuration of the document processing apparatus according to the present embodiment. As shown in FIG. 1, the computer 10 is connected to an external storage device 20 such as a hard disk drive (HDD). The external storage device 20 stores a program 21 executed by the computer 10. The computer 10 and the external storage device 20 constitute a document processing device 30.

  FIG. 2 is a block diagram mainly showing a functional configuration of the document processing apparatus 30 shown in FIG. As shown in FIG. 2, the document processing apparatus 30 includes an input processing unit 31, an analysis unit 32, a term totaling unit 33, a similarity calculation unit, a cluster generation unit 35, a feature calculation unit 36, a synonym set extraction unit 37, and an output. A processing unit 38 is included. In the present embodiment, these units 31 to 38 are realized by the computer 10 illustrated in FIG. 1 executing the program 21 stored in the external storage device 20. This program 21 can be stored in advance in a computer-readable storage medium and distributed. Further, this program 21 may be downloaded to the computer 10 via, for example, a network.

  In addition, the document processing apparatus 30 includes a document database (DB) 22, an analysis result storage unit 23, a term aggregation result storage unit 24, a similarity calculation result storage unit 25, a feature calculation result storage unit 26, and a synonym set storage unit 27. Including. In the present embodiment, the document database (DB) 22, the analysis result storage unit 23, the term aggregation result storage unit 24, the similarity calculation result storage unit 25, the feature calculation result storage unit 26, and the synonym set storage unit 27 are, for example, external It is stored in the storage device 20.

  In the document database 22, a plurality of documents to be processed by the document processing apparatus 30 are stored in advance. The document stored in the document database 22 includes a term consisting of one or more words.

  The input processing unit 31 processes a user instruction input or external data input. The input processing unit 31 receives an instruction to execute processing of the document processing device 30 in accordance with, for example, a user operation.

  The analysis unit 32 analyzes a plurality of documents stored in the document database 22 (for example, morphological analysis and syntax analysis). Thereby, the analysis unit 32 acquires analysis results of a plurality of documents. The analysis results of a plurality of documents include terms included in the plurality of documents and dependency relationships between the terms. Note that the analysis result acquired by the analysis unit 32 is stored in the analysis result storage unit 23.

  Based on the analysis result stored in the analysis result storage unit 23, the term aggregation unit 33 calculates the appearance frequency (appearance frequency in a plurality of documents stored in the document database 22) for each term included in the analysis result. To do. Thereby, the term totaling unit 33 acquires a term totaling result including the appearance frequency for each term. The term totaling result acquired by the term totaling unit 33 is stored in the term totaling result 24.

  Based on the analysis result stored in the analysis result storage unit 23, the similarity calculation unit 34 includes terms included in the analysis result (that is, terms included in a plurality of documents stored in the document database 22). To extract two terms (first and second terms). In this case, the similarity calculation unit 34 extracts, for example, two terms (a term in which the part of speech is a noun) having the same dependency relationship as the same term (a term in which the part of speech is a verb). The similarity calculation unit 34 calculates the similarity between the two extracted terms.

  The similarity calculation unit 34 includes a context similarity calculation unit 341 and a character string similarity calculation unit 342.

  Based on the analysis result stored in the analysis result storage unit 23 and the term aggregation result stored in the term aggregation result storage unit 24, the context similarity calculation unit 341 calculates the two terms extracted by the similarity calculation unit 34. The context similarity is calculated as the similarity. Note that the context similarity is a similarity calculated based on a distribution hypothesis that “a semantically similar word appears in a similar context”.

  The character string similarity calculation unit 342 calculates the character string similarity as the similarity between the two terms extracted by the similarity calculation unit 34. The character string similarity is the similarity of the character strings themselves that constitute the term.

  The similarity calculated by the similarity calculation unit 34 (the context similarity calculated by the context similarity calculation unit 341 and the character string similarity calculated by the character string similarity calculation unit 342) is the similarity calculation result storage unit. 25.

  The cluster generation unit 35 generates a cluster to which a plurality of documents stored in the document database 22 belong.

  The feature degree calculation unit 36 calculates the feature degree of each term included in the term aggregation result based on the term aggregation result stored in the term aggregation result storage unit 24. At this time, the feature calculation unit 36 calculates the feature for each cluster generated by the cluster generation unit 35. The feature degree calculated by the feature degree calculation unit 36 is stored in the feature degree calculation result storage unit 26.

  The synonym set extraction unit 37 uses the two terms extracted by the similarity calculation unit 34 based on the similarity stored in the similarity calculation result storage unit 25 and the feature degrees stored in the feature calculation result storage unit 26. Are extracted as synonyms. The processing result by the synonym set extraction unit 37 is stored in the synonym set storage unit 27.

  The output processing unit 38 processes display output to the user or data output to the outside. The output processing unit 38 outputs the synonym set extracted by the synonym set extraction unit 37, for example.

  FIG. 3 shows an example of the data structure of the document database 22 shown in FIG. A plurality of documents including the document 221 are stored in the document database 22 shown in FIG.

  Each document stored in the document database 22 includes a document ID, text, date, creator ID, and classification code in association with each other.

  The document ID is an identifier for identifying a document. The text indicates the content of the document identified by the associated document ID and includes, for example, a term consisting of one or more words. Note that a term composed of a plurality of words includes, for example, a compound word.

  The date indicates the date when the document identified by the associated document ID was created or updated. The creator ID is an identifier for identifying the creator who created the document identified by the associated document ID.

  The classification code indicates the classification to which the document belongs when the document identified by the associated document ID is classified. This classification code is used when, for example, the creator of a document identified by the associated document ID (that is, the creator identified by the associated creator ID) registers the document in the document database 22. Is preset. The classification code may be determined mechanically by executing automatic clustering on a plurality of documents stored in the document database 22, for example.

  In the example illustrated in FIG. 3, the document 221 includes a document ID “1”, a text “Please bring a pencil, an eraser, and a clock”, a date “2010-1-1”, a creator ID “1”, and a classification. The code “A” is included. According to this, the document 221 is a document identified by the document ID “1”, and it is indicated that the content of the document 221 is “Please bring a pencil, an eraser, and a clock.” Further, the document 221 is created on the date “2010-1-1” by the creator ID “1”, and indicates that it belongs to the classification code “A”.

  Here, the document 221 of the plurality of documents stored in the document database 22 has been described, but the same applies to other documents, and thus detailed description thereof is omitted.

  In the example shown in FIG. 3, the text included in each document is composed of only one sentence for convenience, but the text may be composed of a plurality of sentences (two or more sentences).

  Next, the processing procedure of the document processing apparatus 30 according to the present embodiment will be described with reference to the flowchart of FIG. In the following description, it is assumed that the document database 22 stores a plurality of documents shown in FIG.

  First, the input processing unit 31 inputs an instruction (hereinafter referred to as an execution instruction) to execute the processing of the document processing apparatus 30 from the user in accordance with a user operation (step S1).

  When an execution instruction is input by the input processing unit 31, the analysis unit 32 acquires a plurality of documents stored in the document database 22. The analysis unit 32 analyzes the plurality of acquired documents (step S2). At this time, the analysis unit 32 executes, for example, morpheme analysis processing and syntax analysis processing. Thereby, the analysis part 32 acquires the information (henceforth expressed as dependency relationship information) which shows the dependency relationship between a noun and a verb among the dependency tree structure which shows a syntax analysis result, for example.

  The dependency relationship information extracted by the analysis unit 32 is stored in the analysis result storage unit 23 as an analysis result.

  Here, FIG. 5 and FIG. 6 show an example of the data structure of the analysis result storage unit 23. 5 and 6 stores a plurality of pieces of dependency relationship information including dependency relationship information 231 to 233. The dependency relationship information is information indicating a dependency relationship between a noun and a verb as described above.

  The dependency relationship information stored in the analysis result storage unit 23 includes dependency relationship information ID, term 1, term 2, relationship, and document ID for identifying the dependency relationship information. Note that the dependency relationship information ID is assigned to each dependency relationship information acquired by the analysis unit 23.

  The term 1 indicates a dependency source term (that is, a noun) in the dependency relationship between the noun and the verb indicated by the dependency relationship information identified by the corresponding dependency relationship information ID. The term 2 indicates a destination term (that is, a verb) in a dependency relationship between a noun and a verb indicated by the dependency relationship information identified by the corresponding dependency relationship information ID. The relationship indicates the dependency relationship between the noun and the verb indicated by the dependency relationship information identified by the corresponding dependency relationship information ID (that is, the dependency relationship between the term 1 and the term 2). The document ID is a document from which the dependency relationship information identified by the corresponding dependency relationship information ID is acquired (that is, a document in which a dependency relationship between a noun and a verb indicated by the dependency relationship information appears. ) Is an identifier for identifying.

  For example, the dependency relationship information 231 illustrated in FIG. 5 includes dependency relationship information ID “1”, term 1 “pencil”, term 2 “bringing”, relationship “O”, and document ID “1”. According to the dependency relationship information 231 indicated by the dependency relationship information ID “1”, the dependency relationship between the term 1 “pencil (noun)” and the term 2 “bringing (verb)” is “O”. It is shown. Further, according to the dependency relationship information 231 indicated by the dependency relationship information ID “1”, the dependency relationship information 231 is identified by the document ID “1” among a plurality of documents stored in the document database 22. It is shown that it was obtained from the document

  Further, for example, the dependency relationship information 232 illustrated in FIG. 5 includes dependency relationship information ID “6”, term 1 “question”, term 2 “present”, relationship “ga”, and document ID “3”. According to the dependency relationship information 232 indicated by the dependency relationship information ID “6”, the dependency relationship between the term 1 “question (noun)” and the term 2 “presence (verb)” is “ga”. It is shown. Further, according to the dependency relationship information 232 indicated by the dependency relationship information ID “6”, the dependency relationship information 232 is identified by the document ID “6” of the plurality of documents stored in the document database 22. It is shown that it was obtained from the document

  6 includes, for example, dependency relationship information ID “21”, term 1 “erasure”, term 2 “erasure”, relationship “de”, and document ID “8”. . According to the dependency relationship information 233 indicated by the dependency relationship information ID “21”, the dependency relationship between the term 1 “erasing (noun)” and the term 2 “erasing (verb)” is “de”. It has been shown. Further, according to the dependency relationship information 233 indicated by the dependency relationship information ID “21”, the dependency relationship information 233 is identified by the document ID “21” of the plurality of documents stored in the document database 22. It is shown that it was obtained from the document

  Here, the dependency relationship information 231 to 233 among the plurality of dependency relationship information stored in the analysis result storage unit 23 has been described. However, the same applies to the other dependency relationship information. Is omitted.

  Returning to FIG. 4 again, the term totaling unit 33 totals the appearance frequency of terms based on the dependency relationship information stored in the analysis result storage unit 23 (step S3). Specifically, the term totaling unit 33 determines whether all of the dependency relationship information stored in the analysis result storage unit 23 for each term 1 included in the dependency relationship information stored in the analysis result storage unit 23. Appearance frequency information indicating the frequency of appearance of the term 1 (appearance frequency) is acquired.

  The appearance frequency information acquired by the term totaling unit 33 is stored in the term totaling result storage unit 24 as a term totaling result.

  Here, FIG. 7 shows an example of the data structure of the term aggregation result storage unit 24. Note that a plurality of appearance frequency information including appearance frequency information 241 and 242 is stored in the term aggregation result storage unit 24 illustrated in FIG. The appearance frequency information is stored in all the dependency relationship information stored in the analysis result storage unit 23 of the term 1 included in the dependency relationship information stored in the analysis result storage unit 23 (that is, stored in the document database 22). The appearance frequency in a plurality of documents).

  The appearance frequency information stored in the term aggregation result storage unit 24 includes an appearance frequency information ID for identifying the appearance frequency information, a term, and an appearance frequency in association with each other. Note that the appearance frequency information ID is assigned to each appearance frequency information acquired by the term totaling unit 33.

  The term indicates a term whose appearance frequency is indicated by the appearance frequency information identified by the corresponding appearance frequency information ID, and the term 1 (noun) included in the dependency relationship information stored in the analysis result storage unit 23. ). The appearance frequency is an appearance frequency in all the dependency relationship information stored in the analysis result storage unit 23 of the associated term.

  For example, the appearance frequency information 241 illustrated in FIG. 7 includes the appearance frequency information ID “1”, the term “pencil”, and the frequency “1”. The appearance frequency information 241 indicated by the appearance frequency information ID “1” indicates that the appearance frequency of the term “pencil” is 1.

  For example, the appearance frequency information 242 illustrated in FIG. 7 includes an appearance frequency information ID “2”, the term “eraser”, and a frequency “4”. According to the appearance frequency information 242 indicated by the appearance frequency information ID “2”, it is indicated that the appearance frequency of the term “eraser” is 4.

  Here, the appearance frequency information 241 and 242 of the plurality of appearance frequency information stored in the term totalization result storage unit 24 has been described. However, the same applies to other appearance frequency information, and detailed description thereof is omitted. To do.

  Returning to FIG. 4 again, the similarity calculation unit 34 refers to the analysis result storage unit 23 and the term aggregation result storage unit 24 to execute the similarity calculation process (step S4). According to the similarity calculation process, two terms 1 are extracted from the terms 1 included in the dependency relation information stored in the analysis result storage unit 23, and the similarities between the two terms 1 are calculated. . Details of the similarity calculation process will be described later.

  When the similarity calculation process is executed, the similarity calculated by the similarity calculation unit 34 is stored in the similarity calculation result storage unit 25.

  Next, the cluster generation unit 35 generates a cluster to which a plurality of documents stored in the document database 22 belong (step S5). As described above, when the classification code is included in the document stored in the document database 22, the cluster generation unit 35 generates a cluster based on the classification code. Here, as shown in FIG. 8, among the plurality of documents stored in the document database 22, for example, the cluster to which the document including the classification code “A” belongs includes cluster 1 and the classification code “B”. It is assumed that cluster 2 is generated by the cluster generation unit 35 as a cluster to which a document including the classification code “C” belongs, and cluster 2 as a cluster to which the document being included belongs.

  In the present embodiment, a one-to-one cluster corresponding to each classification code included in a plurality of documents stored in the document database 22 is generated. For example, the classification code has a plurality of digits. In this case, a cluster corresponding to the upper N digits on a one-to-one basis may be generated. That is, a configuration in which one cluster is generated for a plurality of classification codes may be used.

  In the present embodiment, each document stored in the document database 22 has been described as including a classification code. However, when each of the documents does not include a classification code, the document For example, an automatic clustering process may be executed on a plurality of documents stored in the database 22, and a cluster may be generated based on the processing result.

  Next, the feature degree calculation unit 36 calculates a feature degree for each cluster generated by the cluster generation unit 35 for each term included in the appearance frequency information stored in the term aggregation result storage unit 24 (hereinafter, referred to as “feature level”). , Described as feature level calculation processing) (step S6). In this feature degree calculation process, the appearance frequency of terms included in the appearance frequency information stored in the term aggregation result storage unit 24 (that is, the appearance frequency associated with the term in the appearance frequency information) and the cluster Based on the appearance frequency of the term in the document belonging to the cluster generated by the generation unit 35, the feature degree of the term with respect to the cluster is calculated. Details of the feature calculation processing will be described later.

  When the feature degree calculation process is executed, the feature degree calculated by the feature degree calculation unit 36 is stored in the feature degree calculation result storage unit 26.

  Next, the synonym set extraction unit 37 extracts a synonym (set) based on the similarity stored in the similarity calculation result storage unit 25 and the feature degree stored in the feature degree calculation result storage unit 26. (Hereinafter referred to as synonym set extraction processing) is executed (step S7). In this synonym set extraction process, the two terms extracted in the above-described similarity calculation process are extracted as synonyms. Details of the synonym set extraction process will be described later.

  When the synonym set extraction processing is executed, the processing result by the synonym set extraction unit 37 (two terms extracted as synonyms by the synonym set extraction unit 37) is stored in the synonym set storage unit 27 and output processing The data is output via the unit 38 (step S8).

  The two terms stored in the synonym set storage unit 27 can be used as synonyms for processing such as document search or document classification. Further, by outputting the two terms extracted by the synonym set extraction unit 37, the user confirms whether the two terms are appropriate as synonyms, and, for example, should register the two terms as synonyms. Can be instructed.

  Next, a processing procedure of the above-described similarity calculation process (the process of step S4 shown in FIG. 4 described above) will be described with reference to the flowchart of FIG.

  First, the similarity calculation unit 34 reads all the dependency relationship information stored in the analysis result storage unit 23 (step S11).

  Next, the similarity calculation unit 34 reads the term 2 (relationship term) and the relationship (dependency relationship) included in the dependency relationship information read from the analysis result storage unit 23 as one set. In all of the dependency relationship information, the number of differences between the term 2 and the term 1 (relationship source term) associated with the set of relationships is counted (step S12). In other words, the similarity calculation unit 34 is different from the term 1 having the same dependency relationship as the same term 2 among the terms 1 included in the dependency relationship information read from the analysis result storage unit 23. Count the number. Note that the process of step S12 is executed for all sets of term 2 and relationships included in each of the dependency relationship information read from the analysis result storage unit 23.

  Next, the similarity calculation unit 34 refers to the analysis result storage unit 23, and associates each of the terms 2 and the relations associated with the pair of terms 2 when the number of differences counted in the process of step S 12 is 2 or more. The appearance frequency of term 1 is counted (step S13). Specifically, as described above, when the number of different terms 1 associated with a set of terms 2 and relationships included in the dependency relationship information is 2 or more, the similarity calculation unit 34 For every one, the appearance frequency (number) of the dependency relationship information including the term 1, the term 2, and the relationship is counted.

  The similarity calculation unit 34 generates intermediate process result information indicating an intermediate process result in the similarity calculation process based on the process results of steps S12 and S13 (step S14). The intermediate processing result information is generated for each term 1 (the term 2 and the set of relations) whose appearance frequency is counted in step S13. The intermediate process result information generated by the similarity calculation unit 34 includes an intermediate process result ID for identifying the intermediate process result information, the term 1 included in the dependency relation information whose appearance frequency is counted in step S13, The term 2 and the set of relationships, and the counted appearance frequency (hereinafter referred to as dependency relationship appearance frequency) are included.

  The similarity calculation unit 34 extracts all combinations of the two terms 1 based on the generated intermediate processing result information (step S15). Specifically, based on the generated intermediate processing result information, the similarity calculation unit 34 selects all combinations of two terms 1 in terms 1 having the same dependency relationship as the same term 2 ( Here, a permutation) is extracted.

  Hereinafter, in the combination of the two terms 1 extracted in step S14, the front term 1 is term A and the rear term 1 is term B. The combination of terms A and B is simply referred to as a set of terms A and B.

  When step S <b> 14 is executed, the similarity calculation unit 34 stores each of the extracted term A and term B pairs in the similarity calculation result storage unit 25. At this time, in addition to the terms A and B, the similarity calculation result storage unit 25 stores the dependency relationship appearance frequency included in the intermediate processing result information in association with the term A.

  Next, in the processes of steps S16 and S17, the context similarity of terms A and B stored in the similarity calculation result storage unit 25 is calculated.

  The context similarity calculation unit 341 included in the similarity calculation unit 34 acquires the appearance frequency of the term A based on the appearance frequency information stored in the term aggregation result storage unit 24 (step S16). In this case, the context similarity calculation unit 341 acquires the appearance frequency associated with the term A in the appearance frequency information stored in the term aggregation result storage unit 24. The appearance frequency of the term A acquired by the context similarity calculation unit 341 is stored in the similarity calculation result storage unit 25 in association with the term A.

  The process of step S16 is executed for each term A stored in the similarity calculation result storage unit 25. Thereby, when the process of step S16 is performed, the appearance frequency for each term A is stored in the similarity calculation result storage unit 25 in association with the term A.

  Next, the context similarity calculation unit 341 refers to the similarity calculation result storage unit 25 and calculates the context similarity of terms A and B stored in the similarity calculation result storage unit 25 (step S17). ). The context similarity of terms A and B is determined by using the dependency relationship appearance frequency and the appearance frequency of term A stored in the similarity calculation result storage unit 25 in association with the terms A and B. It is calculated by “relationship appearance frequency / appearance frequency of term A”. When the context similarity calculated by the context similarity calculation unit 341 is equal to or less than a threshold (for example, 0.25), the context similarity is assumed to be 0.

  The context similarity of terms A and B calculated by the context similarity calculation unit 341 is stored in the similarity calculation result storage unit 25 in association with the terms A and B.

  The process of step S17 is executed for each set of terms A and B stored in the similarity calculation result storage unit 25. Thereby, when the process of step S <b> 17 is executed, the context similarity for each set of the terms A and B is stored in the similarity calculation result storage unit 25.

  Next, in the processes of steps S18 and S19, the character string similarity of terms A and B stored in the similarity calculation result storage unit 25 is calculated.

  The character string similarity calculation unit 342 included in the similarity calculation unit 34 includes the number of characters common to the terms A and B stored in the similarity calculation result storage unit 25 (hereinafter referred to as the common character number of the terms A and B). ) Is counted (step S18).

  Next, the character string similarity calculation unit 342 calculates the character string similarity of the terms A and B based on the counted number of common characters of the terms A and B (step S19). The string similarity of terms A and B is calculated by “number of common characters of terms A and B / number of characters of term A”. The character string similarity calculated by the character string similarity calculation unit 342 is stored in the similarity calculation result storage unit 25 in association with the terms A and B.

  Note that the processing in steps S18 and S19 described above is executed for each set of terms A and B stored in the similarity calculation result storage unit 25. Thereby, when the processing of steps S18 and S19 is executed, the character string similarity for each set of terms A and B is stored in the similarity calculation result storage unit 25.

  Hereinafter, the above-described similarity calculation processing will be described in detail with reference to FIGS. Here, it is assumed that the analysis result storage unit 23 stores a plurality of pieces of dependency relationship information shown in FIGS. 5 and 6 described above. Further, it is assumed that the term count result storage unit 24 stores a plurality of appearance frequency information shown in FIG.

  First, the similarity calculation unit 34 reads all the dependency relationship information from the analysis result storage unit 23. Next, the similarity calculation unit 34 counts the number of differences of the term 1 associated with the set for each term 2 and relationship included in each of the read dependency relationship information. That is, the similarity calculation unit 34 counts the number of different terms 1 having the same dependency relationship as the same term 2 based on the read dependency relationship information.

  5 and FIG. 6 described above, for example, the term 1 associated with the set of the term 2 “bringing” and the relationship “wo” is “pencil”, “eraser” and It is a “clock”. For this reason, in the dependency relationship information shown in FIG. 5 and FIG. 6, the dependency of the term 2 “bring” and the relationship “w” is associated with the term 1 (that is, the term 2 “bring” and “w”). The number of differences in the related terms 1) is 3. Here, the different numbers of the term 1 associated with the term 2 “bringing” and the relationship “ha” are specifically described, but the term 1 associated with the other term 2 and the relationship set. The number of differences is counted in the same way.

  Next, the similarity calculation unit 34 counts the appearance frequency of each term 1 associated with the term 2 and the set of relationships when the counted difference number is 2 or more. For example, since the difference number of the term 1 associated with the pair of the term 2 “bringing” and the relationship “ha” is 2 or more, the similarity calculation unit 34 uses the term 2 “bringing” and the relationship “ha”. The frequency of appearance of each of the terms 1 “pencil”, “eraser” and “clock” associated with the set of is counted.

  Here, the appearance frequency of the term 1 “pencil” associated with the set of the term 2 “bringing” and the relationship “ha” as described above will be specifically described with reference to FIGS. 5 and 6. In this case, the appearance frequency (number) of the dependency relationship information including the set of the term 1 “pencil”, the term 2 “bringing” and the relationship “O” is counted in the analysis result storage unit 23. According to FIGS. 5 and 6, the appearance frequency of the term 1 “pencil” associated with the set of the term 2 “bringing” and the relationship “ha” is 1.

  Although detailed explanation is omitted, according to the dependency relationship information shown in FIGS. 5 and 6, the terms 1 “eraser” and “clock” associated with the set of terms 2 “bringing” and relationship “ha” Similarly, the appearance frequency of “is also 1.

  Here, the appearance frequency of each of the terms 1 “pencil”, “eraser”, and “clock” associated with the set of the term 2 “bringing” and the relationship “wo” has been mainly explained. In the case described above, the appearance frequency is counted for all of the term 1 and the term 1 associated with the set of relations.

  Next, the similarity calculation unit 34 generates intermediate processing result information. In this case, the similarity calculation unit 34 generates intermediate processing result information 101 to 110 illustrated in FIG.

  As shown in FIG. 10, the intermediate processing result information 101 to 110 includes an intermediate processing result ID, a term 1, a term 2, and a set of relationships included in the dependency relationship information whose appearance frequency is counted as described above, and The counted appearance frequency (dependency relationship appearance frequency) is included.

  The intermediate process result ID included in the intermediate process result information 101 to 110 is an identifier for identifying the intermediate process result information, and is a numerical value ( An identifier) and a numerical value (identifier) assigned to the associated term 1.

  For example, the intermediate processing result information 101 shown in FIG. 10 includes an intermediate processing result ID “1-1”, term 1 “pencil”, term 2 “eraser”, relationship “O”, and appearance frequency “1”. . According to the intermediate processing result information 101 identified by the intermediate processing result ID “1-1”, the dependency relationship information including the term 1 “pencil”, the term 2 “bringing”, and the relationship “O” (set) is included. It is shown that the appearance frequency (the dependency relationship appearance frequency) is 1. In the intermediate processing result ID “1-1”, “1” on the left side is a numerical value assigned to the set of the term 2 “bringing” and the relationship “O”, and “1” on the right side is It is a numerical value given to the term 1 “pencil”.

  Similarly, the intermediate processing result 102 includes the intermediate processing result ID “1-2”, the term 1 “eraser”, the term 2 “bringing”, the relationship “O”, and the appearance frequency “1”. According to the intermediate processing result information 102 identified by the intermediate processing result ID “1-2”, the dependency relationship information including the term 1 “eraser”, the term 2 “bringing” and the relationship “O” (a set) is included. It is shown that the appearance frequency is 1. In the intermediate process result ID “1-2”, “1” on the left side is a numerical value assigned to the set of the term 2 “bringing” and the relationship “to” in the same way as the intermediate process result information 101 described above. “2” on the right side is a numerical value assigned to the term 1 “eraser”.

  Further, the intermediate processing result 103 includes an intermediate processing result ID “1-3”, term 1 “clock”, term 2 “bringing”, relationship “O”, and appearance frequency “1”. According to the intermediate processing result information 103 identified by the intermediate processing result ID “1-3”, the dependency relationship information including the term 1 “clock”, the term 2 “bringing”, and the relationship “O” (set) is included. It is shown that the appearance frequency is 1. Note that “1” on the left side of the intermediate processing result ID “1-3” is assigned to the set of the term 2 “bringing” and the relationship “to” in the same manner as the intermediate processing result information 101 and 102 described above. “3” on the right side is a numerical value assigned to the term 1 “clock”.

  Although detailed description of the intermediate processing result information 104 to 110 is omitted, for example, in the intermediate processing result information 104 and 105, since the set of the term 2 “entry” and the relationship “O” is common, the intermediate processing result information The numerical value on the left side in the intermediate processing result ID for identifying the result information 104 and 105 is common. Similarly, in the intermediate process result information 106 to 108, since the set of the term 2 “push” and the relationship “O” is common, the left side in the intermediate process result ID for identifying the intermediate process result information 106 to 108 The numbers are common. Also, the intermediate process result information 109 and 110 have the same term 2 “delete” and the relation “de”, so the left side in the intermediate process result ID for identifying the intermediate process result information 109 and 110 The numbers are common.

  Next, the similarity calculation unit 34 includes all combinations of the two terms 1 among the terms 1 associated with the intermediate processing result ID having the same left numerical value in the intermediate processing result information 101 to 110 ( Extract permutation).

  In the intermediate processing result information 101 to 110, for example, corresponding to the intermediate processing result ID (here, the intermediate processing result IDs “1-1”, “1-2”, and “1-3”) whose numerical value on the left side is “1”. The term 1 attached is “pencil”, “eraser” and “clock”. In this case, the similarity calculation unit 34 includes “pencil” and “eraser”, “pencil” and “clock”, “eraser” and “pencil”, “eraser” and “clock”, “clock” and “pencil”, Six combinations (permutations) of “clock” and “eraser” are extracted. In the intermediate processing result information 101 to 110, for example, the same applies to the term 1 associated with the intermediate processing result ID whose left numerical value is “2” and the intermediate processing result ID whose left numerical value is “3”. is there.

  Thus, each combination of the terms 1 extracted by the similarity calculation unit 34 is stored in the similarity calculation result storage unit 25 as the terms A and B.

  Here, FIG. 11 shows an example of the data structure of the similarity calculation result storage unit 25 after the combination of the term A and the term B is stored.

  As illustrated in FIG. 11, the similarity calculation result storage unit 25 stores a set ID, an original ID, an original ID appearance frequency, a term A, and a term B in association with each other.

  The set ID is an identifier for identifying a set of terms A and B. The original ID is an intermediate processing result ID included in the intermediate processing result information shown in FIG. The original ID appearance frequency is an appearance frequency (dependency relation appearance frequency) included in the intermediate processing result information shown in FIG.

  In the example illustrated in FIG. 11, the similarity calculation result storage unit 25 includes, for example, the set ID “1”, the original ID “1-1”, the original ID appearance frequency “1”, the term A “pencil”, and the term B “eraser”. Are stored in association with each other.

  Here, the set of the term A “pencil” and the term B “eraser” has been described, but the same applies to other sets of the term A and the term B as shown in FIG.

  Next, the context similarity calculation unit 341 included in the similarity calculation unit 34 calculates the context similarity of the terms A and B stored in the similarity calculation result storage unit 25 illustrated in FIG.

  In this case, the context similarity calculation unit 341 calculates an appearance frequency (appearance frequency of the term A) associated with each term A in the appearance frequency information stored in the term aggregation result storage unit 24 illustrated in FIG. get. The appearance frequency of each term A acquired by the context similarity calculation unit 341 is stored in the similarity calculation result storage unit 25 in association with the term A (and term B), as shown in FIG.

  Next, the context similarity calculation unit 341 uses the original ID appearance frequency (dependency relationship appearance frequency) and the appearance frequency of the term A that are associated with the terms A and B in the similarity calculation result storage unit 25. The context similarity of terms A and B is calculated.

  In the similarity calculation result storage unit 25 shown in FIG. 12, for example, the original ID appearance frequency associated with the term A “pencil” and the term B “eraser” is 1, and the appearance frequency of the term A “pencil” is 1. It is. For this reason, the context similarity of the term A “pencil” and the term B “eraser” is calculated as 1/1 = 1.

  In the similarity calculation result storage unit 25 shown in FIG. 12, for example, the original ID appearance frequency associated with the term A “eraser” and the term B “pencil” is 1, and the appearance frequency of the term A “eraser” Is 4. For this reason, the context similarity of the term A “eraser” and the term B “pencil” is calculated as ¼ = 0.25. Here, if the above-described threshold is 0.25, the context similarity of the term A “eraser” and the term B “pencil” is 0 because it is equal to or less than the threshold.

  Note that context similarities are similarly calculated for other terms A and B stored in the similarity calculation result storage unit 25 shown in FIG.

  As described above, the context similarity of terms A and B calculated by the context similarity calculation unit 341 is stored in the similarity calculation result storage unit 25 in association with the terms A and B. FIG. 13 shows an example of the data structure of the similarity calculation result storage unit 25 after the context similarity of the terms A and B is stored.

  Next, the character string similarity calculation unit 342 included in the similarity calculation unit 34 calculates the character string similarity of terms A and B stored in the similarity calculation result storage unit 25.

  In this case, the character string similarity calculation unit 342 counts the number of common characters of the terms A and B for each set of terms A and B stored in the similarity calculation result storage unit 25. Specifically, referring to FIG. 13, for example, in the case of the term A “pencil” and the term B “eraser” stored in the similarity calculation result storage unit 25, the term A “pencil” and the term B “eraser” The number of common characters is zero. For example, in the case of the term A “eraser” and the term B “eraser” stored in the similarity calculation result storage unit 25, the number of common characters of the term A “eraser” and the term B “eraser” is two.

  The character string similarity calculation unit 342 calculates the character string similarity of the terms A and B based on the counted number of common characters of the terms A and B and the number of characters of the term A. For example, in the case of the term A “pencil” and the term B “eraser”, the common character number of the term A “pencil” and the term B “eraser” is 0, and the number of characters of the term A is 2. Therefore, the character string similarity of the term A “pencil” and the term B “eraser” is calculated as 0/2 = 0. Further, for example, in the case of the term A “eraser” and the term B “eraser”, the common character number of the term A “eraser” and the term B “eraser” is 2 and the number of characters of the term A is 3 as described above. . Therefore, the character string similarity of the term A “eraser” and the term B “eraser” is calculated as 2 / 3≈0.67.

  The character string similarity is similarly calculated for the other terms A and B stored in the similarity calculation result storage unit 25 shown in FIG.

  As described above, the context similarity of terms A and B calculated by the context similarity calculation unit 341 is stored in the similarity calculation result storage unit 25 in association with the terms A and B. FIG. 14 is an example of the data structure of the similarity calculation result storage unit 25 after the character string similarity of terms A and B is stored.

  Next, with reference to the flowchart of FIG. 15, the processing procedure of the above-described feature degree calculation processing (the processing of step S6 shown in FIG. 4 described above) will be described.

  First, the feature degree calculation unit 36 reads all the appearance frequency information stored in the term aggregation result storage unit 24 (step S21). The appearance frequency information read from the term aggregation result storage unit 24 is stored in the feature degree calculation result storage unit 26. Thus, the term count result storage unit 24 stores the appearance frequency information ID, the term, and the appearance frequency of the term in association with each other.

  Next, for each cluster generated by the cluster generation unit 35 in step S5 shown in FIG. 4 described above, the feature calculation unit 36 acquires a document (set) belonging to the cluster from the document database 22.

  The feature calculation unit 36 counts the appearance frequency of each term stored in the feature calculation result storage unit 26 in each cluster (step S22). Specifically, the feature calculation unit 36 counts the appearance frequency of terms in the acquired documents belonging to each cluster. Note that the process of step S <b> 22 is executed for all terms stored in the feature calculation result storage unit 26.

  When the process of step S22 is executed, the appearance frequency in each cluster of terms stored in the feature calculation result storage unit 26 counted by the feature calculation unit 36 is stored in the feature calculation result in association with the term. Stored in the unit 26.

  Next, the feature degree calculation unit 36 calculates the feature degree for each cluster of the term based on the appearance frequency of the term stored in the feature degree calculation result storage unit 26 and the appearance frequency of the term in each cluster ( Step S23). In addition, the characteristic degree with respect to each cluster of a term represents that the term is a characteristic term in the cluster as the value is large.

  The characteristic degree of a term with respect to a cluster is calculated by “(appearance frequency in the cluster of the term−1) / appearance frequency of the term”. Note that the process of step S23 is executed for all terms stored in the feature calculation result storage unit 26.

  When the process of step S23 described above is executed, the feature degrees for each cluster of terms calculated by the feature degree calculation unit 36 are stored in the feature degree calculation result storage unit 26 in association with the terms (step S24). ). When the process of step S24 is executed, the feature degree calculation process ends.

  Hereinafter, with reference to FIGS. 16 to 18, the above-described feature calculation processing will be specifically described. Here, it is assumed that the term count result storage unit 24 stores a plurality of pieces of appearance frequency information shown in FIG. Further, it is assumed that the clusters 1 to 3 are generated by the cluster generation unit 35 as described in FIG.

  First, the feature degree calculation unit 36 reads all appearance frequency information from the term aggregation result storage unit 24. The appearance frequency information read from the term aggregation result storage unit 24 is stored in the feature degree calculation result storage unit 26. Thereby, the feature degree calculation result storage unit 26 stores, for example, the same information as in FIG.

  Next, the feature calculation unit 36 acquires documents belonging to each of the clusters 1 to 3 generated by the cluster generation unit 35 from the document database 22. Here, as described above, a document including the classification code “A” belongs to the cluster 1, a document including the classification code “B” belongs to the cluster 2, and a document including the classification code “C” belongs to the cluster 3. Shall belong. Therefore, a document including the classification code “A” as a document belonging to the cluster 1 is acquired from the document database 22. A document including the classification code “B” as a document belonging to the cluster 2 is acquired from the document database 22. A document including the classification code “C” as a document belonging to the cluster 3 is acquired from the document database 22.

  For each term stored in the feature value calculation result storage unit 26, the feature value calculation unit 36 counts the appearance frequency (hereinafter referred to as cluster appearance frequency) in each of the clusters “1” to “3” of the term. . In this case, the feature calculation unit 36 obtains the appearance frequency of each term in the document belonging to the cluster 1 acquired from the document database 22, the appearance frequency of each term in the document belonging to the cluster 2, and each term in the document belonging to the cluster 3. Count appearance frequency.

  The cluster appearance frequency in each of the clusters 1 to 3 counted by the feature degree calculation unit 36 is stored in the feature degree calculation result storage unit 26 for each term. FIG. 16 shows an example of the data structure of the feature calculation result storage unit 26 after the cluster appearance frequency in each of the clusters 1 to 3 is stored for each term. In FIG. 16, for the sake of convenience, the cluster appearance frequency in cluster 1 is shown as cluster appearance frequency 1, the cluster appearance frequency in cluster 2 is shown as cluster appearance frequency 2, and the cluster appearance frequency in cluster 3 is shown as cluster appearance frequency 3. In the example illustrated in FIG. 16, the feature calculation result storage unit 26 has, for example, 1 as the cluster appearance frequency in cluster 1 of the term “eraser”, 3 as the cluster appearance frequency in cluster 2, and 0 as the cluster appearance frequency in cluster 3. Stored. Although detailed explanation is omitted here, the feature calculation result storage unit 26 stores the cluster appearance frequency in each of the clusters 1 to 3 similarly for other terms as shown in FIG.

  Next, for each term stored in the feature value calculation result storage unit 26, the feature value calculation unit 36 calculates the term based on the appearance frequency of the term and the cluster appearance frequency in each of the clusters 1 to 3 of the term. The feature degree for each of the clusters 1 to 3 is calculated. It should be noted that the degree of feature for a term cluster N (here, N = 1, 2, 3) is calculated by “(cluster appearance frequency in the cluster N of the term−1) / appearance frequency of the term”.

  Here, with reference to FIG. 16, the characteristic degree with respect to each of the clusters 1 to 3 of the term “eraser” stored in the characteristic degree calculation result storage unit 26 will be specifically described. As shown in FIG. 16, the appearance frequency of the term “eraser” stored in the feature calculation result storage unit 26 is 4. Further, the cluster appearance frequency of the term “eraser” stored in the feature degree calculation result storage unit 26 in the cluster 1 is 1. In this case, the characteristic degree of the term “eraser” with respect to cluster 1 is calculated as (1-1) / 4 = 0. As shown in FIG. 16, the cluster appearance frequency in the cluster 2 of the term “eraser” stored in the feature calculation result storage unit 26 is 3. In this case, the characteristic degree of the term “eraser” with respect to the cluster 2 is calculated as (3-1) /4=0.5. As shown in FIG. 16, the cluster appearance frequency 3 in the cluster 3 of the term “eraser” stored in the feature calculation result storage unit 26 is zero. In this case, the characteristic degree of the term “eraser” with respect to cluster 1 is calculated as 0.

  It should be noted that the feature values are similarly calculated for other terms stored in the feature value calculation result storage unit 25 shown in FIG.

  As described above, the feature degrees for each of the clusters 1 to 3 calculated by the feature degree calculation unit 36 are stored in the feature degree calculation result storage unit 26 in association with the term. FIG. 17 shows an example of the data structure of the feature value calculation result storage unit 26 after the feature values for each of the clusters 1 to 3 of each term are stored. In FIG. 17, for convenience, the feature degree for cluster 1 is shown as feature degree 1, the feature degree for cluster 2 is shown as feature degree 2, and the feature degree for cluster 3 is shown as feature degree 3. In addition, according to the characteristic degree with respect to each of the clusters 1 to 3 of each term stored in the characteristic degree calculation result storage unit 26 illustrated in FIG. 17, the terms “eraser”, “dedicated ink”, “Character” and “erasure” are characteristic terms for cluster “2”, and the terms “cancel”, “input” and “cancel” are characteristic terms for cluster “3”.

  Next, a processing procedure of the above-described synonym set extraction process (the process of step S7 shown in FIG. 4 described above) will be described with reference to the flowchart of FIG.

  First, the synonym set extraction unit 37 reads the similarity calculation result from the similarity calculation result storage unit 25 (step S31). In this case, the synonym set extraction unit 37 is associated with the set of terms A and B and the set of terms A and B stored in the similarity calculation result storage unit 25 as the similarity calculation result. The context similarity (context similarity of terms A and B) and the character string similarity (character string similarity of terms A and B) are read. The similarity calculation result (a set of terms A and B, context similarity, and character string similarity) read from the similarity calculation result storage unit 25 is stored in the synonym set storage unit 27.

  Next, the synonym set extraction unit 37 executes the following steps S32 and S33 for each of the clusters generated by the cluster generation unit 35. A cluster to be processed is referred to as a target cluster.

  The synonym set extraction unit 37 reads the feature value calculation result from the feature value calculation result storage unit 26 (step S32). In this case, the synonym set extraction unit 37 reads the feature degree for the target cluster of the term A and the feature degree for the target cluster of the term B stored in the synonym set storage unit 27 as the feature degree calculation result. The feature value calculation results read from the feature value calculation result storage unit 26 (feature values for the target cluster of the term A and feature values for the target cluster of the term B) are stored in the synonym set storage unit 27.

  For each set of terms A and B stored in the synonym set storage unit 27, the synonym set extraction unit 37 determines the context similarity of the terms A and B stored in the synonym set storage unit 27, the terms A And whether the term A and the term B are synonyms based on the character string similarity of the term B, the feature degree of the term A with respect to the target cluster, and the feature degree of the term B with respect to the target cluster. . In this case, for example, the product of four values of the context similarity of terms A and B, the string similarity of terms A and B, the feature of term A for the target cluster, and the feature of term B for the target cluster If it is greater than 0, the terms A and B are determined to be synonyms. In other words, all of the four values of the context similarity of terms A and B, the string similarity of terms A and B, the features of term A for the target cluster, and the features of term B for the target cluster are all If it is not 0, the terms A and B are determined to be synonyms.

  By executing such determination processing for all pairs of the term A and the term B stored in the synonym set storage unit 27, the synonym set extraction unit 37 extracts the term A and the term B as synonyms. (Step S33).

  Next, the synonym set extraction unit 37 determines whether or not the processing of steps S32 and S33 has been executed for all the clusters generated by the cluster generation unit 35 (step S34).

  When it is determined that the processing has not been executed for all the clusters (NO in step S34), the process returns to the above-described step S32 and the processing is repeated. In this case, the process is executed with the cluster for which the processes of steps S32 and S33 have not been executed as the target cluster.

  On the other hand, if it is determined that the processing has been executed for all the clusters (YES in step S34), the synonym set extraction processing is terminated.

  Hereinafter, the above-described synonym set extraction processing will be specifically described with reference to FIGS. Here, it is assumed that the similarity calculation result storage unit 25 stores various types of information illustrated in FIG. Further, it is assumed that the characteristic degree calculation result storage unit 26 stores various types of information shown in FIG. Further, it is assumed that the clusters 1 to 3 are generated by the cluster generation unit 35 as described in FIG.

  First, the synonym set extraction unit 37 sets a set ID, a set of terms A and B, a context similarity, and a character string similarity stored in association with the similarity calculation result storage unit 25 as a similarity calculation result. All are read from the similarity calculation result storage unit 25. The similarity calculation result (a set ID, a set of terms A and B, a context similarity, and a character string similarity) read from the similarity calculation result storage unit 25 is stored in the synonym set storage unit 27. FIG. 20 shows an example of the data structure of the synonym set storage unit 27 after the similarity calculation result is stored.

  Hereinafter, the processing is executed for each of the clusters 1 to 3 generated by the cluster generation unit 35. Here, the cluster 2 will be described in detail.

  In this case, the synonym set extraction unit 37 obtains the characteristic degree of the term A with respect to the cluster 2 and the characteristic degree of the term B with respect to the cluster 2 stored in the synonym set storage unit 27 as the characteristic degree calculation result. Read from.

  The feature value calculation results read from the feature value calculation result storage unit 26 (the feature values of the term A with respect to the cluster 2 and the feature values of the term B with respect to the cluster 2) are stored in the synonym set storage unit 27. Here, the characteristic degree of the term A with respect to the cluster 2 and the characteristic degree of the term B with respect to the cluster 2 stored in the synonym set storage unit 27 are simply the characteristic degree of the term A and the characteristic degree of the term B. FIG. 21 shows an example of the data structure of the synonym set storage unit 27 after the feature calculation result is stored.

  Next, the synonym set extraction unit 37 calculates the context similarity of the terms A and B stored in the synonym set storage unit 27 for each set of terms A and B stored in the synonym set storage unit 27. It is determined whether or not the terms A and B are synonyms (a set of) based on the character string similarity of the terms A and B, the features of the term A, and the features of the term B. When the product of the four values of the context similarity of terms A and B, the string similarity of terms A and B, the features of terms A and B and the features of term B as described above is greater than 0, The terms A and B are determined to be synonyms.

  For example, in the case of the term A “pencil” and the term B “eraser” stored in the synonym set storage unit 27 illustrated in FIG. 21, the context similarity of the term A “pencil” and the term B “eraser” is 1. The term A “pencil” and the term B “eraser” have a character string similarity of 0, the term A “pencil” has a feature degree of 0, and the term B “eraser” has a feature degree of 0.5. In this case, the context similarity of the term A “pencil” and the term B “eraser”, the character string similarity of the term A “pencil” and the term B “eraser”, the feature of the term A “pencil”, and the term Since the product of the four values of the characteristic values of B is 0, it is determined that the term A “pencil” and the term B “eraser” are not synonyms.

  On the other hand, in the case of the term A “eraser” and the term B “eraser” stored in the synonym set storage unit 27 shown in FIG. 21, the context similarity of the term A “eraser” and the term B “eraser” is 0. 75, the character string similarity of the term A “eraser” and the term B “eraser” is 0.67, the characteristic degree of the term A “eraser” is 0.75, and the term B “eraser” The characteristic degree is 0.5. In this case, the context similarity of terms A “eraser” and B “eraser”, the string similarity of terms A “eraser” and B “eraser”, the characteristics of term A “eraser”, and the terms Since the product of the four values of the characteristic values of B “eraser” is not 0, the terms A “eraser” and B “eraser” are determined to be synonyms.

  As described above, the synonym set extraction unit 37 executes the determination process for all of the sets of the term A and the term B stored in the synonym set storage unit 27 illustrated in FIG.

  Here, FIG. 22 shows a determination result by the synonym set extraction unit 37 described above. In the example illustrated in FIG. 22, only the term A “eraser” and the term B “eraser” are determined to be synonyms from the set of the term A and the term B stored in the synonym set storage unit 27. In this case, the synonym set extraction unit 37 extracts the term A “eraser” and the term B “eraser” as synonyms.

  Although the cluster 2 has been described here, the same processing is executed for the cluster 1 and the cluster 3 as well. For example, in the case of cluster 1, the synonyms are not extracted because the features of terms A and B are all 0 as shown in FIG. On the other hand, in the case of the cluster 3, as shown in FIG. 24, the terms A “cancel” and the term B “cancel” and the terms A “cancel” and the term B “cancel” are determined to be synonyms. The terms A “cancel” and B “cancel” (and the terms A “cancel” and B “cancel”) are extracted as synonyms.

  As described above, the two terms (term A and term B) extracted as synonyms by the synonym set extraction unit 37 are output (displayed) by the output processing unit 38. Here, FIG. 25 shows an example of a display screen when two terms extracted as synonyms by the synonym set extraction unit 37 are displayed. In the example shown in FIG. 25, the term “eraser” and the term “eraser” extracted as synonyms in the case of cluster 2 (that is, when processing is performed on cluster 2) and the cluster 3 as described above. The term “cancel” and the term “cancel” extracted as synonyms are displayed in this case (that is, when processing is performed on cluster 3). The user can instruct whether or not to register two terms displayed on the display screen as synonyms with reference to the display screen as shown in FIG.

  As described above, in the present embodiment, two terms (first and second terms) are extracted from terms included in a plurality of documents stored in the document database 22, and the extracted two The degree of similarity of terms is calculated, a cluster to which a plurality of documents stored in the document database 22 belong is generated, the characteristic degree of each of the two terms with respect to the generated cluster is calculated, and the two terms By extracting the two terms as synonyms based on the similarity and the characteristics of each of the two terms for the cluster, only the appropriate terms are extracted as synonyms from the document without extracting the wrong terms as synonyms. It becomes possible to do.

  Specifically, when a synonym is extracted based only on, for example, the context similarity and the character string similarity shown in FIG. 14 described above, the product of two values of the context similarity and the character string similarity is not 0 2 Two terms (here, the term “cancel” and the term “eraser”, the term “cancel” and the term “cancel”, the term “eraser” and the term “eraser”) are extracted as synonyms, respectively. However, the term “cancellation” and the term “eraser” which are not appropriate in this case are also extracted as synonyms. On the other hand, in the present embodiment, in addition to the context similarity and the character string similarity, the term “cancellation” and the term that are not appropriate are considered by taking into consideration the cluster (characteristic for) to which the document from which the term is extracted belongs. By excluding “eraser”, as described above, only the term “cancel” and term “cancel”, the term “eraser” and the term “eraser” can be extracted as synonyms.

  That is, according to the present embodiment, synonyms are extracted based on local similarity such as context similarity or character string similarity and global similarity such as cluster feature. ) Can be suppressed from being extracted due to local similarity in the document.

  Further, the synonyms extracted in the present embodiment can be registered in, for example, a synonym dictionary and used for document search or document classification.

  In this embodiment, the context similarity and the character string similarity are calculated as the similarity between two terms. However, only one of the context similarity and the character string similarity is calculated. There may be. When only the context similarity is calculated, for example, steps S11 to S17 shown in FIG. 9 may be executed. On the other hand, when only the character string similarity is calculated, for example, the processing of steps S18 and S19 shown in FIG. 9 for each combination of two terms among a plurality of terms stored in the term aggregation result storage unit 24. May be executed.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. The functional configuration of the document processing apparatus according to the present embodiment is the same as that of the first embodiment described above, and will be described with reference to FIG. 2 as appropriate.

  This embodiment is different from the first embodiment described above in that the similarity calculation process by the similarity calculation unit 34 is executed after the feature calculation process by the feature calculation unit 36 is executed. That is, in the similarity calculation process in the present embodiment, the feature degree calculation stored in the feature degree calculation result storage unit 26 instead of the appearance frequency information (term aggregation result) stored in the term tabulation result storage unit 24. The result is used.

  Here, a processing procedure of the document processing apparatus 30 according to the present embodiment will be described with reference to a flowchart of FIG.

  First, the processes of steps S41 to S45 corresponding to the processes of steps S1 to S3, S5, and S6 shown in FIG. 4 are executed.

  When the process of step S42 is executed, the dependency relationship information (analysis result) is stored in the analysis result storage unit 23 as shown in FIGS. 5 and 6 described above. When the process of step S43 is executed, the appearance frequency information (term aggregation result) is stored in the term aggregation result storage unit 24 as shown in FIG. Further, when the process of step S45 is executed, the characteristic degree calculation result as shown in FIG. 17 is stored in the characteristic degree calculation result storage unit 26.

  Next, the similarity calculation unit 34 performs a similarity calculation process with reference to the analysis result storage unit 23 and the feature calculation result storage unit 25 (step S46). According to the similarity calculation process, two terms out of terms satisfying a predetermined condition described later in terms of the features for each cluster generated by the cluster generation unit 35 stored in the feature calculation result storage unit 26 are described. (First and second terms) are extracted, and the similarity between the two terms is calculated. That is, in the similarity calculation process, the process is executed only for terms that satisfy a predetermined condition instead of all the terms stored in the feature calculation result storage unit 26. Details of the similarity calculation process will be described later.

  When the similarity calculation process is executed, the similarity calculated by the similarity calculation unit 34 is stored in the similarity calculation result storage unit 25.

  Next, the synonym set extraction unit 37 extracts, based on the similarity stored in the similarity calculation result storage unit 25, the two terms extracted in the above-described similarity calculation process as a synonym (set). (Synonym set extraction processing) is executed (step S47). Details of the synonym set extraction process will be described later.

  When the synonym set extraction process is executed, the process of step S48 corresponding to the process of step S8 shown in FIG. 4 described above is executed.

  Next, the procedure of the similarity calculation process described above (the process of step S46 shown in FIG. 26 described above) will be described with reference to the flowchart of FIG.

  In the similarity calculation process, the following steps S51 to S61 are executed for each of the clusters generated by the cluster generation unit 35. A cluster to be processed is referred to as a target cluster.

  First, the similarity calculation unit 34 reads a feature calculation result from the feature calculation result storage unit 26 (step S51). In this case, the similarity calculation unit 34 reads the terms stored in the feature calculation result storage unit 26 and the features for the target cluster of the terms as the feature calculation results.

  Next, the similarity calculation unit 34 extracts terms satisfying a predetermined condition from the terms read from the feature calculation result storage unit 26 (step S52). Here, the predetermined condition includes, for example, that the feature degree with respect to the target cluster is not zero. In this case, based on the terms read from the feature value calculation result storage unit 26 and the feature values of the terms with respect to the target cluster, the similarity calculation unit 34 extracts terms whose feature values for the target cluster are not zero.

  Similarity calculation part 34 reads all the dependency relation information stored in analysis result storage part 23 like Step S11 shown in Drawing 9 mentioned above (Step S53).

  Next, the similarity calculation unit 34 sets the term 2 and the relationship included in each of the dependency relationship information read from the analysis result storage unit 23 as one set, and uses the term 2 in the read dependency relationship information. And the number of different terms 1 associated with the set of relationships (that is, the term 1 in the same dependency relationship as the same term 2) is counted (step S54). In this step S54, only the terms extracted in the above-described step S52 (that is, terms whose feature degrees for the target cluster are not 0) are targeted.

  Thereafter, the processes of steps S55 to S61 corresponding to the processes of steps S13 to S19 shown in FIG. 9 are executed. Note that the context similarity calculated in step S59 and the character string similarity calculated in step S61 are stored in the similarity calculation result storage unit 25 as in the first embodiment.

  When the process of step S61 is executed, it is determined whether or not the processes of steps S51 to S61 described above have been executed for all clusters generated by the cluster generation unit 35 (step S62).

  When it is determined that the processing has not been executed for all the clusters (NO in step S62), the process returns to the above-described step S51 and is repeated. In this case, the processing is executed with the cluster for which the processing of steps S51 to S61 has not been executed as the target cluster.

  On the other hand, when it is determined that the process has been executed for all the clusters (NO in step S62), the similarity calculation process ends.

  When the similarity calculation processing is executed as described above, the similarity calculation result storage unit 25 stores the similarity calculation results (context similarity and character string similarity) for each cluster generated by the cluster generation unit 35. Stored.

  Hereinafter, with reference to FIG. 28 and FIG. 29, the above-described similarity calculation processing will be specifically described. Here, it is assumed that the analysis result storage unit 23 stores a plurality of pieces of dependency relationship information shown in FIGS. 5 and 6 described above. Further, it is assumed that the clusters 1 to 3 are generated by the cluster generation unit 35 as described in FIG. Further, it is assumed that the characteristic degree calculation result storage unit 26 stores various types of information shown in FIG.

  As described above, in the similarity calculation process, the process is executed for each of the clusters 1 to 3 generated by the cluster generation unit 35. Here, the cluster 2 will be specifically described.

  First, for each term stored in the feature calculation result storage unit 26, the similarity calculation unit 34 reads out the term and the feature level of the term with respect to the cluster 2 from the feature calculation result storage unit 26. In the example illustrated in FIG. 17, the similarity calculation unit 34 reads the term “pencil” and the characteristic degree “0” of the term “pencil” with respect to the cluster 2. The same applies to other terms.

  Next, the similarity calculation unit 34 extracts, for example, terms whose feature degrees for the cluster 2 are not 0 from the terms read from the feature degree calculation result storage unit 26. Specifically, using terms (namely, read terms) stored in the feature calculation result storage unit 26 shown in FIG. 17, the terms “eraser”, “dedicated ink”, “ The degree of feature for the cluster 2 of terms other than “character” and “eraser” is zero. Therefore, the similarity calculation unit 34 extracts the terms “eraser”, “dedicated ink”, “character”, and “eraser” as terms whose feature degrees for the cluster 2 are not zero.

  The similarity calculation unit 34 reads all the dependency relationship information stored in the analysis result storage unit 23.

  The similarity calculation unit 34 counts the number of different terms associated with each set of terms 2 and relationships included in each of the read dependency relationship information. That is, the similarity calculation unit 34 counts the number of different terms 1 having the same dependency relationship as the same term 2 based on the read dependency relationship information.

  At this time, the similarity calculation unit 34 targets only the terms “eraser”, “dedicated ink”, “character”, and “eraser” extracted as described above, that is, the same term 2 Count the number of different terms 1 that have the same dependency relationship.

  5 and FIG. 6, for example, the term 1 associated with the pair of the term 2 “erasing” and the relationship “de” is “erasing” and “erasing”. . For this reason, in the dependency relationship information stored in the analysis result storage unit 23, the term 1 associated with the pair of the term 2 “erase” and the relationship “de” (that is, the term 2 “erase” and “de”). The number of different terms 1) in the dependency relationship is 2.

  In the example shown in FIGS. 5 and 6, the term 1 associated with the set of term 2 “bringing” and the relationship “ha” is “pencil”, “eraser”, and “clock”. Since only the terms “eraser”, “dedicated ink”, “character” and “eraser” are targeted, the difference between term 2 associated with term 2 “bring” and relationship “a” The number is 1.

  Next, the similarity calculation unit 34 counts the appearance frequency of each term 1 associated with the term 2 and the set of relationships when the counted difference number is 2 or more.

  For example, since the difference number of the term 1 associated with the pair of the term 2 “erase” and the relationship “de” is two or more, the similarity calculation unit 34 determines the term 2 “erase” and the relationship “de”. The frequency of appearance of each of the terms 1 “eraser” and “eraser” associated with the set of is counted. According to the example shown in FIG. 5 and FIG. 6, the appearance frequency of the term 1 “erasure” associated with the set of term 2 “erasure” and the relationship “de” is 2, and the term 1 “eraser” The appearance frequency is 1. In the example shown in FIGS. 5 and 6, there is no term 1 having a number of two or more different from the term 1 associated with the term 2 “erasing” and the relationship “de”.

  Next, the similarity calculation unit 34 generates intermediate processing result information. In this case, the similarity calculation unit 34 generates intermediate processing result information shown in FIG. Since the generation processing of the intermediate processing result information is as described in the first embodiment, detailed description thereof is omitted.

  Next, the similarity calculation unit 34 extracts all combinations (permutations) consisting of the two terms 1 based on the intermediate processing result information shown in FIG. In this case, the similarity calculation unit 34 extracts two combinations of “eraser” and “eraser”, “eraser” and “eraser”.

  Thus, each combination of the term 1 extracted by the similarity calculation unit 34 is stored in the similarity calculation result storage unit 25 as a term A and a term B (a set).

  Hereinafter, although detailed description is omitted, as in the first embodiment described above, the context similarity and the character string similarity are determined for each set of terms A and B stored in the similarity calculation result storage unit 25. Calculated. The context similarity and the character string similarity calculated in this way are stored in the similarity calculation result storage unit 25.

  In addition, although the case where the process was performed about the cluster 2 was demonstrated here, since it is the same also about the clusters 1 and 3, the detailed description is abbreviate | omitted.

  FIG. 29 shows an example of the data structure of the similarity calculation result storage unit 25 after the processing is executed for each of the clusters 1 to 3.

  For example, the group IDs “1” and “2” (information related to) shown in FIG. 29 are information stored in the similarity calculation result storage unit 25 when the process is executed on the cluster 2 as described above (the information on the cluster 2). Although the detailed description is omitted, the set IDs “3” and “4” (information regarding the similarity calculation result) are stored in the similarity calculation result storage unit 25 when the process is performed on the cluster 3. Information (cluster 3 similarity calculation result).

  Next, a processing procedure of the above-described synonym set extraction processing (the processing of step S47 shown in FIG. 26 described above) will be described with reference to the flowchart of FIG.

  First, the synonym set extraction unit 37 reads the similarity calculation result from the similarity calculation result storage unit 25 (step S71). In this case, the synonym set extraction unit 37 is associated with the set of terms A and B and the set of terms A and B stored in the similarity calculation result storage unit 25 as the similarity calculation result. The context similarity (context similarity of terms A and B) and the character string similarity (character string similarity of terms A and B) are read. The similarity calculation result read from the similarity calculation result storage unit 25 is stored in the synonym set storage unit 27.

  Next, the synonym set extraction unit 37 calculates the context similarity between the terms A and B stored in the synonym set storage unit 27 for each set of terms A and B stored in the synonym set storage unit 27, and Based on the string similarity of terms A and B, it is determined whether or not terms A and B are synonyms (a set of). In this case, for example, when the product of two values of the context similarity of terms A and B and the string similarity of terms A and B is greater than 0 (that is, both the context similarity and the string similarity) The term A and the term B are determined to be synonyms.

  By executing such determination processing for all pairs of the term A and the term B stored in the synonym set storage unit 27, the synonym set extraction unit 37 extracts the term A and the term B as synonyms. (Step S72).

  For example, assuming that the various kinds of information shown in FIG. 29 described above are stored in the similarity calculation result storage unit 25, the term A “eraser” and the term B “eraser” are synonyms according to the synonym set extraction process described above. Extracted as In addition, the term A “cancel” and the term B “cancel” (and the term A “cancel” and the term B “cancel”) are extracted as synonyms.

  Therefore, also in the present embodiment, synonyms similar to those in the first embodiment described above are extracted.

  As described above, in the present embodiment, a cluster to which a plurality of documents stored in the document database 22 belongs is generated, and a characteristic degree for the cluster is obtained for each term included in the plurality of documents stored in the document database. Calculate, extract two terms (first and second terms) from terms that satisfy the predetermined condition, calculate the similarity between the two extracted terms, and calculate With the configuration in which the two terms are extracted as synonyms based on the similarity, the appropriate terms are extracted from the document as synonyms without extracting the erroneous terms as synonyms as in the first embodiment. In addition, since it becomes possible to narrow down the terms that are subject to similarity calculation processing, the processing amount (calculation amount) in the similarity calculation processing is reduced. Theft is possible.

  According to at least one embodiment described above, it is possible to provide a document processing apparatus and program capable of extracting an appropriate term as a synonym from a document.

  Note that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in each embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 10 ... Computer, 20 ... External storage device, 22 ... Document database, 23 ... Analysis result storage part, 24 ... Term totaling result storage part, 25 ... Similarity calculation result storage part, 26 ... Feature degree calculation result storage part, 27 ... Synonym set storage unit, 30 ... document processing device, 31 ... input processing unit, 32 ... analysis unit, 33 ... term aggregation unit, 34 ... similarity calculation unit, 35 ... cluster generation unit, 36 ... feature calculation unit, 37 ... Synonym set extraction unit, 38 ... output processing unit, 341 ... context similarity calculation unit, 342 ... character string similarity calculation unit.

Claims (5)

Document storage means for storing a plurality of documents including a term consisting of one or more words;
Term extracting means for extracting first and second terms from terms contained in a plurality of documents stored in the document storing means;
Similarity calculating means for calculating the similarity between the extracted first and second terms;
Cluster generation means for generating a cluster to which each of the plurality of documents stored in the document storage means belongs;
Based on the appearance frequency of the extracted first term in the plurality of documents stored in the document storage means and the document belonging to the generated cluster, the feature degree of the first term for the cluster is calculated, A feature of calculating a feature level of the second term for the cluster based on the appearance frequency of the extracted second term in the plurality of documents stored in the document storage unit and the document belonging to the generated cluster. Degree calculation means;
Based on the similarity calculated by the similarity calculation unit, the feature level of the first term and the feature level of the second term calculated by the feature level calculation unit, the first and second terms are synonymous. And a synonym extracting means for extracting as: a document processing apparatus. The term extraction means extracts first and second terms that are in the same dependency relationship with the same term among the terms contained in the plurality of documents stored in the document storage means,
The similarity calculation means includes a frequency of appearance of a first term having the same dependency relationship as the same term in a plurality of documents stored in the document storage means, and a plurality of similarity degrees stored in the document storage means. The document processing apparatus according to claim 1, wherein the similarity between the extracted first and second terms is calculated based on the appearance frequency of the first term in the document.   2. The document according to claim 1, wherein the similarity calculation means calculates the similarity of the first and second terms based on the number of characters common to the extracted first and second terms. Processing equipment. Document storage means for storing a plurality of documents including a term consisting of one or more words;
Cluster generation means for generating a cluster to which a plurality of documents stored in the document storage means belong;
For each term contained in a plurality of documents stored in the document storage means, based on the appearance frequency of the terms in the plurality of documents stored in the document storage means and the documents belonging to the generated cluster, A feature calculation means for calculating the feature of the term for the cluster;
Term extraction means for extracting the first and second terms from terms satisfying a predetermined degree of feature calculated by the feature calculation means;
Similarity calculating means for calculating the similarity between the extracted first and second terms;
And a synonym extracting unit that extracts the extracted first and second terms as synonyms based on the similarity calculated by the similarity calculating unit. In a document processing apparatus comprising an external storage device having a document storage means for storing a plurality of documents including a term composed of one or a plurality of words and a computer using the external storage device, the document processing device is executed by the computer. A program
In the computer,
Extracting first and second terms from terms contained in a plurality of documents stored in the document storage means;
Calculating a similarity between the extracted first and second terms;
Generating a cluster to which each of a plurality of documents stored in the document storage means belongs;
Based on the appearance frequency of the extracted first term in the plurality of documents stored in the document storage means and the document belonging to the generated cluster, the feature degree of the first term for the cluster is calculated, Calculating a characteristic value of the second term for the cluster based on the appearance frequency of the extracted second term in the plurality of documents stored in the document storage means and the document belonging to the generated cluster When,
Extracting the first and second terms as synonyms based on the calculated similarity, the calculated feature of the first term, and the feature of the second term. program.

Images