JP4945599B2 - Feature word extraction system and program - Google Patents

Description

  The present invention relates to text mining, and relates to a technique for detecting a concept change for a word generated between concept dictionaries derived from another document data group.

  In recent years, many apparatuses and methods relating to text mining have been proposed. These conventional techniques are mainly intended for document retrieval.

  Therefore, in text mining, data representing a concept of a word called a concept dictionary in a vector format is generated based on a document data group. The concept dictionary is used to conceptually search for documents.

  However, this concept dictionary can be used in addition to searching for documents.

JP 2001-184351 A JP 2003-141134 A Yasuhiro Takayama et al., 3 others, “Information retrieval system InfoMAP based on word association”, Informatics Foundation No. 053-001, March 1999 Yasuyuki Aikawa, 3 others, automatic updating of concept dictionary in large-scale search system, FIT 2002 Information Science and Technology Forum General Lecture Collection, Second Volume, September 13, 2002, p. 73, 74

  The present invention utilizes the prior art, and the object is to generate differences between document data groups by extracting differences between concept dictionaries based on a plurality of concept dictionaries derived from different document data groups. The purpose is to detect differences and changes in the concept of words.

The feature word extraction system according to the present invention includes:
(1) The frequency at which the combined words appear together in the predetermined range group included in the first document data group is calculated as the co-occurrence frequency related to the combination of the words. And generating a first co-occurrence frequency table for storing the co-occurrence frequencies related to the combination of the plurality of words in a matrix format, and further combining the words in the predetermined range group included in the second document data group Is calculated as a co-occurrence frequency related to the combination of the words, and a co-occurrence frequency for generating a second co-occurrence frequency table storing the respective co-occurrence frequencies related to the combination of the words in a matrix format The calculation unit (2) performs singular value decomposition on the first co-occurrence frequency table, calculates a first concept dictionary composed of concept vectors for each word in the degenerated dimension semantic space, and further generates a second co-occurrence frequency. Special table A singular value decomposition unit that performs value decomposition and calculates a second concept dictionary composed of concept vectors for each word in the reduced dimension semantic space as a contraction matrix (3) based on the concept vector for each word in the first concept dictionary Calculating the difference in concept of the comparison target word with respect to the reference word, and further calculating the difference in concept of the comparison target word with respect to the reference word based on the concept vector for each word in the second concept dictionary, Based on the difference between concepts and the difference between the concepts in the second concept dictionary, it is determined whether or not the comparison target word corresponds to a difference between concept dictionaries, and the word corresponding to the difference is used as concept dictionary difference information Dictionary difference extraction unit.

The concept dictionary difference extraction unit has the following elements: (3-1) Based on a concept vector for each word in the first concept dictionary, calculates a difference in concept of the comparison target word with respect to the reference word; When a difference in concept in the first concept dictionary is smaller than a predetermined reference, a first close concept word list for storing the comparison target word as a close concept word with respect to the reference word is generated, and further in the second concept dictionary Based on the concept vector for each word, the difference of the concept of the comparison target word with respect to the reference word is calculated, and when the difference of the concept in the second concept dictionary is smaller than a predetermined reference, the comparison target word is determined as the reference word A near concept word extraction unit for generating a second near concept word list stored as a near concept word for (3-2) a near concept word and a second near concept word list in the first near concept word list Near the concept word comparison section that compares the near concept word of the theft, the word of the difference between the concept dictionary difference information.

The concept dictionary difference extraction unit has the following elements: (3-1) Based on a concept vector for each word in the first concept dictionary, calculates a difference in concept of the comparison target word with respect to the reference word; A first concept difference table for storing the calculated concept difference is generated, and further, based on the concept vector for each word in the second concept dictionary, the difference in concept of the comparison target word with respect to the reference word is calculated, and the calculated concept A concept difference calculation unit (3-2) that generates a second concept difference table that stores the difference. For the concept difference of the comparison target word with respect to the reference word, the concept difference in the first concept difference table is smaller than a predetermined reference, And when the difference in the concept in the second concept difference table is not smaller than the predetermined reference, the comparison target word and the concept in the first concept difference table Min, not less than a predetermined reference, and the difference between the concepts in the second concept difference table, the comparison target word when less than a predetermined reference, the concept difference comparison unit that the concept dictionary difference information.

The concept dictionary difference extraction unit has the following elements: (3-1) Based on a concept vector for each word in the first concept dictionary, calculates a difference in concept of the comparison target word with respect to the reference word; A first concept difference table for storing the calculated concept difference is generated, and further, based on the concept vector for each word in the second concept dictionary, the difference in concept of the comparison target word with respect to the reference word is calculated, and the calculated concept A concept difference calculation unit (3-2) that generates a second concept difference table that stores the difference. For the concept difference of the comparison target word with respect to the reference word, the concept difference in the first concept difference table is smaller than a predetermined reference, In addition, the amount of change in the concept difference obtained by subtracting the concept difference in the first concept difference table from the concept difference in the second concept difference table exceeds a predetermined standard. The difference between the comparison target word and the concept in the second concept difference table is smaller than a predetermined reference, and the concept difference obtained by subtracting the concept difference in the second concept difference table from the concept difference in the first concept difference table. A concept difference comparison unit that uses the comparison target word when the amount of change exceeds a predetermined criterion as concept dictionary difference information.

The near concept word comparison unit
A word that is stored as a near concept word for the reference word in the first near concept word list and that is not stored as a near concept word for the reference word in the second near concept word list is determined as a decrease word, and the decrease word If there is no other near concept word stored in the second near concept word list with respect to the reference word, the decrease type is “annihilation”, and there is the other near concept word. A reduced word comparison processing unit to be distinguished;
A word that is stored as a near concept word for the reference word in the second near concept word list and is not stored as a near concept word for the reference word in the second near concept word list is determined as an increase word, and the increase word When there is no other near concept word stored in the first near concept word list with respect to the reference word, the increase type is “occurrence”, and there is the other near concept word. And an increased word comparison processing unit for distinguishing.

The concept difference comparison unit
Regarding the difference in concept of the comparison target word with respect to the reference word, the difference in concept in the first concept difference table is smaller than a predetermined reference, and the difference in concept in the second concept difference table is not smaller than the predetermined reference. When the comparison target word is determined to be a decrease word and is determined to be the decrease word, the decrease type when there is no comparison target word whose concept difference is smaller than a predetermined reference with respect to the reference word in the second concept difference table is “annihilation” ”As a reduced word comparison processing unit for distinguishing from the case where there is a comparison target word whose difference in concept is smaller than a predetermined reference,
Regarding the difference in concept of the comparison target word with respect to the reference word, the difference in concept in the second concept difference table is smaller than a predetermined reference, and the difference in concept in the first concept difference table is not smaller than the predetermined reference. If the comparison target word is determined to be an increase word, and the increase word is determined to be an increase word when there is no comparison target word whose concept difference is smaller than a predetermined reference in the first concept difference table, ”As an increase word comparison processing unit for distinguishing from the case where there is a comparison target word whose concept difference is smaller than a predetermined reference.

The feature word extraction system according to the present invention includes:
(1) calculating the frequency that the combined words appear together in the predetermined range group included in the document data group as the co-occurrence frequency related to the combination of the words, A co-occurrence frequency calculation unit that generates a co-occurrence frequency table that stores each co-occurrence frequency related to a combination of a plurality of words in a matrix format. (2) The co-occurrence frequency table is singularly decomposed and reduced as a degenerate matrix. A singular value decomposition unit that calculates a concept dictionary consisting of concept vectors for each word in a three-dimensional meaning space; (3) calculates a difference in concept of the comparison target word with respect to the reference word based on the concept vector for each word in the concept dictionary; The number of comparison target words whose difference in concept of the comparison target word with respect to the reference word is smaller than a predetermined reference is counted to obtain the number of near concept words with respect to the reference word, and the near concept for each reference word Near concept word number calculating unit for generating a near-concept word number table of the number of words.

The feature word extraction system further includes the following elements: (4) A near concept word number sorting unit that sorts the near concept word number table using the number of near concept words as a key.

The feature word extraction system according to the present invention includes:
(1) The frequency at which the combined words appear together in the predetermined range group included in the first document data group is calculated as the co-occurrence frequency related to the combination of the words. And generating a first co-occurrence frequency table for storing the co-occurrence frequencies related to the combination of the plurality of words in a matrix format, and further combining the words in the predetermined range group included in the second document data group Is calculated as a co-occurrence frequency related to the combination of the words, and a co-occurrence frequency for generating a second co-occurrence frequency table storing the respective co-occurrence frequencies related to the combination of the words in a matrix format The calculation unit (2) performs singular value decomposition on the first co-occurrence frequency table, calculates a first concept dictionary composed of concept vectors for each word in the degenerated dimension semantic space, and further generates a second co-occurrence frequency. Special table A singular value decomposition unit that performs value decomposition and calculates a second concept dictionary composed of concept vectors for each word in the reduced dimension semantic space as a contraction matrix (3) based on the concept vector for each word in the first concept dictionary The difference in concept of the comparison target word with respect to the reference word is calculated, and the number of comparison target words whose difference in concept of the comparison target word with respect to the reference word is smaller than a predetermined reference is counted. A first concept word count table composed of the number of concept words for each reference word, and based on the concept vector for each word in the second concept dictionary, the difference in concept of the comparison target word with respect to the reference word is calculated. Calculate the number of comparison target words for which the difference in concept of the comparison target word with respect to the reference word is smaller than a predetermined reference, and obtain the number of close concept words with respect to the reference word A near concept word number calculation unit for generating a second concept word number table composed of the number of concept words for each reference word (4) For each reference word, the number of near concept words in the first concept word number table and the second A near-concept word number comparison unit that compares the number of near-concept words in the near-concept word number table, obtains a difference in the number of near-concept words, and generates a near-concept word number difference table that includes differences in the number of near-concept words for each reference word.

The feature word extraction system further includes the following elements: (5) A near concept word number difference sorting unit that sorts the near concept word number difference table using a difference in the number of near concept words as a key.

The near concept word comparison unit
Decreasing words that are stored as near concept words with respect to the reference word in the first near concept word list and that are not stored as near concept words with respect to the reference word in the second near concept word list are reduced words A comparison processing unit;
An increase word for determining an increase word for a word that is stored as a close concept word for the reference word in the second close concept word list and that is not stored as a close concept word for the reference word in the second close concept word list A comparison processing unit,
For each reference word, the feature word extraction system calculates the number of near concept words by multiplying the number of increased words, counts the number of decreased words, and calculates the number of near concept words. It has a near concept word number comparison unit that subtracts the number of concept word decreases and calculates a difference in the number of near concept words as a difference.

The concept difference comparison unit
Regarding the difference in concept of the comparison target word with respect to the reference word, the difference in concept in the first concept difference table is smaller than a predetermined reference, and the difference in concept in the second concept difference table is not smaller than the predetermined reference. A reduced word comparison processing unit that determines a comparison target word as a reduced word;
Regarding the difference in concept of the comparison target word with respect to the reference word, the difference in concept in the second concept difference table is smaller than a predetermined reference, and the difference in concept in the first concept difference table is not smaller than the predetermined reference. An increased word comparison processing unit that determines a comparison target word as an increased word,
For each reference word, the feature word extraction system calculates the number of near concept words by multiplying the number of increased words, counts the number of decreased words, and calculates the number of near concept words. It has a near concept word number comparison unit that subtracts the number of concept word decreases and calculates a difference in the number of near concept words as a difference.

The concept difference comparison unit
Regarding the difference in concept of the comparison target word with respect to the reference word, the difference in concept in the first concept difference table is smaller than a predetermined reference, and the difference in concept in the first concept difference table is calculated from the difference in concept in the second concept difference table. A reduced word part comparison processing unit that determines that the comparison target word is a reduced word when the subtracted conceptual difference change amount exceeds a predetermined reference;
Regarding the difference in concept of the comparison target word with respect to the reference word, the difference in concept in the second concept difference table is smaller than a predetermined reference, and the difference in concept in the second concept difference table is calculated from the difference in concept in the first concept difference table. An increased word comparison processing unit that determines that the comparison target word is an increased word when the amount of change in the subtracted concept difference exceeds a predetermined reference;
For each reference word, the feature word extraction system calculates the number of near concept words by multiplying the number of increased words, counts the number of decreased words, and calculates the number of near concept words. It has a near concept word number comparison unit that subtracts the number of concept word decreases and calculates a difference in the number of near concept words as a difference.

  The near concept word number comparison unit further sets the increase type when the near concept word number difference is positive and the number of near concept words to the reference word in the first near concept word number table is 0 as “occurrence”, Differentiating from the case where the number of near concept words is greater than 0, the decrease type when the near concept word number difference is negative and the number of near concept words for the reference word is 0 in the second near concept word number table is “ Disappear ”is distinguished from the case where the number of near-concept words is larger than zero.

  The near concept word number difference calculation unit further sets the increase type as “occurrence” when the near concept word number difference is positive and there is no near concept word for the reference word in the first near concept word list, Distinguishing from the case where there is a concept word, the near concept word number difference is negative, and the decrease type when there is no near concept word for the reference word in the second near concept word list is set to “annihilation”, the near concept word It is distinguished from the case where there is.

  The near concept word number difference calculating unit further increases the type when the near concept word number difference is positive and there is no comparison target word whose concept difference is smaller than a predetermined reference in the first concept difference table. Is “occurrence”, and the difference between the concepts is compared with the case where there is a comparison target word that is smaller than a predetermined reference, the difference in the number of near concept words is negative, and the difference in concept for the reference word in the second concept difference table The type of decrease when there is no comparison target word smaller than a predetermined reference is “disappeared”, and is distinguished from the case where there is a comparison target word whose difference in concept is smaller than a predetermined reference.

The program according to the present invention is:
A program for causing a computer to be a feature word extraction system to execute the following procedure. (1) Combined words appear together in a predetermined range group included in the first document data group. To generate a first co-occurrence frequency table that stores the co-occurrence frequencies related to a plurality of word combinations in a matrix format, and further generates second document data. The frequency with which the combined words appear in the predetermined range group included in the group is calculated as the co-occurrence frequency related to the combination of the words, and the co-occurrence frequencies related to the combination of the plurality of words are calculated in a matrix format The co-occurrence frequency calculation processing procedure for generating the second co-occurrence frequency table stored in (2) singular value decomposition of the first co-occurrence frequency table, and in the meaning space of the reduced dimension as a degenerate matrix Calculate a first concept dictionary consisting of concept vectors for each word, further decompose the second co-occurrence frequency table into singular values, and use the second concept consisting of concept vectors for each word in the reduced dimension semantic space as a degenerate matrix Singular value decomposition processing procedure for calculating a dictionary (3) Based on the concept vector for each word in the first concept dictionary, the difference in concept of the comparison target word with respect to the reference word is calculated, and further the concept for each word in the second concept dictionary Based on the vector, the difference of the concept of the comparison target word with respect to the reference word is calculated, and the comparison target word is changed between the concept dictionaries based on the difference of the concept in the first concept dictionary and the difference of the concept in the second concept dictionary. The concept dictionary difference extraction process procedure which determines whether it corresponds to the difference of this, and uses the word equivalent to a difference as concept dictionary difference information.

The program according to the present invention is:
A program for causing a computer to be a feature word extraction system to execute the following procedure: (1) Frequency of occurrence of combined words in a predetermined range group included in a document data group Is calculated as a co-occurrence frequency related to the combination of the words, and a co-occurrence frequency calculation processing procedure (2) for generating a co-occurrence frequency table storing the co-occurrence frequencies related to the plurality of word combinations in a matrix format. Singular value decomposition of the occurrence frequency table and calculation of a concept dictionary composed of concept vectors for each word in the reduced dimension semantic space as a degenerate matrix (3) A concept vector for each word in the concept dictionary Based on the comparison, the difference of the concept of the comparison target word with respect to the reference word is calculated, and the comparison of the concept of the comparison target word with respect to the reference word is smaller than a predetermined reference By counting the number of elephants words, determine the number of near-concept word with respect to a reference word, the near concept word count calculation processing procedure for generating a near-concept word number table of near-concept word count of each reference word.

The program according to the present invention is:
A program for causing a computer to be a feature word extraction system to execute the following procedure. (1) Combined words appear together in a predetermined range group included in the first document data group. To generate a first co-occurrence frequency table that stores the co-occurrence frequencies related to a plurality of word combinations in a matrix format, and further generates second document data. The frequency with which the combined words appear in the predetermined range group included in the group is calculated as the co-occurrence frequency related to the combination of the words, and the co-occurrence frequencies related to the combination of the plurality of words are calculated in a matrix format The co-occurrence frequency calculation processing procedure for generating the second co-occurrence frequency table stored in (2) singular value decomposition of the first co-occurrence frequency table, and in the meaning space of the reduced dimension as a degenerate matrix Calculate a first concept dictionary consisting of concept vectors for each word, further decompose the second co-occurrence frequency table into singular values, and use the second concept consisting of concept vectors for each word in the reduced dimension semantic space as a degenerate matrix Singular value decomposition processing procedure for calculating a dictionary (3) Based on the concept vector for each word in the first concept dictionary, the difference of the concept of the comparison target word with respect to the reference word is calculated, and the concept of the comparison target word with respect to the reference word Counting the number of comparison target words whose difference is smaller than a predetermined reference, obtaining the number of close concept words for the reference word, generating a first close concept word number table comprising the number of close concept words for each reference word, and Based on the concept vector for each word in the second concept dictionary, the difference in concept of the comparison target word with respect to the reference word is calculated, and the difference in concept of the comparison target word with respect to the reference word is predetermined Counts the number of comparison target words that are smaller than the quasi, obtains the number of near concept words with respect to the reference word, and generates a second concept word count table including the number of near concept words for each reference word Step (4) For each reference word, the number of near concept words in the first near concept word number table is compared with the number of near concept words in the second near concept word number table, and a difference in the number of near concept words is obtained. The near concept word number comparison processing procedure which produces | generates the near concept word number difference table which consists of a near concept word number difference.

  In the present invention, since differences in concept dictionaries are extracted based on a plurality of concept dictionaries derived from different document data groups, it is possible to detect a difference or change in concept for words occurring between document data groups. it can. This difference or change in concept is useful for predicting the difference or change in the consciousness of the creator who created the document data group. For example, when a questionnaire for stratified respondents is used as a document data group, it is useful to grasp the difference in the consciousness of the respondents by stratification.

It is a figure which shows the structure of a feature word extraction system. It is a figure which shows the whole processing flow. It is a figure which shows the example of a 1st co-occurrence frequency table. It is a figure which shows the example of a 2nd co-occurrence frequency table. It is a figure which shows the example of a 1st concept dictionary. It is a figure which shows the example of a 2nd concept dictionary. It is a figure which shows the structure of the concept dictionary difference extraction part which concerns on Embodiment 1. FIG. It is a figure which shows the concept dictionary difference extraction process flow which concerns on Embodiment 1. FIG. It is a figure which shows the example of a 1st near concept word list. It is a figure which shows the example of a 2nd near concept word list. It is a figure which shows a near concept word extraction process flow. It is a figure which shows a near concept word comparison process flow. It is a figure which shows the example of concept dictionary difference information. It is a figure which shows the reduction word part comparison processing flow which concerns on Embodiment 1. FIG. It is a figure which shows the increase word part comparison processing flow which concerns on Embodiment 1. FIG. It is a figure which shows the structure of the concept dictionary difference extraction part which concerns on Embodiment 2. FIG. It is a figure which shows the concept dictionary difference extraction process flow which concerns on Embodiment 2. FIG. It is a figure which shows the example of a 1st concept difference table. It is a figure which shows the example of a 2nd concept difference table. It is a figure which shows a concept difference calculation process flow. It is a figure which shows the reduction word part comparison processing flow which concerns on Embodiment 2. FIG. It is a figure which shows the increase word part comparison processing flow which concerns on Embodiment 2. FIG. FIG. 10 is a diagram showing a reduced word part comparison processing flow according to Embodiment 3; FIG. 10 is a diagram showing an increased word comparison processing flow according to the third embodiment. It is a figure which shows the example of the 1st near concept word list which concerns on Embodiment 4. FIG. It is a figure which shows the example of the 2nd near concept word list which concerns on Embodiment 4. FIG. It is a figure which shows the reduction word part comparison processing flow which concerns on Embodiment 4. FIG. It is a figure which shows the increase word part comparison processing flow which concerns on Embodiment 4. FIG. It is a figure which shows the example of the concept dictionary difference information which concerns on Embodiment 4. FIG. It is a figure which shows the reduction word part comparison processing flow which concerns on Embodiment 5. FIG. FIG. 10 is a diagram showing an increased word comparison processing flow according to the fifth embodiment. It is a figure which shows the example of the concept dictionary difference information which concerns on Embodiment 6. FIG. It is a figure which shows the structure of the feature word extraction system which concerns on Embodiment 7. FIG. FIG. 20 is a diagram showing an overall processing flow according to Embodiment 7. It is a figure which shows the structure of a near concept word number calculation part. It is a figure which shows the near concept word number calculation processing flow. It is a figure which shows a near concept word number count processing flow. FIG. 38 is a diagram showing an example of a near concept word list according to the seventh embodiment. It is a figure which shows the example of a near concept word number table. It is a figure which shows the example of the near concept word number table sorted. FIG. 20 is a diagram showing a near concept word number calculation processing flow according to the eighth embodiment. It is a figure which shows a part of structure of the feature word extraction system which concerns on Embodiment 9. FIG. FIG. 20 is a diagram showing an overall processing flow according to the ninth embodiment. It is a figure which shows a near concept word number comparison processing flow. It is a figure which shows the example of a 2nd near concept word number table. It is a figure which shows the example of a near concept word number difference table. It is a figure which shows the example of the sorted table of near concept words. It is a figure which shows a part of structure of the feature word extraction system which concerns on Embodiment 10. FIG. FIG. 20 is a diagram showing an overall processing flow according to Embodiment 10; FIG. 38 is a diagram showing a processing flow for calculating the number of near concept words according to the tenth embodiment. FIG. 38 shows an example of conceptual dictionary difference information according to Embodiment 10. It is a figure which shows the near concept word number comparison processing flow which concerns on Embodiment 11. FIG. It is a figure which shows the example of the 1st near concept word number table which concerns on Embodiment 11. FIG. It is a figure which shows the example of the 2nd near concept word number table which concerns on Embodiment 11. FIG. It is a figure which shows the example of the near concept word number difference table which concerns on Embodiment 11. FIG. It is a figure which shows the hardware structural example of a feature word extraction system.

Embodiment 1 FIG.
Hereinafter, the present invention will be described based on embodiments shown in the drawings. FIG. 1 is a diagram illustrating a configuration of a feature word extraction system. The feature word extraction system includes a first document data group storage unit 101, a co-occurrence frequency calculation unit 102, a first co-occurrence frequency table storage unit 103, a singular value decomposition unit 104, a first concept dictionary storage unit 105, and second document data. It has elements of a group storage unit 106, a second co-occurrence frequency table storage unit 107, a second concept dictionary storage unit 108, a concept dictionary difference extraction unit 109, a concept dictionary difference information storage unit 110, and a concept dictionary difference information output unit 111. is doing.

  FIG. 2 is a diagram showing an overall processing flow. First, a first co-occurrence frequency calculation process (S201) is performed by the co-occurrence frequency calculation unit 102 for the first document data group, and a first singular value decomposition process is performed by the singular value decomposition unit 104 for the first co-occurrence frequency table. (S202) is performed. Next, a second co-occurrence frequency calculation process (S203) is performed by the co-occurrence frequency calculation unit 102 for the second document data group, and a second singular value decomposition unit 104 is further processed by the singular value decomposition unit 104 for the second co-occurrence frequency table. Processing (S204) is performed. Then, with respect to the first concept dictionary and the second concept dictionary, the concept dictionary difference extraction unit 109 performs concept dictionary difference extraction processing (S205), and finally the concept dictionary difference information output unit 111 performs concept dictionary difference output processing (S206). Do.

  In the co-occurrence frequency calculation process by the co-occurrence frequency calculation unit 102, the frequency at which two word combinations appear together in a predetermined range (the same document, the same paragraph, the same sentence, etc.) is calculated as the co-occurrence frequency. In the first co-occurrence frequency calculation process (S201), the co-occurrence frequency is calculated for the first document data group, and the co-occurrence frequency is stored in the first co-occurrence frequency table for each word combination. In the second co-occurrence frequency calculation process (S203), the co-occurrence frequency is calculated for the second document data group, and the co-occurrence frequency is stored in the second co-occurrence frequency table for each word combination.

  An example of a specific procedure of the co-occurrence frequency calculation process will be described. First, a predetermined range group (for example, a document, a paragraph, or a sentence) included in the document data group is specified. In addition, words included in the document data group are extracted, and those words are combined to identify a combination of words for which the co-occurrence frequency is obtained. Then, the following processing is repeated for each word combination. The number of predetermined ranges including both words related to the combination is calculated. Then, the number is divided by the total number in a predetermined range, and the co-occurrence frequency of the word combination is obtained as a quotient. Co-occurrence frequencies are stored in the co-occurrence frequency table in association with word combinations. This process is performed for all word combinations.

  FIG. 3 is a diagram illustrating an example of the first co-occurrence frequency table. FIG. 4 is a diagram illustrating an example of the second co-occurrence frequency table. All the co-occurrence frequency tables have the same configuration. A record is provided for each word, and the co-occurrence frequency related to the combination with the other word is stored. That is, the co-occurrence frequency table is a table that stores the co-occurrence frequencies in a combination of one word (row) and the other word (column) in a matrix format.

  In the above-described co-occurrence frequency table, there are many word groups (T1, T2, T3,...) In the column and the capacity is enormous, so the first singular value decomposition process (S202) by the concept dictionary difference extraction unit 109 and the second In the singular value decomposition process (S204), dimension reduction is performed. Singular value decomposition is a calculation process of linear algebra that decomposes an arbitrary m × n matrix.

A = UΣVT
The matrices U and V are orthogonal matrices, and are matrices composed of eigenvectors. The matrix Σ is a diagonal matrix and is called a singular value matrix. The subscript T indicates transposition. In this process, the co-occurrence frequency table is used as a matrix A, and a singular value decomposition is performed to obtain a degenerate matrix U, which is used as a concept dictionary.

  The calculation of the above-described concept vector uses, for example, the method disclosed in Non-Patent Document 1 (“Information Search System InfoMAP Based on Word Association”).

FIG. 5 is a diagram illustrating an example of the first concept dictionary. FIG. 6 is a diagram illustrating an example of the second concept dictionary. The concept dictionary is a dictionary that stores words and concept vectors in association with each other. Therefore, a record is provided for each word, and a concept vector is stored. The concept vector is composed of values (U1, U2, U3,...) For the basis vectors (e1, e2, e3,...). The basis vector is a linear sum of a plurality of words as shown below.
e1 = x11 · T1 + x12 · T2 + x13 · T3 +
e2 = x21 · T1 + x22 · T2 + x23 · T3 +
e3 = x31 · T1 + x32 · T2 + x33 · T3 +
These basis vectors (e1, e2, e3,..., EN) constitute an N-dimensional semantic space. The concept vector indicates the concept of the word as a vector in an N-dimensional meaning space. Therefore, the vectors of the words of the concepts that are approximated indicate directions close to each other.

  Next, in order to detect a change in the concept of the word in the first document data group and the concept of the word in the second document data group, a difference between the concept dictionaries is obtained. In the present embodiment, for each reference word, a near concept word is extracted from the concept dictionary, and based on the near concept word, a word whose concept has changed as a whole is grasped.

  FIG. 7 is a diagram showing a configuration of the concept dictionary difference extraction unit according to the first embodiment. The concept dictionary difference extraction unit 109 includes elements of a near concept word extraction unit 701, a first near concept word list storage unit 702, a second near concept word list storage unit 703, and a near concept word comparison unit 704.

  FIG. 8 is a diagram showing a conceptual dictionary difference extraction processing flow according to the first embodiment. First, with respect to the first concept dictionary, the near concept word extraction unit 701 performs a first near concept word extraction process (S801). Next, with respect to the second concept dictionary, the second concept word extraction unit 701 performs the second concept word extraction process (S802). Then, the near concept word comparison unit 704 performs near concept word comparison processing (S803) on the first near concept word list and the second near concept word list.

  In the present invention, words close to each other in concept vectors (words having a small concept difference) are treated as near concept words. This is because concepts that are close to each other are considered close in concept. In addition, it is also possible to distinguish and handle according to the strength of relevance, such as a synonym, a synonym, and a related word, in order from the closest degree. These synonyms, synonyms, and related terms are examples of near-concept words.

  FIG. 9 is a diagram illustrating an example of the first near concept word list. FIG. 10 is a diagram illustrating an example of the second near concept word list. A record is provided for each reference word, and items of near concept words are included. That is, a plurality of conceptual words close to the word can be stored for the reference word.

  FIG. 11 is a diagram showing a near concept word extraction processing flow.

  For each word, the following processing is repeated using the word as a reference (S1101). This corresponds to a line in the concept dictionary. Further, for each other word other than the reference word, the following processing is repeated with the other word as an extraction candidate (S1102). This corresponds to the other lines of the concept dictionary.

  A difference in concept between the reference word and the extraction candidate word is calculated (S1103). For example, when calculating the difference between the concepts of “word a” and “word b”, the concept vector 551 and the concept vector 552 in FIG. 5 or the concept vector 651 and the concept vector 652 in FIG. As a basis. The difference in concept can be determined based on the proximity of the concept vector. For example, the angle θ formed by both vectors is calculated, cos (θ) is calculated, and this is set as the proximity. In this case, the greater the degree of proximity is, the closer the degree of proximity is to 1. Therefore, it can be determined that the greater the degree of proximity, the smaller the difference in concept, so 1-cos (θ) is treated as the difference in concept. Further, the angle θ itself formed by both vectors can be regarded as a conceptual difference. sin (θ) and tan (θ) may be the difference in concept. Further, both unit vectors may be (x1, y1) and (x2, y2), and ((x1-x2) 2+ (y1-y2) 2) 1/2 may be obtained and treated as a difference in concept.

  It is determined whether the concept difference is equal to or less than a predetermined threshold. That is, it is determined whether the concept is approximated by a predetermined standard (S1104). If it is less than or equal to a predetermined threshold (when the concept is closer to a predetermined reference), the extraction candidate word is stored as a near concept word in association with the reference word (reference word) in the near concept word list. (S1105).

  When this process is processed for all other words (S1106), the process proceeds to the process based on the next word, and when all the reference words are processed (S1107), the process ends.

  Thus, in the first near concept word extraction process (S801), a first near concept word list is generated from the first concept dictionary, and in the second near concept word extraction process (S802), the first concept word list is extracted from the second concept dictionary. A near concept word list is generated.

  In the first near concept word list in FIG. 9, “word c” and “word ho” are stored as near concept words for “word a”, but in the second near concept word list in FIG. Only “word c” is stored as a near concept word for “i”. In other words, “word ho” is decreasing as a near concept word for “word a”. Further, in the first near concept word list of FIG. 9, “word d” is stored as a near concept word for “word b”, but in the second near concept word list of FIG. In addition to “word d”, “word f” is stored as a near concept word. That is, “to word” is increasing as a near concept word for “word b”. Next, the change of such a near concept word is grasped.

  FIG. 12 is a diagram showing a near concept word comparison processing flow. The near concept word comparison process (S803) includes a decrease word part comparison process (S1201) and an increase word part comparison process (S1202).

  First, the concept dictionary difference information obtained by the near concept word comparison process will be described. FIG. 13 is a diagram illustrating an example of the concept dictionary difference information. A record is provided for each reference word, and items of an increase word and a decrease word are provided. In this way, a plurality of increased words and decreased words can be stored in association with the reference word.

  In the reduced word part comparison process, near concept words that are in the first near concept word list but not in the second near concept word list are obtained.

  FIG. 14 is a diagram showing a reduced word part comparison processing flow according to the first embodiment. The following processing is repeated for each reference word in the first near concept word list (S1401). This is a process for each row of the list. Further, the following processing is repeated for each near concept word associated with the reference word in the first near concept word list (S1402). This is a process for each column in the list.

  The second near concept word list is compared with any of the near concept word groups associated with the reference word (S1403). For example, when the reference word is “word a” and processing is performed on the word “word ho” of the near concept word (second column of 951 rows of the first near concept word list shown in FIG. 9), the processing is shown in FIG. Then, it is determined whether or not “word e” is included in the near concept word group in line 1051 of the second near concept word list. If it is not included, that is, if it does not match either (S1404), the near concept word in the first near concept word list is stored as the decrease word in the concept dictionary difference information in association with the reference word (S1405). ). On the other hand, if it matches either, the conceptual dictionary difference information is not updated. In the example of the conceptual dictionary difference information illustrated in FIG. 13, “word e” is a reduced word with respect to “word a” as the reference word, and “word i” is also a reduced word with respect to “word ho” as the reference word. ing.

  When all near concept words have been processed (S1406), the process proceeds to processing of the next reference word, and when all reference words have been processed (S1407), the process ends.

  In the increased word comparison processing, near concept words that are in the second near concept word list but not in the first near concept word list are obtained.

  FIG. 15 is a diagram showing an increased word comparison processing flow according to the first embodiment. The following processing is repeated for each reference word in the second near concept word list (S1501). Further, the following processing is repeated for each near concept word associated with the reference word in the second near concept word list (S1502).

  The first near concept word list is compared with any of the near concept word groups associated with the reference word (S1503). For example, when the reference word is “word b” and the near concept word “to word” (second column of line 1052 in the second near concept word list shown in FIG. 10) is processed, In addition, it is determined whether or not “to word” is included in the near concept word group of 952 lines in the first near concept word list. If it is not included, that is, if it does not match any of them (S1504), the near concept word in the second near concept word list is stored as an increase word in the concept dictionary difference information in association with the reference word (S1505). ). On the other hand, if it matches either, the conceptual dictionary difference information is not updated. In the example of the conceptual dictionary difference information illustrated in FIG. 13, “word f” is an increased word regarding “word b” of the reference word, and “word b” is an increased word regarding “word f” of the reference word. ing.

  When all near concept words have been processed (S1506), the process proceeds to the next reference word processing, and when all reference words have been processed (S1507), the process ends.

  Thus, the obtained concept dictionary difference information is stored in the concept dictionary difference information storage unit 110. And then. The concept dictionary difference information output unit 111 reads the concept dictionary difference information from the concept dictionary difference information storage unit 110 and outputs it. In the present embodiment, it is possible to extract a difference between near concept words in a plurality of concept dictionaries.

Embodiment 2. FIG.
In Embodiment 1, the near concept word is determined for each word by the near concept word extraction process, and the difference in the concept dictionary is obtained based on the determined near concept word. In the present embodiment, a concept difference calculation process calculates a concept difference between a word and another word, and calculates a concept dictionary difference based on the concept difference between each word. In the present embodiment, conceptual dictionary difference information similar to that in the first embodiment can be obtained, but the processing procedure is different.

The present embodiment is characterized by the configuration and processing of the concept dictionary difference extraction unit. FIG. 16 is a diagram illustrating a configuration of the concept dictionary difference extraction unit according to the second embodiment. The concept dictionary difference extraction unit 109 according to Embodiment 2 includes a concept difference calculation unit 1601, a first concept difference table storage unit 1602, a second concept difference table storage unit 1603, and a concept difference comparison unit 1604. FIG. 17 is a diagram illustrating a conceptual dictionary difference extraction processing flow according to the second embodiment. First, with respect to the first concept dictionary, the concept difference calculation unit 1601 performs a first concept difference calculation process (S1701). Next, with respect to the second concept dictionary, the concept difference calculation unit 1601 performs a second concept difference calculation process (S1702). Then, with respect to the first near concept difference table and the second near concept difference table, the concept difference comparison unit 1604 performs a near concept word comparison process (S1703).

FIG. 18 is a diagram illustrating an example of the first concept difference table. FIG. 19 is a diagram illustrating an example of the second concept difference table. A record is provided for each reference word, and a conceptual difference from the comparison target word is stored. That is, it is configured so that the difference in concept with the reference word relative to the reference word can be stored in association with the combination of the reference word and the comparison word. The concept difference calculation method is the same as the concept difference calculation method in the first embodiment. In this example, a smaller value means a smaller concept difference. Regardless of this example, when the significance of the concept difference is given using the degree of approximation, a larger value of the degree of approximation means a smaller concept difference.

  The concept difference calculation process in the first concept difference calculation process (S1701) and the second concept difference calculation process (S1702) will be described.

  FIG. 20 is a diagram illustrating a conceptual difference calculation processing flow. For each word, the following processing is repeated using the word as a reference (S2001). This is a process for each line of the concept dictionary. Then, for each other word other than the reference word, the following processing is repeated using the other word as a comparison target (S2002). This is another line-by-line process of the concept dictionary.

  The difference between the concepts is calculated for the reference word and the comparison target word (S2003), and the difference between the concepts is stored in the first concept difference table in association with the reference word and the comparison target word (S2004). For example, when “word a” in the first concept dictionary shown in FIG. 5 is a reference word and “word b” is a comparison target word, the concept is based on the concept vector of 551 lines and the concept vector of 552 lines. And the difference of the concept is stored in the area (line 1831: 1862) for identifying those words in the first concept difference table shown in FIG.

  When all other words have been processed as comparison targets (S2005), the process proceeds to the processing of the next reference word, and when all the reference words have been processed (S2006), the process ends.

  Based on the first concept difference table and the second concept difference table thus obtained, concept difference comparison processing (S1703) is performed. The concept difference comparison process includes a decrease word part comparison process and an increase word part comparison process, similarly to the near concept word comparison process shown in FIG.

  In the reduced word comparison process, a reduced word that belongs to the near concept range in the first concept difference table and does not belong to the near concept range in the second concept difference table is obtained.

  FIG. 21 is a diagram showing a reduced word part comparison processing flow according to the second embodiment. The following processing is repeated for each reference word in the first concept difference table (S2101). This is a process for each row of the table. Further, the following process is repeated for each comparison target word associated with the reference word in the first concept difference table (S2102). This is a process for each column of the table.

  In the first concept difference table, the difference between the concepts associated with the reference word and the comparison target word is acquired (S2103), and the difference between the concepts (in the first concept difference table) is the threshold of the near concept range. If smaller than (S2104), the following processing is performed. For example, when the reference word is the word a and the comparison target word is the word ho, the concept difference '0.096' is read from the 1831 line: 1865 column of the first concept difference table shown in FIG. It is compared with the threshold value of the conceptual range (for example, “0.1”). Since it is smaller than the threshold value, it is determined whether the comparison target word corresponds to a decrease word by the following processing. If it is larger than the near concept range threshold value in S2104, the comparison target word does not originally correspond to the near concept word, so it is not necessary to determine a decrease word.

In the second concept difference table, the difference between the concepts associated with the reference word and the comparison target word is acquired (S2105), and the difference between the concepts (in the second concept difference table) is the threshold of the near concept range. If it is not smaller (S2106), the comparison target word is stored as the reduced word in the concept dictionary difference information in association with the reference word (S2107). For example, when the reference word is “word a” and the comparison target word is “word e”, the concept difference “0.241” from the 1931st row: 1965 column of the second concept difference table shown in FIG. Is compared with a threshold value in the near concept range (for example, “0.1”). Since it is not smaller than the threshold, “word ho” is treated as a reduced word. On the other hand, when it is smaller than the threshold of the near concept range, the concept dictionary difference information is not updated.

  When all the comparison target words have been processed (S2108), the process proceeds to the next reference word processing, and when all the reference words have been processed (S2109), the process ends.

  In the increased word comparison process, an increased word that belongs to the near concept range in the second concept difference table and does not belong to the near concept range in the first concept difference table is obtained.

  FIG. 22 is a diagram showing an increased word comparison processing flow according to the second embodiment. The following processing is repeated for each reference word in the second concept difference table (S2201). Further, the following process is repeated for each comparison target word associated with the reference word in the second concept difference table (S2202).

  In the second concept difference table, the difference between the concepts associated with the reference word and the comparison target word is acquired (S2203), and the difference between the concepts (in the second concept difference table) is the threshold of the near concept range. If smaller than (S2204), the following processing is performed. For example, when the reference word is “word b” and the comparison target word is “to word”, the concept difference “0.065” from the 1932 line: 1966 column of the second concept difference table shown in FIG. Is compared with a threshold value in the near concept range (for example, “0.1”). Since it is smaller than the threshold value, it is determined in the following process whether the comparison target word corresponds to the increased word. If it is greater than the near concept range threshold value in S2204, the comparison target word does not correspond to the near concept word in the second concept difference, and therefore it is not necessary to determine the increased word.

  In the first concept difference table, a difference between concepts associated with the reference word and the comparison target word is acquired (S2205), and the difference between the concepts (in the first concept difference table) is a threshold of the near concept range. If it is not smaller than (S2206), the comparison target word is stored as an increased word in the concept dictionary difference information in association with the reference word (S2207). For example, when the reference word is “word b” and the comparison target word is “word f”, the concept difference “0.281” from the 1832 row: 1866 column of the first concept difference table shown in FIG. Is compared with a threshold value in the near concept range (for example, “0.1”). Since it is not smaller than the threshold, “to word” is treated as an increased word. On the other hand, when it is smaller than the threshold of the near concept range, the concept dictionary difference information is not updated.

  When all the comparison target words have been processed (S2208), the process proceeds to the next reference word processing, and when all the reference words have been processed (S2209), the process ends. In the present embodiment, similar to the first embodiment, it is possible to extract the difference between near concept words in a plurality of concept dictionaries. In this process, the system stores a difference in concept between all words. As a high applicability.

Embodiment 3 FIG.
In the second embodiment, in the increased word portion comparison process and the decreased word portion comparison process, it is determined whether or not the concept word is a near concept word based on whether or not the concept difference exceeds the threshold value of the near concept range. The increase or decrease as was extracted as a difference. In this embodiment, an increase word and a decrease word are determined based on a change amount changed by a concept difference in the second concept difference table with respect to a concept difference in the first concept difference table.

The present embodiment differs from the second embodiment in the reduced word portion comparison processing and the increased word portion comparison processing.

  In the reduced word comparison process, a reduced word that belongs to the near concept range in the first concept difference table and whose concept difference has greatly changed in the second concept difference table is obtained.

FIG. 23 is a diagram showing a reduced word part comparison processing flow according to the third embodiment. The following processing is repeated for each reference word in the first concept difference table (S2301). Further, the following processing is repeated for each comparison target word associated with the reference word in the first concept difference table (S2302).
In the first concept difference table, a difference between concepts associated with the reference word and the comparison target word is acquired (S2303), and the difference between the concepts (in the first concept difference table) is a threshold of the near concept range. If smaller than (S2304), the following processing is performed. In the second concept difference table, the difference between the concepts associated with the reference word and the comparison word is acquired (S2305), and the concept difference in the first concept difference table is obtained from the concept difference in the second concept difference table. The difference is subtracted to obtain the change amount of the concept difference (S2306). If the change amount of the concept difference exceeds a threshold value indicating concept variation (S2307), the comparison target word is stored as the decrease word in the concept dictionary difference information in association with the reference word (S2308). For example, when the reference word is “word a” and the comparison target word is “word e”, the difference between the concepts obtained from the 1931 row: 1965 column of the second concept difference table shown in FIG. The concept difference “0.096” obtained from the 1831 row: 1865 column of the first concept difference table shown in FIG. 18 is subtracted from 241 ′ to obtain the concept difference “0.145”. Then, this difference is compared with a threshold value indicating conceptual variation (for example, “0.08”). Since the threshold value is exceeded, “word ho” is treated as a reduced word. If the change amount of the concept difference does not exceed the threshold value indicating the concept variation, the concept dictionary difference information is not updated.

When all the comparison target words have been processed (S2309), the process proceeds to the next reference word processing, and when all the reference words have been processed (S2310), the process ends.
FIG. 24 is a diagram showing an increased word comparison processing flow according to the third embodiment.

  The following processing is repeated for each reference word in the second concept difference table (S2401). Further, the following processing is repeated for each comparison target word associated with the reference word in the second concept difference table (S2402).

  In the second concept difference table, the difference between the concepts associated with the reference word and the comparison target word is acquired (S2403), and the difference between the concepts (in the second concept difference table) is the threshold of the near concept range. If smaller than (S2404), the following processing is performed. In the first concept difference table, a difference between concepts associated with the reference word and the comparison target word is acquired (S2405), and the concept difference in the second concept difference table is obtained from the concept difference in the first concept difference table. The difference is subtracted to obtain the change amount of the concept difference (S2406). If the change amount of the concept difference exceeds a threshold indicating concept variation (S2407), the comparison target word is stored as an increase word in the concept dictionary difference information in association with the reference word (S2408). For example, when the reference word is “word b” and the comparison target word is “to word”, the difference between the concepts obtained from row 1832: 1866 of the first concept difference table shown in FIG. The concept difference '0.065' obtained from 1932: 1966 column of the second concept difference table shown in FIG. 19 is subtracted from 281 ′ to obtain the concept difference “0.216”. Then, this difference is compared with a threshold value indicating conceptual variation (for example, “0.08”). Since the threshold value is exceeded, “to word” is treated as an increased word. If the change amount of the concept difference does not exceed the threshold value indicating the concept variation, the concept dictionary difference information is not updated.

When all the comparison target words have been processed (S2409), the process proceeds to the next reference word processing, and when all the reference words have been processed (S2410), the process ends.

  In the present embodiment, differences can be extracted based on the amount of concept variation in a plurality of concept dictionaries.

Embodiment 4 FIG.
In the present embodiment, among the increasing words, in the first near concept word list, the increasing word for the reference word for which no near concept word exists is particularly distinguished as “occurrence”, and among the decreasing words, the second word is identified as the second word. In the near concept word list, an explanation will be given of a mode in which a decrease word with respect to a reference word in which a near concept word no longer exists is distinguished as “disappearance”.

  In the first to third embodiments as well, an increase word corresponding to “occurrence” and a decrease word corresponding to “disappearance” are extracted as concept dictionary difference information. In the present invention, these are particularly distinguished.

  The near concept word list of FIGS. 25 and 26 will be described as an example. FIG. 25 is a diagram illustrating an example of a first near concept word list according to the fourth embodiment. FIG. 26 is a diagram illustrating an example of a second near concept word list according to the fourth embodiment. The configuration of the near concept word list is the same as that in the above example. In this example, “word le” with respect to the reference word “word” becomes an increased word related to “occurrence”, and “word n” with respect to the reference word “word” becomes a reduced word related to “annihilation”. .

  Next, the processing of the present embodiment will be described based on the first embodiment. A process for determining the reduction type is added to a part of the reduced word part comparison process flow of FIG. 14 described in the first embodiment, and is added to a part of the increased word part comparison process flow of FIG. 15 described in the first embodiment. Then, a process for determining the increase type is added.

  FIG. 27 is a diagram showing a reduced word part comparison processing flow according to the fourth embodiment. The following processing is performed between S1405 and S1406 in FIG. In S1405, the reduced word is stored in the concept dictionary difference information, and subsequently, the type of the reduced word is determined. Note that a reduction word other than “annihilation”, that is, a reduction word with respect to a reference word in which a near concept word remains in the second near concept word list, is distinguished as a “remaining” reduction type.

  It is determined whether there is a near concept word associated with the reference word in the second near concept word list (S2701). If there is a near concept word, the decrease type is associated with the decrease word. As “remaining” is stored in the concept dictionary difference information (S2702), and if there is no near concept word, “disappear” is stored in the concept dictionary difference information as a decrease type in association with the decreased word (S2703). .

  FIG. 28 is a diagram showing an increased word comparison processing flow according to the fourth embodiment. The following processing is performed between S1505 and S1506 in FIG. In S1505, the increased word is stored in the concept dictionary difference information, and subsequently, the type of the increased word is determined. Note that an increase word other than “occurrence”, that is, an increase word with respect to a reference word in which a near concept word exists in the first near concept word list, is distinguished as “addition” as an increase type.

  It is determined whether there is a near concept word associated with the reference word in the first near concept word list (S2801). If there is a near concept word, the increase type is associated with the increase word. "Addition" is stored in the concept dictionary difference information (S2802), and "occurrence" is stored in the concept dictionary difference information as an increase type in association with the increased word (S2803).

FIG. 29 is a diagram illustrating an example of conceptual dictionary difference information according to the fourth embodiment. As shown in the figure, items of an increase type 2904 and a decrease type 2905 are added to a record provided for each reference word, and an increase type for an increase word and a decrease type for a decrease word can be stored.

  Of the decreasing words, “word i” for the reference word “word a” and “word a” for the reference word “word e” are “remaining”, whereas “word i” The “word” for the reference word is distinguished as “disappearance”.

  Among the increased words, “to word” and “to word” for the reference word “to word” and “to word” have “increase” as the increase type, The “word” for the reference word is distinguished as “occurrence”.

  In the present embodiment, an increase word related to occurrence and a decrease word related to disappearance are distinguished, which is useful for extracting feature words related to concept occurrence and disappearance.

Embodiment 5 FIG.
In the present embodiment, as in the fourth embodiment, an embodiment will be described in which a process for distinguishing an increase word related to occurrence and a decrease word related to disappearance is realized based on the second embodiment. A process for determining the reduction type is added to a part of the reduced word part comparison process flow of FIG. 21 described in the second embodiment, and a part of the increased word part comparison process flow of FIG. 22 described in the second embodiment. Then, a process for determining the increase type is added.

  FIG. 30 is a diagram showing a reduced word part comparison processing flow according to the fifth embodiment. The following processing is performed between S2107, S2108, and S2109 in FIG. In S2107, the reduced word is stored in the concept dictionary difference information, and subsequently, it is stored internally that there is a reduced word (S3001). The information stored here is used only in S3002, and is cleared after the determination in S3002. When all the comparison target words have been processed (S2108), it is determined whether or not there are reduced words (S3002).

  If it is not stored that there is a decrease word, it is not necessary to determine the decrease type, and the process proceeds to S2109. On the other hand, when it is stored that there is a decrease word, the decrease type is determined by the following processing.

  The following processing is repeated for each comparison target word in the second concept difference table (S3003). In the second concept difference table, the difference between the concepts associated with the reference word and the comparison target word is acquired (S3004), and it is determined whether the concept difference is smaller than the threshold of the near concept range (S3005). . If any of the comparison target words has a concept difference smaller than the near concept range threshold, it means that there is a near concept word. Is stored in the concept dictionary difference information (S3008). Even if all of the comparison target words are processed (S3006), if there is no comparison target word whose concept difference is smaller than the threshold value of the near concept range, it means that there is no near concept word, so the reduced word In association with, “disappear” is stored in the conceptual dictionary difference information as the reduction type (S3007).

  FIG. 31 is a diagram showing an increased word comparison processing flow according to the fifth embodiment. The following processing is performed between S2207, S2208, and S2209 in FIG. In S2207, the increased word is stored in the concept dictionary difference information, and subsequently, it is internally stored that there is an increased word (S3101). The information stored here is used only in S3102 and is cleared after the determination in S3102. When all the comparison target words have been processed (S2208), it is determined whether there is an increased word (S3102).

  If it is not stored that there is an increased word, there is no need to determine the increased type, and the process proceeds to S2209. On the other hand, when it is stored that there is an increased word, the increase type is determined by the following process.

  The following processing is repeated for each comparison target word in the first concept difference table (S3103). In the first concept difference table, the difference between the concepts associated with the reference word and the comparison target word is acquired (S3104), and it is determined whether the concept difference is smaller than the threshold of the near concept range (S3105). ). If any of the comparison target words has a concept difference smaller than the threshold of the near concept range, it means that there is a near concept word, so “add” as an increase type in association with the increase word. Is stored in the concept dictionary difference information (S3108). Even if all of the comparison target words are processed (S3106), if there is no comparison target word whose concept difference is smaller than the threshold value of the near concept range, it means that there is no near concept word. In association with, “occurrence” as the increase type is stored in the conceptual dictionary difference information (S3107).

  In the present embodiment, as in the fourth embodiment, the increased word related to occurrence and the decreased word related to disappearance are distinguished, which is useful for extracting feature words related to the occurrence and disappearance of concepts.

Embodiment 6 FIG.
In the present embodiment, another configuration example is shown for the concept dictionary difference information used in the fourth embodiment and the fifth embodiment. FIG. 32 is a diagram illustrating an example of conceptual dictionary difference information according to the sixth embodiment. In this example, for each reference word, there are items of an additional word and a generated word, and a remaining reduced word and an erasure word. As a result, the increase word is stored separately as an additional word and an occurrence word, and the decrease word is stored as a remaining decrease word and an extinction word.

  When this configuration is used in the fourth embodiment, in S1405 of FIG. 27, the decrease word is temporarily stored in the area of the remaining decrease word (or disappearance word), and the decrease temporarily stored in S2702 The word is stored again in the remaining reduced word area, and the disappeared word area is cleared. In S2703, the temporarily stored reduced word is stored again in the erasure word area, and the remaining reduced word area is cleared. Further, in S1505 of FIG. 28, the increased word is temporarily stored in the additional word (or generated word) area, and in S2802, the increased word temporarily stored is stored again in the additional word area. Clear the word area. In S2703, the temporarily stored increase word is stored again in the generated word area, and the additional word area is cleared.

  When this configuration is used in the fifth embodiment, the reduced word is temporarily stored in the remaining reduced word (or disappeared word) area in S2107 of FIG. 30, and the temporarily stored decrease in S3007. The words are stored again in the disappeared word area, and the remaining reduced word area is cleared. In S3008, the temporarily stored reduced word is stored again in the remaining reduced word area, and the disappeared word area is cleared. In S2207 of FIG. 31, the increased word is temporarily stored in the additional word (or generated word) area, and in S3107, the increased word temporarily stored is newly stored in the generated word area and added. Clear the word area. In step S3108, the temporarily stored increase word is stored again in the additional word area, and the generated word area is cleared.

  In the present embodiment, another data structure for distinguishing an increase word related to occurrence and a decrease word related to annihilation can be employed.

Embodiment 7 FIG.
In the first embodiment to the sixth embodiment, the apparatus and method for obtaining the difference between near concept words between two concept dictionaries as a difference have been described, but in this embodiment, the near concept words included in the concept dictionary An apparatus and a method for obtaining the number of the above will be described.

  FIG. 33 is a diagram illustrating a configuration of a feature word extraction system according to Embodiment 7. In this example, the number of near concept words for each reference word is obtained from one concept dictionary. The document data group storage unit 3301 has the same configuration as the first document data group storage unit 101. The co-occurrence frequency table storage unit 3302 has the same configuration as the first co-occurrence frequency storage unit 103. The concept dictionary storage unit 3303 has the same configuration as the first concept dictionary storage unit 105. The near concept word number calculation unit 3304 is configured to count the number of near concept words for each reference word. The near concept word number table storage unit 3305 is configured to store a near concept word number table that stores the number of near concept words for each reference word. The near concept word number sorting unit 3306 is configured to sort records in the concept word number table using the number of near concept words as a key. The near concept word number table output unit 3307 is configured to output a near concept word number table.

  Next, processing will be described. FIG. 34 is a diagram showing an overall processing flow according to the seventh embodiment. The co-occurrence frequency calculation unit 102 performs co-occurrence frequency calculation processing (S3401), and the singular value decomposition unit 104 performs singular value decomposition processing (S3402). These processes are the same as those described above.

  The near concept word number calculation processing (S3403) by the near concept word number calculation unit 3304 will be described. FIG. 35 is a diagram illustrating a configuration of the near concept word number calculation unit. The near concept word number calculation unit 3304 includes a near concept word extraction unit 701, a near concept word list storage unit 3501, and a near concept word count unit 3502. The near concept word list storage unit 3501 has the same configuration as the first near concept word list storage unit 702 described above.

  FIG. 36 is a diagram showing a processing flow for calculating the number of near concept words. The near concept word extraction process (S3601) by the near concept word extraction unit 701 is the same as described above. Next, the near concept word number counting process (S3602) by the near concept word number counting unit 3502 will be described.

  FIG. 37 is a diagram showing a near concept word number count processing flow. The following processing is repeated for each reference word in the near concept word list (S3701). In the near concept word list, the number of near concept words associated with the reference word is counted (S3702), and the counted number of near concept words is stored in the near concept word number table in association with the reference word ( S3703). Then, the process ends when all the reference words have been processed (S3704).

  As shown in the example of the near concept word list in FIG. 38, the number of near concept words varies depending on the reference word. The result of counting these numbers is stored as a near concept word number table. FIG. 39 is a diagram illustrating an example of a near concept word number table. As shown in the figure, a record is provided for each reference word, and the reference word and the number of nearby concept words are stored in association with each other.

  When a reference word having a large number of near concept words is to be extracted or a reference word having a small number of near concept words is to be extracted, the near concept word number sorting process is performed by the near concept word number sorting unit 3306 (S3404 in FIG. 34). It is effective to do. In this example, sorting is performed in descending order using the number of near concept words as a key, and processing is performed so as to obtain a sorted near concept word number table as shown in FIG.

Finally, the near concept word number table output process (S3405 in FIG. 34) by the near concept word number table output unit 3307 outputs the near concept word number table. When outputting the sorted near concept word count table, only records having a number of near concept words equal to or greater than a predetermined threshold (or less than a predetermined threshold) may be partially output. It is also possible to output an unsorted near concept word count table.

  A reference word having a large number of near-concept words (for example, the number of synonyms, the number of synonyms, or the number of related words) means that it is frequently used in the document data group. Therefore, the reference word is a feature word of the concept dictionary. Can be. According to the present embodiment, since reference words having a large number of near concept words are listed, feature units can be extracted by comparing concept units in concept dictionaries.

Embodiment 8 FIG.
In the near concept word count calculation process in the seventh embodiment, the near concept word list is once obtained from the concept dictionary, and then the number of near concept words in the list is counted. In the present embodiment, a calculation process for counting the number of close concept words directly from the concept dictionary will be described.

  FIG. 41 is a diagram showing a processing flow for calculating the number of near concept words according to the eighth embodiment. For the concept dictionary, the following processing is repeated for each word with reference to the word (S4101). For each other word other than the reference word, the following process is further repeated with the other word as a count candidate (S4102). A concept difference is calculated for the reference word and the count candidate word (S4103), and it is determined whether the concept difference is equal to or less than a predetermined threshold (S4104). If it is equal to or less than the threshold value, it means that the count candidate is a near concept word, so 1 is added to the number of near concept words associated with the reference word in the near concept word number table (S4105). Note that the initial value of the number of near concept words is zero. When all the other words have been processed (S4106), the process proceeds to the next word, and finally when all the words have been processed (S4107), the process ends.

  In the present embodiment, the same effect as in the seventh embodiment can be obtained by a procedure for calculating the number of near concept words different from that in the seventh embodiment.

Embodiment 9 FIG.
In the eighth embodiment, the apparatus and method for obtaining the number of near concept words included in one concept dictionary have been described. However, in the present embodiment, the number of near concept words is obtained for each of the two concept dictionaries, An apparatus and method for obtaining the difference in the number of near concept words by comparing them will be described.

  FIG. 42 is a diagram illustrating a part of the configuration of the feature word extraction system according to the ninth embodiment. The feature word extraction system according to the present embodiment has the elements 101 to 108 shown in FIG. Furthermore, a near concept word number calculation unit 3304, a first near concept word number table storage unit 4201, a second near concept word number table storage unit 4202, a near concept word number comparison unit 4203, a near concept word number difference table storage unit 4204, It has elements of a near concept word number difference sorting unit 4205 and a near concept word number difference table output unit 4206.

  Next, processing will be described. FIG. 43 is a diagram showing an overall processing flow according to the ninth embodiment. As in S201 and S202 of FIG. 2, the first co-occurrence frequency calculation process (S4301) and the first singular value decomposition process (S4302) are performed on the first document data group, and the seventh embodiment or the eighth embodiment. As described above, the near concept word number calculation unit 3304 performs the first near concept word number calculation process (S4303) for generating the first near concept word number table from the first concept dictionary. Further, the second co-occurrence frequency calculation process (S4304) and the second singular value decomposition process (S4305) are performed on the second document data group as in S203 and S204 of FIG. As in Form 8, the near concept word number calculation unit 3304 performs a second near concept word number calculation process (S4306) for generating a second near concept word number table from the second concept dictionary.

  Next, the near concept word number comparison unit 4203 performs a near concept word number comparison process (S4307). FIG. 44 is a diagram showing a near concept word number comparison processing flow. The following processing is repeated for each reference word in the first near concept word number table (or second near concept word number table) (S4401). The number of near concept words (the number of first near concept words) associated with the reference word is acquired from the first near concept word number table (S4402), and the reference from the second near concept word number table. The number of near concept words associated with the word (second near concept word number) is acquired (S4403). Then, the first near concept word number is subtracted from the second near concept word number to obtain a near concept word number difference as a difference (S4404), the near concept word number difference is associated with the reference word, It memorize | stores in a concept word number difference table (S4405). Then, the process ends when all the reference words have been processed (S4406).

  The near concept word number table shown in FIG. 39 is used as the first near concept word number table, and the result of comparison of the number of near concept words with the second near concept word number table shown in FIG. 45 is shown in FIG. . FIG. 46 is a diagram illustrating an example of a near concept word number difference table. This table is configured to store the reference word and the difference in the number of close concept words in association with each reference word.

  FIG. 47 shows the result of performing the near concept word number difference sorting process (S4308) for sorting the record groups in this table using the near concept word number difference as a key. FIG. 47 is a diagram illustrating an example of a sorted difference word count difference table. In the figure, the reference word having a positive difference in the number of near concept words indicates that the number of near concept words in the second concept dictionary exceeds the number of near concept words in the first concept dictionary. A reference word having a negative concept word number difference indicates that the number of near concept words in the second concept dictionary is less than the number of near concept words in the first concept dictionary.

  Finally, in the near concept word number difference table output unit 4206 by the near concept word number difference table output process (S4309), the sorted near concept word number difference table is output. In addition, when outputting the sorted near concept word number difference table, only the record group having the number of near concept words equal to or greater than a predetermined threshold or the number of near concept words equal to or less than the predetermined threshold may be output. . An unsorted near concept word number difference table can also be output.

  According to the present embodiment, the number of near concept words is obtained for each of the two concept dictionaries, and the difference between the number of near concept words is obtained by comparing them. Therefore, the scale of the concept unit between the two concept dictionaries is increased. The feature words based on the change in the concept scale can be extracted.

Embodiment 10 FIG.
In the present embodiment, as in the ninth embodiment, another form for obtaining the difference in the number of near concept words for two concept dictionaries will be described. In the present embodiment, the difference in the number of near concept words is obtained based on the concept dictionary difference information obtained by the processing according to any one of the first to third embodiments.

  FIG. 48 is a diagram illustrating a part of the configuration of the feature word extraction system according to the tenth embodiment. The configuration for obtaining the concept dictionary difference information is the same as that in any one of the first to third embodiments. In addition to the concept dictionary difference information storage unit 110 that stores the concept dictionary difference information, the near concept word number difference calculation unit 4801, the near concept word number difference table storage unit 4204, the near concept word number difference sort unit 4205, A conceptual word number difference table output unit 4206 is provided.

Processing will be described. FIG. 49 is a diagram showing an overall processing flow according to the tenth embodiment. Similar to S201 to S205 in FIG. 2, the first co-occurrence frequency calculation process (S4901), the first singular value decomposition process (S4902), the second co-occurrence frequency calculation process (S4903), and the second singular value decomposition process (S4904). ) And conceptual dictionary difference extraction processing (S4905). Subsequently, the near concept word number difference calculation unit 4801 performs a near concept word number difference calculation process (S4906).

  The near concept word number difference calculation process (S4906) will be described in detail. FIG. 50 is a diagram showing a processing flow for calculating the number of near concept words according to the tenth embodiment. The following processing is repeated for each reference word in the concept dictionary difference information (S5001). The number of increased words associated with the reference word is counted to obtain an increased number of near concept words (S5002). Further, the number of reduced words associated with the reference word is counted to obtain the number of near concept words reduced (S5003). Then, the near concept word count difference is obtained by subtracting the near concept word count from the near concept word increment count (S5004). The near concept word number difference is stored in the near concept word number difference table in association with the reference word (S5005). When these processes are performed for all the reference words, the process ends (S5006).

  FIG. 51 is a diagram showing an example of the concept dictionary difference information according to the tenth embodiment. Taking the concept dictionary difference information shown in FIG. 51 as an example, the near concept word number difference table shown in FIG. 46 can be obtained.

  The near concept word number difference sorting process (S4907) and the near concept word number difference table output process (S4908) are the same as those in the ninth embodiment.

  According to the present embodiment, the scale of the concept unit is compared between the two concept dictionaries and the feature word based on the change in the concept scale is extracted by a procedure different from that of the ninth embodiment. be able to.

Embodiment 11 FIG.
While obtaining the difference in the number of near-concept words, as described in the fourth to sixth embodiments, a mode for obtaining the increase type and the decrease type related to the difference will be described.

  A description will be given based on the ninth embodiment. Subsequent to the processing of S4405 shown in FIG. 44, processing for determining an increase type and a decrease type is added.

  FIG. 52 is a diagram showing a processing flow for comparing the number of close concept words according to the eleventh embodiment. First, the sign of the difference in the number of near concept words stored in S4405 is determined (S5201). If the difference is 0, it is not necessary to determine the type, and the process proceeds to S4406.

  When the difference in the number of near concept words is positive, the number of near concept words has increased, so the increase type is determined. For this purpose, it is determined whether or not the number of first near-concept words is 0 (S5202). If the number of near concept words with respect to the reference word in the first near concept word number table is 0, “occurrence” is stored in the near concept word number difference table as an increase type in association with the reference word (S5203). When the number of close concept words for the reference word is other than 0 (greater than 0) in the first close concept word number table, “add” is added as an increase type in association with the reference word, and the difference in the number of close concept words It is stored in the table (S5204).

  If the difference in the number of near-concept words is negative, the number of near-concept words has decreased, so the decrease type is determined. Therefore, it is determined whether or not the number of second near-concept words is 0 (S5205). If the number of near concept words with respect to the reference word in the second near concept word number table is 0, “disappear” is stored in the near concept word number difference table as a decrease type in association with the reference word (S5206). In the second near concept word number table, if the number of near concept words for the reference word is other than 0 (greater than 0), “remaining” is set as the decrease type in association with the reference word, and the difference in the number of near concept words It is stored in the table (S5207).

  53 shows an example of a first near concept word number table according to Embodiment 11, and FIG. 54 shows an example of a second near concept word number table according to Embodiment 11. Based on these assumptions, a near concept word number difference table can be obtained as shown in FIG. FIG. 55 is a diagram showing an example of a near concept word number difference table according to Embodiment 11.

  According to the present embodiment, when extracting feature words based on a change in concept scale, it is possible to pay particular attention to the occurrence and disappearance of concepts.

Embodiment 12 FIG.
Based on the tenth embodiment, the type of increase and the type of decrease related to the difference can be obtained.

  In the present embodiment, the processing from S5201 to S5207 in FIG. 52 is performed between S5005 and S5006 in FIG.

  Taking Embodiment 1 as an example, whether or not the number of first near-concept words is 0 (S5202) is determined based on whether there is a near-concept word for the reference word in the first near-concept word list. . Further, whether or not the second number of close concept words is 0 is determined (S5205) based on whether there is a close concept word for the reference word in the second close concept word list.

  Taking Embodiment 2 as an example, the determination as to whether or not the number of first near-concept words is 0 (S5202) is a comparison word whose concept difference is smaller than a predetermined reference in the first concept difference table. Judge whether there is any. Further, whether or not the number of second near concept words is 0 is determined (S5205) based on whether or not there is a comparison target word whose concept difference is smaller than a predetermined reference in the second concept difference table. To do.

  According to the present embodiment, similar to the eleventh embodiment, when extracting feature words based on a change in the concept scale, it is possible to pay particular attention to the occurrence and disappearance of concepts.

Embodiment 13 FIG.
In the present invention, based on a plurality of concept dictionaries derived from different document data groups, it is possible to detect a difference or change in concept for words occurring between document data groups. In the following embodiment, an example of document data group stratification will be described.

  For example, if questionnaires are stratified into respondents' age-specific document data groups, concept dictionaries are generated from the respective document data groups, and the present invention is applied to these concept dictionaries, the differences in concepts according to age are reflected. Feature words can be extracted. Similarly, when using a document data group stratified according to gender, occupation, residential area, or the like, it is possible to extract feature words that reflect differences in concepts among the tiers.

Embodiment 14 FIG.
As an example of stratification, if a questionnaire is divided into document data groups for each period, a concept dictionary is generated from each document data group, and the present invention is applied to these concept dictionaries, the concept over time It is possible to extract feature words that reflect changes in The change in concept with the passage of time affects the fashion at that time, and can be used commercially, such as marketing performed by producers, distributors, retailers, and the like. It may also be used in cultural or academic fields, such as public awareness surveys.

Embodiment 15 FIG.
In addition, a concept dictionary for some layers is generated from the layered document data group, and similarly, a concept dictionary for all layers is generated using all the layered document data groups. When the present invention is applied to the concept dictionary and the concept dictionary related to all layers, it is possible to extract feature words reflecting the characteristics of some layers with respect to the whole.

Embodiment 16 FIG.
If text mining is performed using the extracted feature word as a search axis according to the present invention, a more effective analysis result can be obtained.

  Currently, several types of products related to text mining are sold. In any of the products, a word serving as a search axis must be input by the user based on experience. On the other hand, if the present invention is used, it is possible to find the initial stage of the epidemic using the material (words serving as search axes) that the user has overlooked.

  The above-described feature word extraction system is a computer, and each element can execute processing by a program. Further, the program can be stored in a storage medium so that the computer can read the program from the storage medium.

  FIG. 56 is a diagram illustrating a hardware configuration example of the feature word extraction system. An arithmetic device 5601, a data storage device 5602, and a memory 5603 are connected to the bus. The data storage device 5602 is, for example, a ROM (Read Only Memory) or a hard disk. The memory 5603 is a normal RAM (Random Access Memory).

  The program is normally stored in the data storage device 5602, and is loaded into the memory 5603 and sequentially read into the arithmetic device 5601 for processing.

  101 first document data group storage unit, 102 co-occurrence frequency calculation unit, 103 first co-occurrence frequency table storage unit, 104 singular value decomposition unit, 105 first concept dictionary storage unit, 106 second document data group storage unit, 107 Second co-occurrence frequency table storage unit, 108 Second concept dictionary storage unit, 109 Concept dictionary difference extraction unit, 110 Concept dictionary difference information storage unit, 111 Concept dictionary difference information output unit, 701 Near concept word extraction unit, 702 First Near Concept Word List Storage Unit, 703 Second Near Concept Word List Storage Unit, 704 Near Concept Word Comparison Unit, 1601 Concept Difference Calculation Unit, 1602 First Concept Difference Table Storage Unit, 1603 Second Concept Difference Table Storage Unit, 1604 Concept Difference comparison unit 3301 Document data group storage unit, 3302 Co-occurrence frequency table storage unit, 3303 Concept dictionary storage unit, 3304 Near concept word number calculation unit 3305 Near concept word number table storage unit, 3306 Near concept word number sort unit, 3307 Near concept word number table output unit, 3501 Near concept word list storage unit, 3502 Near concept word number count unit, 4201 First near concept word number table Storage unit 4202 Second conceptual word number table storage unit 4203 Near conceptual word number comparison unit 4204 Near conceptual word number difference table storage unit 4205 Near conceptual word number difference sorting unit 4206 Near conceptual word number difference table output unit , 4801 Near concept word number difference calculation section.

Claims (3)

( 1) The frequency at which the combined words appear in the predetermined range group included in the document data group is calculated as a co-occurrence frequency related to the combination of the words, and each of the common words related to the combination of the plurality of words is calculated. A co-occurrence frequency calculation unit that generates a co-occurrence frequency table that stores the occurrence frequency in a matrix format ;
(2) Singular value decomposition of the co-occurrence frequency table, and a singular value decomposition unit that calculates a concept dictionary composed of concept vectors for each word in the reduced dimension semantic space as a degenerate matrix ;
(3) Based on the concept vector for each word in the concept dictionary, a difference in concept indicating the proximity of the direction of the concept vector of the comparison target word with respect to the concept vector of the reference word is calculated, and the concept of the comparison target word with respect to the reference word The number of comparison target words with a difference of less than a predetermined reference is counted, the number of comparison target words counted is set as the number of near concept words with respect to the reference word, and the number of near concept words for each reference word is stored. A feature word extraction system comprising a near concept word number calculation unit for generating a number table. (4) The feature word extraction system according to claim 1, further comprising: (4) a near concept word number sorting unit that sorts the near concept word number table using the number of near concept words as a key. A program for causing a computer as a feature word extraction system to execute the following procedure: (1) The frequency with which the combined words appear in the predetermined range group included in the document data group is set as the combination of the words. Co-occurrence frequency calculation processing procedure for generating a co-occurrence frequency table for calculating the co-occurrence frequency and storing each co-occurrence frequency related to a combination of a plurality of words in a matrix format (2) Singular value decomposition of the co-occurrence frequency table Then, a singular value decomposition processing procedure for calculating a concept dictionary composed of concept vectors for each word in the reduced dimension semantic space as a contraction matrix. (3) Based on the concept vector for each word in the concept dictionary, the concept of the reference word calculates the difference between the concepts indicating the closeness of the direction of the concept vectors being compared word for the vector, the difference between the concept of comparing the target word for the reference words in a predetermined By counting the number of comparison target word is smaller than the standards, the number of comparison target words counted by the near concept word count to the reference word, producing near concepts number of words table for storing the near concept number words each reference word Near conceptual word count calculation processing procedure.

Images