JP2004246491A - Text mining system and text mining program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform an analysis based on a new reference not defined in an original document. <P>SOLUTION: This text mining device is constituted so as to store index information for retrieving/mining an object document group 101 in an index storage part 103. A retrieval/mining execution part 110 executes the retrieval/mining processing of the object document group 101 in reference to the index information according to a retrieval/mining condition. A retrieval/mining result storage part 11 stores the retrieval/mining result in a retrieval/mining result storage part 107 with the relevance ratio of each document with the retrieval/mining condition attached to each document. Further, a retrieval/mining result editing part 112 reads a plurality of designated retrieval/mining results from the storage part 107, and selects a read retrieval/mining result for every document in conformation to a designated attribute name to determine the attribute value. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a text mining apparatus and a text mining program that can analyze a set of a large number of digitized documents from various viewpoints.
[0002]
[Prior art]
Text mining has attracted attention as a technique for supporting the analysis of a large amount of document data such as questionnaires and electronic news. Text mining is a technique for extracting frequency and relevance of words from text information of document data to be processed to discover new knowledge. Conventionally, a method of supporting interactive analysis of a large amount of document data has been proposed. For example, in a text mining device disclosed in Patent Literature 1, the following processing is performed.
[0003]
A phrase (characteristic phrase) characteristically appearing in the target document set is extracted from a target document set to be subjected to text mining processing, and a phrase co-occurring with a component of the analysis axis specified by the user is extracted from the extracted words. For example, when a newspaper article set regarding “O157” is to be processed, the characteristic phrase “elementary school, mass infection, patients, plural, bleeding, diarrhea, symptoms, hospitalization,...” Is extracted. From among them, the publication month of the newspaper is designated as the analysis axis, and words co-occurring with the constituent elements (July, August, September) are acquired. As a result, in association with “July”, “infection, patient, symptom, hospitalization,...”, And in association with “August”, “impact, lunch, hospitalization, mass infection,. In addition, co-occurrence phrases such as “sales, minus, food, fresh,...” Are acquired.
[0004]
The analysis axis and the analysis result are stored as an analysis history, and a plurality of different analysis axes are provided. When an analysis axis is added or an arbitrary analysis axis is changed, the constituent elements of the plurality of analysis axes are changed. Are narrowed down by using the analysis history, and the words and phrases that are likely to co-occur with each are realized by the combination of different analysis axes.
[0005]
When adding an analysis axis that includes a specified characteristic phrase, the analysis axis is added to the specified component, and a set of the specified component and the added analysis axis component is created. A phrase that is highly likely to co-occur with each of the created set of constituent elements is extracted from the analysis history as a co-occurrence phrase candidate. Then, from the extracted co-occurrence word candidates, each set of constituent elements is co-existed within a predetermined range (in the same document, in the same paragraph, in the same sentence, or in m words or n sentences). Get the words that occur. For example, an analysis axis including the co-occurrence phrases “infection” and “symptom” acquired in association with the component “July” is added.
[0006]
Then, as a co-occurrence phrase candidate having a high possibility of co-occurring with the specified component “July” and the added component of the analysis axis “July-infection” and “July-symptom”, “7 The phrase "infection, patient, symptom, hospitalization, ..." co-occurring with "month" is extracted from the analysis history. Then, from these co-occurrence word candidates, words that co-occur within a predetermined range as a set of component elements “July-infected” and “July-symptom” are acquired. As a result, as a phrase co-occurring with "July-infection", "patient, symptom, prophylaxis, population, ..." and as a phrase co-occurring with "July-symptom", "nausea, diarrhea, hospitalization, severe illness, ... To get. Here, “July-infected” means a set of “July” and “infected”, and “co-occurs within a predetermined range with“ July-infected ”” means , "Co-occurs within a predetermined range with" July "and co-occurs with" infection "within a predetermined range."
[0007]
When changing the added analysis axis, the analysis axis is changed in accordance with the user's instruction, and a phrase that is highly likely to co-occur with the set of components is extracted from the analysis history as a co-occurrence word candidate. Then, from the extracted co-occurrence word candidates, words that co-occur within a predetermined range with the set of constituent elements are acquired. If "Analysis axis was added to component" July "" is deleted and analysis axis containing co-occurrence phrase "feeding" acquired in association with "August" is added, specified component As a co-occurrence word candidate having a high possibility of co-occurring with the set of components of the analysis axis “August” and “August-lunch”, the phrase “shock, lunch, hospitalization, Mass infection, ... "is extracted from the analysis results. Then, from the co-occurrence word candidates, a word that co-occurs within a predetermined range with a set of constituent elements “August-meal” is acquired.
[0008]
When the representative document acquisition instruction is input from the user together with the co-occurrence word specified from the text mining result, the text mining target is specified by the specified co-occurrence word and the phrase included in the component corresponding to the co-occurrence word. Then, a document set having a high score, the latest document, a document having designated bibliographic information, and the like are acquired as a representative document.
[0009]
As described above, in the text mining apparatus of Patent Document 1, the analysis axis and the analysis result are stored as an analysis history, and a plurality of different analysis axes are provided. When the analysis is performed, words having a high possibility of being co-occurring with each of the components of the plurality of analysis axes are narrowed down using the analysis history, and the analysis using a combination of a plurality of different analysis axes is realized. In addition, since the co-occurrence words are narrowed down using the analysis history, the words that actually co-occur with each of the constituent elements are examined. I can do it.
[0010]
[Patent Document 1]
JP 2001-318939A (Page 14, FIG. 4)
[0011]
[Problems to be solved by the invention]
Since the conventional text mining device has been configured as described above, when performing the analysis work interactively, the analysis is always performed in a narrowing direction, and the stored analysis result is further narrowed down under another condition. It was only used as intermediate information. Therefore, there is a problem in that the user cannot create a new analysis standard by associating a plurality of stored analysis results and perform analysis based on the standard.
[0012]
For example, a new analysis axis (attribute) of “social impact” is created for the analysis results obtained by the characteristic phrases “lunch”, “sales”, and “food”, and “prevention”, “prevention”, A new analysis axis (attribute) called "measures" is created for the analysis results obtained by the characteristic words "cooking" and "information disclosure". Processing that analyzes the relationship between attributes cannot be performed.
[0013]
The present invention has been made in order to solve the above-described problems. By associating stored analysis results with each other, a new analysis target and a new attribute can be created, and the original document can be created. It is an object to obtain a text mining device and a text mining program that can perform analysis based on a criterion (analysis axis) defined by an undefined user.
[0014]
That is, in Patent Literature 1, as shown in FIG. 33, an analysis process 404 for performing segmentation is repeatedly performed on an analysis result 403 output by performing an analysis process 402 on a target document set 401. On the other hand, in the present invention, as shown in FIG. 34, the analysis processing 412 is performed on the target document set 411 to output a plurality of analysis results 413 and 414, and the analysis processing 415 is associated with the respective analysis results 413 and 414. To obtain a new analysis result 416. As a result, according to the present invention, analysis across a plurality of analysis axes defined by the user can be performed.
[0015]
[Means for Solving the Problems]
A text mining device according to the present invention includes an index storage unit storing index information for searching and mining a target document set, and index information stored in the index storage unit according to a specified search / mining condition. And a search / mining execution unit for executing the search / mining process of the target document set with reference to the search / mining results by the search / mining execution unit. A search / mining result storage unit that is assigned to each document and saved in the search / mining result storage unit, and a plurality of specified search / mining results are read from the search / mining result storage unit and assigned to the specified attribute name Correspondingly, a search / mining result editor that selects the search / mining result read for each document and determines the attribute value It includes those were.
[0016]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, an embodiment of the present invention will be described.
Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of a text mining device according to Embodiment 1 of the present invention. From the target document set 101, index information for searching and mining the target document set 101 is generated by the index generation unit 102 and stored in the index storage unit 103. The execution unit 104 reads the index information in the index storage unit 103, executes a search or mining process according to the search condition or the mining condition input from the input unit 105, and outputs a search / mining result to the display unit 106. When the user receives an instruction to save the search / mining result, the execution unit 104 saves the search / mining result in the search / mining result storage unit 107.
[0017]
In the execution unit 104, the index reading unit 108 reads the index information stored in the index storage unit 103. The search / mining result reading unit 109 reads the search / mining result stored in the search / mining result storage unit 107. The search / mining execution unit 110 refers to the index information stored in the index storage unit 103 according to the specified search / mining conditions, and searches / mines the target document set 101 or the search / mining result reading unit 109. Execute the search / mining process of the result.
[0018]
In the execution unit 104, the search / mining result storage unit 111 assigns the degree of relevance between each document and the search / mining condition to each document based on the search / mining result obtained by the search / mining execution unit 110, and performs search / mining. The result is stored in the result storage unit 107. The search / mining result editing unit 112 reads a plurality of specified search / mining results from the search / mining result storage unit 107 and stores the read search / mining results for each document in accordance with the specified attribute name. To determine the attribute value.
[0019]
FIG. 2 is a detailed block diagram showing a configuration of the text mining apparatus in which the index generation unit 102, the index storage unit 103, and the search / mining execution unit 110 in FIG. 1 are detailed. The index generation unit 102 includes a concept dictionary generation unit 201, a document vector index generation unit 202, and a mining index generation unit 203. The concept dictionary generation unit 201 and the mining index generation unit 203 generate a concept dictionary 204 and a mining index 206 from the target document set 101, respectively, and the document vector index generation unit 202 refers to the concept dictionary 204 to generate a document from the target document set 101. Generate a vector index 205. The search / mining execution unit 110 includes a search unit 207 that executes a search process and a mining unit 208 that executes a mining process.
[0020]
FIG. 3 is a diagram illustrating an example of documents in the target document set 101. The first embodiment will be described based on an example of analyzing a questionnaire on a mobile phone. The target document set 101 has a predetermined range of values such as “sex”, “age”, “model”, “region”, and “date”. It is composed of a field described in free text such as "Opinion". The former field is hereinafter referred to as an attribute in the document. The item name of the attribute is called an attribute name, and its value is called an attribute value. For example, the documents in the target document set 101 shown in FIG. 3 have an attribute value of “male” for an attribute name of “sex”.
[0021]
In the example shown in FIG. 3, one attribute value is assigned to one attribute name. However, when there is no answer in a questionnaire, the attribute value does not necessarily have to be assigned to the attribute name. Further, when there is a case where the respondent specifies two or more such as “model”, two or more attribute values may be given to one attribute name.
[0022]
FIG. 4 is a flowchart showing the flow of the process of the concept dictionary generator 201 shown in FIG. 2, and FIG. 5 is a diagram for explaining the process of the concept dictionary generator 201.
In step ST11 in FIG. 4, the concept dictionary generation unit 201 morphologically analyzes the text included in the document set 301 in FIG. 5 to divide a character string in the text into words. Since the morphological analysis is a widely known technique, a detailed description is omitted here. At this time, the learning target document set 301 shown in FIG. 5 is not necessarily the target document set 101 itself, but may be another document set in the same field as the target document set 101.
[0023]
In step ST12, the concept dictionary generation unit 201 extracts a compound word from the morphological analysis result.
FIG. 6 is a flowchart showing the flow of the compound word extraction process of the concept dictionary generation unit 201. In step ST21 of FIG. 6, the concept dictionary generation unit 201 extracts compound word candidates based on a morpheme concatenation pattern described in a compound word candidate extraction dictionary stored therein.
[0024]
FIG. 7 is a diagram showing an example of a compound word candidate extraction dictionary describing a morpheme concatenation pattern for extracting compound word candidates. FIG. 7 shows a compound word candidate extraction dictionary for extracting compound word candidates from the concatenation relationship between two morphemes. An example is shown. For example, pattern number 001 indicates that "when two morphemes whose nouns are part of speech continue, the arrangement of the morphemes is extracted as a compound word candidate." As a result, a character string in which two nouns are consecutive in the text, such as “model / change” and “confirmation / screen”, is extracted as a compound word candidate.
[0025]
Similarly, from the pattern number 002, a character string in which a morpheme having a noun as a part of speech and a morpheme having a suffix as a suffix, such as “mobile / sex” and “function / target”, is extracted as a compound word candidate. From the number 003, character strings such as “season / feeling” and “use / feeling” in which a morpheme having a part of speech as a noun and a morpheme having a notation of “feeling” are respectively extracted as compound word candidates. Note that “/” is used as a symbol indicating a morpheme delimiter.
FIG. 8 is a diagram illustrating a result of extracting compound word candidates from the target document set 101, focusing on a character string “model change”.
[0026]
In step ST22 of FIG. 6, from the compound word candidates extracted in step ST21, the concept dictionary generation unit 201 selects a compound word to be registered in the co-occurrence frequency table using the statistical information. The co-occurrence frequency table is a table showing the co-occurrence relationship of words as described later. Since the number of compound word candidates extracted in step ST21 is enormous, it is registered in the co-occurrence frequency table by narrowing down important words using statistical information of words. As the statistical information, statistical information calculated by a known method such as an appearance frequency or a tf * idf value calculated by the following equation is used.
tf * idf (w) = f (w) * log (Nd / d (w))
Here, f (w) indicates the frequency of appearance of the word w, Nd indicates the number of documents in the target document set, and d (w) indicates the number of documents in which the word w appears.
[0027]
In step ST22, a compound word candidate whose statistical information value such as an appearance frequency value or a tf * idf value is equal to or greater than a preset threshold is selected as a compound word to be registered in the co-occurrence frequency table. Alternatively, the number of compound words to be extracted may be determined in advance, and compound word candidates having a high statistical information value may be selected from the top. In the first embodiment, a description has been given by taking an example of a continuation of two words, but the present invention is not limited to two words, and a concatenation relationship of three or more words is described in a compound word candidate extraction dictionary, and compound word extraction is performed. Processing may be performed.
[0028]
As described above, the concept dictionary generation unit 201 extracts a compound word, so that a word having a higher degree of specialty is extracted from a word obtained by combining general words, and a concept index for the compound word is generated by processing described later. Therefore, a mining process based on a more appropriate word can be performed, and the accuracy of the mining process can be improved.
[0029]
Next, in step ST13 of FIG. 4, the concept dictionary generation unit 201 calculates a co-occurrence frequency which is the number of times a word and a word appear simultaneously in each document of the document set 301, and generates a co-occurrence frequency table 302 shown in FIG. Ask for. The co-occurrence frequency table 302 is a table showing how many times the word on the vertical axis co-occurs with the word on the horizontal axis.
[0030]
In step ST14, the concept dictionary generation unit 201 performs singular value decomposition of the co-occurrence frequency table 302. The singular value decomposition is a known linear algebra technique for decomposing a matrix A, here, a co-occurrence frequency table 302 shown in FIG. 5 into a product of three matrices (UΣV) 303, 304, and 305. For example, a document search method using a concept dictionary created by using singular value decomposition is described in Document 3 (“Information search system InfoMAP based on word associations, Takayama et al., Informatics Basics 53-1, 1999-3”). There is a description about. Note that eigenvalue decomposition may be used instead of singular value decomposition.
[0031]
In step ST15, the concept dictionary generation unit 201 designates a matrix U (left singular matrix) 303 obtained by performing singular value decomposition in step ST14, and designates a matrix Σ (singular value matrix) 304 from the largest singular value included in the matrix. k columns (k is smaller than the number of columns of the original matrix A) are output as the concept dictionary 204. The horizontal axis of the concept dictionary 204 indicates k horizontal axis components of the matrix U. The concept dictionary 204 is more dimensionally compressed than the co-occurrence frequency table 302, and each row can be regarded as a concept vector of a word including a higher-order correlation. Hereinafter, the concept vector of a word is referred to as a word vector. Here, the concept vector is a vector in which the left k dimension of the matrix U generated as a result of performing the singular value decomposition of the co-occurrence frequency table 302 is a component value.
[0032]
The concept dictionary 204 created in this way has a feature that words co-occurring with the same word have similar word vectors, that is, words having similar concepts have similar word vectors. .
[0033]
Next, the processing of the document vector index generation unit 202 will be described.
FIG. 9 is a diagram illustrating the generation of a document vector index by the document vector index generation unit 202. The document vector index generation unit 202 generates a document vector index 205 from the target document set 101 with reference to the concept dictionary 204.
[0034]
FIG. 10 is a flowchart showing the flow of processing of the document vector index generation unit 202. In step ST31, the document vector index generation unit 202 morphologically analyzes each document in the document set 101 and divides the text in the document into words. In step ST32, the document vector index generation unit 202 calculates the frequency of each word that appears in each document. In step ST33, the document vector index generation unit 202 extracts a word vector for each word from the concept dictionary 204.
[0035]
In step ST34, the document vector index generation unit 202 outputs, as a document vector index 205 of the document, a value obtained by adding a vector obtained by multiplying the word vector of the word appearing in each document by using the frequency calculated in step ST32 as a coefficient. I do. Note that a vector corresponding to each document stored in the document vector index 205 is called a document vector.
[0036]
FIG. 11 is a diagram showing the document vector index 205, which has a structure in which each document is associated with a document vector. The document vector has the same dimension as the word vector in the concept dictionary 204, and documents having similar document vectors have a feature that they have similar contents.
[0037]
Next, the processing of the mining index generation unit 203 will be described.
FIG. 12 is a flowchart showing the flow of processing of the mining index generation unit 203. In step ST41, the mining index generation unit 203 acquires a list of attribute names included in the target document set 101. In the example shown in FIG. 3, “sex”, “age”, “model”, “region”, and “date” are acquired as attribute names. In step ST42, the mining index generation unit 203 acquires, for each document in the target document set 101, an attribute value corresponding to the attribute name acquired in step ST41. For example, from the document shown in FIG. 3, “sex” is “male”, “age” is “20s”, “model” is “model 1”, and “region” is “model 1”. “2002-01-14” is obtained for “Tokyo” and “Date”.
[0038]
The mining index generation unit 203 obtains attribute values for all documents in the target document set 101, and writes the obtained attribute names and attribute values to the mining index 206 in step ST43.
FIG. 13 is a diagram illustrating an example of a mining index generated by the mining index generation unit 203.
[0039]
Next, the processing of the execution unit 104 will be described.
FIG. 14 is a flowchart illustrating the flow of the process of the execution unit 104. In step ST51, the index reading unit 108 reads, from the index storage unit 103, three pieces of index information of the concept dictionary 204, the document vector index 205, and the mining index 206.
[0040]
In step ST52, the search / mining result reading unit 109 reads the search / mining result specified by the user from the search / mining result storage unit 107. At first, no information is stored in the search / mining result storage unit 107. Is not written, and the flow passes without performing any processing in step ST52. In the first embodiment, the search / mining result indicates a document set stored in a format shown in FIG. 25 described later. The search / mining target at this time is the entire target document set 101 registered in the document vector index 205.
[0041]
In step ST53, the search / mining execution section 110 determines whether or not there is a search / mining condition input from the user. If there is a search / mining condition input, in step ST54, the search / mining condition is determined. Is received as an input, and in step ST55, a search / mining process is executed.
[0042]
Here, the search processing executed by the search unit 207 among the search and mining processing in step ST55 shown in FIG. 14 will be described.
FIG. 15 is a flowchart illustrating the flow of a search process performed by the search unit 207. The search unit 207 searches the target document set 101 using the document vector index 205. In step ST61, the search unit 207 inputs search conditions specified by the user from the input unit 105 as text. Here, for example, “model with a large screen” is input as a search condition.
[0043]
In step ST62, the search unit 207 generates a concept vector for the input search condition with reference to the concept dictionary 204. Here, a concept vector is a combination of word vectors for “screen”, “large”, and “model”. Hereinafter, the concept vector corresponding to the search condition is referred to as a search vector. The search vector generation processing in step ST62 is performed in the same manner as the processing in steps ST31 to ST34 in FIG.
[0044]
In step ST63, the search unit 207 calculates a cosine value between the search vector and a document vector corresponding to each document of the stored target document set 101, and sets the calculated cosine value as the similarity. In step ST64, the search unit 207 outputs the search results arranged in the order of similarity.
[0045]
FIG. 16 is a diagram illustrating an example of a search result obtained by the search unit 207. The search result is obtained when “large model” is input as a search condition, and a character string for identifying a document (document n in FIG. 16), The degree of similarity and the free description part of the document are displayed.
[0046]
It should be noted that the target document set 101 displayed as a search result may be provided with a threshold value for similarity, and only documents having the threshold value or more may be displayed. Alternatively, a maximum value of the number of cases to be displayed may be set in advance, and only the number of cases specified from the highest similarity may be displayed. Further, narrowing down by document attributes such as “sex” and “age” may be used together. Further, in the first embodiment, the similarity between vectors is calculated using the concept dictionary 204 and the document vector index 205. However, if the similarity between the input search condition and the document can be calculated, Other methods may be used.
[0047]
Next, the mining process executed by the mining unit 208 among the search / mining processes in step ST55 shown in FIG. 14 will be described. By performing this mining process, the degree of association between the keyword and the attribute can be analyzed.
FIG. 17 is a flowchart showing the flow of the mining process executed by the mining unit 208. In step ST71, the mining unit 208 inputs mining conditions from the input unit 105. Here, the input mining conditions are attributes and keywords. When "model" is specified as the attribute name, "model 1", "model 2", "model 3" and "model 4" are specified as the attribute values, and "screen" and "ringtone" are specified as keywords. Will be described.
[0048]
In step ST72, the mining unit 208 acquires the target document set 101 having the specified attribute value. That is, for the attribute name “model”, a target document set 101 including “model 1” in the attribute value, a target document set 101 including “model 2”, a target document set 103 including “model 3”, and “model” The target document set 101 including “4” is acquired. In step ST73, the mining unit 208 creates an attribute vector for each of the acquired target document sets 101. The attribute vector is a vector created by averaging the elements of the document vector of the target document set 101 having the same attribute value.
[0049]
In step ST74, the mining unit 208 obtains the degree of association between each attribute value and the keyword. The degree of association is calculated based on the attribute vector and the cosine value of the word vector extracted from the concept dictionary 204 corresponding to the keyword. In step ST75, the mining unit 208 outputs the obtained degree of association between the attribute and the keyword as a graph as a mining result.
[0050]
FIG. 18 is a diagram illustrating a mining result by the mining unit 208. The attribute value of the model is used as an axis, and the value of the degree of association between each attribute value and the keyword is represented using a line graph. From the graph of FIG. 18, it can be seen that the analysis result indicates that “model 3” has a low interest in the screen and high interest in the ringtone, and conversely, “model 4” has a high interest in the screen and low interest in the ringtone.
[0051]
The attribute may be specified by specifying only the attribute name, and the mining unit 208 may automatically extract all the attribute values corresponding to the attribute name from the mining index 206 and perform the analysis. Alternatively, when specifying an attribute value, a relationship of a product (AND), a sum (OR), or a negation (NOT) of each attribute value may be specified, such as model 1 or model 2.
[0052]
In addition, the keyword may specify not only one word such as “screen” and “ringtone” but also a plurality of words such as “screen, color” and “ringtone, chord”. When a plurality of words are specified for a keyword, the vector corresponding to the keyword is a vector created by averaging word vectors for words included in the keyword. When an attribute name such as “date” is specified, the target document set 101 is acquired in the attribute value range such as “from 2002-01-01 to 2002-01-31” in step ST72. , The relationship with the keyword may be analyzed.
[0053]
Next, an example in which the correlation between two attribute values in the mining processing executed by the mining unit 208 is analyzed by keyword distribution will be described.
FIG. 19 is a flowchart showing the flow of the mining process executed by the mining unit 208. In step ST81, the mining unit 208 inputs mining conditions from the input unit 105. Here, the mining conditions input by the user are two attribute values and a keyword. A case will be described in which the attribute values “male” and “female” of the attribute name “sex” are specified as the attribute values, and “flip”, “charge”, “ringtone”, and “character” are specified as the keywords.
[0054]
In step ST82, the mining unit 208 acquires the target document set 101 having the specified attribute value. That is, for the attribute name “sex”, the target document set 101 whose attribute value is “male” and the target document set 101 whose attribute value is “female” are acquired. In step ST83, the mining unit 208 creates an attribute vector for each of the acquired target document sets 101. The method of creating the attribute vector is the same as the process of step ST73 in FIG.
[0055]
In step ST84, the mining unit 208 obtains the degree of association between each attribute vector and the keyword input as the mining condition. The calculation of the degree of association is the same as the processing in step ST74 in FIG. In step ST85, the mining unit 208 displays the coordinates of each keyword using the attribute value as a coordinate axis and outputs the result as a mining result.
[0056]
FIG. 20 is a diagram illustrating a mining result by the mining unit 208. In this example, “male” is set on the X coordinate axis, “female” is set on the Y coordinate axis, and the relationship between the concept vector of each keyword and the attribute vector of “male” The degree is defined as the X coordinate, and the degree of association with the attribute vector of “female” is defined as the Y coordinate. From the graph of FIG. 20, it can be seen that "character" and "ringtone" have a high degree of association with a woman, and "flip" has a high degree of association with a man.
[0057]
The attribute may be specified by combining a plurality of attributes, such as “teens and genders”. In this case, in step ST82, a process of extracting the target document set 101 in which the attribute value of the attribute name “age” is “teens” and the attribute value of the attribute name “sex” is “female” is performed. Also, a keyword may be specified by combining a plurality of words. Further, the user may automatically specify a keyword from the target document set 101 using a dictionary or statistical information without specifying a keyword, and input the extracted keyword as a mining condition.
[0058]
Next, among mining processes executed by the mining unit 208, an example will be described in which the correlation between keywords is analyzed based on the distribution of documents.
FIG. 21 is a flowchart showing the flow of the mining process executed by the mining unit 208. In step ST91, the mining unit 208 inputs mining conditions from the input unit 105. Here, the mining conditions input by the user are keywords. A case where “melody” and “song” are designated as keywords will be described.
[0059]
In step ST92, the mining unit 208 obtains the degree of association between the word vector extracted from the concept dictionary 204 for each keyword and the document vector corresponding to each document in the target document set 101 by the cosine value of the vector. In step ST93, the mining unit 208 displays the coordinates of each document using the keyword as a coordinate axis and outputs the result as a mining result.
[0060]
FIG. 22 is a diagram illustrating a mining result by the mining unit 208. In this example, the relation between the document vector of each document and the concept vector of “melody” is indicated by “melody” on the X coordinate axis and “tune” on the Y coordinate axis. A point where the degree is the X coordinate and the degree of association with the "song" is the Y coordinate is displayed on the graph. From the graph of FIG. 22, it can be read that a document having a high correlation with “melody” also has a high correlation with “song”, and thus “melody” and “song” are words having a high relation.
[0061]
FIG. 23 is a diagram similarly showing a mining result by the mining unit 208. In this example, the mining result when the processing of FIG. 21 is performed by specifying “sound” and “color” as keywords is shown. Then, it can be read that "sound" and "color" are words having low relation.
[0062]
In the first embodiment, the relevance between a word and an attribute and the relevance between a word and a document are calculated by using the concept dictionary 204 and the document vector index 205. In the concept dictionary 204, for example, "screen" and "liquid crystal" co-occur with the same word such as "size", "brightness", and "color", so that the distance between word vectors of words that appear in similar contexts Becomes closer. Therefore, when calculating the degree of association between a document and the word “screen”, if the document does not include the word “screen”, but includes the word “liquid crystal”, the document and the “screen” Is higher. In this way, even when the word notation is different, mining that determines the degree of relevance based on closeness in meaning can be realized.
[0063]
Returning to the flowchart of FIG. 14, the search / mining execution section 110 displays the search / mining result executed in step ST55 in step ST56. In step ST57, the search / mining result storage unit 111 checks whether the user has issued an instruction to save the mining result. If there is no instruction, the process returns to step ST53, and the search / mining execution unit 110 receives the next search / mining condition and executes a search / mining process. If there is no next search / mining condition, the process ends.
[0064]
In step ST57, when the search / mining result storage unit 111 receives an instruction to save the search / mining result from the user, in step ST58, the search / mining result storage unit 111 searches / mines the search / mining result. The result is stored in the result storage unit 107.
[0065]
FIG. 24 is a flowchart showing a processing flow of the search / mining result storage unit 111 for storing the search / mining result. In step ST101, a document set to be saved as a search / mining result is determined by the user. For example, in the case of a search result, the search result output in step ST64 of FIG. 15 may be used as a set of documents to be stored as it is, or a threshold of similarity, a condition such as the number of searches, and the like may be set from the search result. Alternatively, a set of documents to be stored may be narrowed down.
[0066]
In the case of the mining process shown in FIGS. 17 and 18 for analyzing the degree of association between an attribute and a keyword, the attribute and the keyword are designated, and a document set having a specific attribute is associated with a specific keyword. A document set with a high degree may be used. For example, a document set whose attribute value is “model 4” and has a high degree of association with the keyword “screen” may be set as a document set to be stored.
[0067]
Furthermore, in the example of the mining process for analyzing the correlation between the attributes described in FIGS. 19 and 20, the attribute and the keyword are designated, and the specific keyword and the relevance of the relevance are set in the document set having the specific attribute. A high document set may be used. For example, a document set whose attribute value of the attribute name “sex” is “female” and has a high keyword “character” may be set as a document set to be stored.
[0068]
Further, in the example of the mining process for analyzing the correlation between keywords described with reference to FIGS. 21, 22, and 23, a document set having a high relation with the keyword specified as the coordinate axis may be set as a document set to be stored. For example, a document set to be stored may be a document having a high degree of relevance to one or both of the keyword “melody” and the keyword “song”.
[0069]
Next, in step ST102 in FIG. 24, the search / mining result storage unit 111 explicitly assigns a name to the search / mining result so that it can be referred to later, in accordance with a user's instruction. In step ST103, the search / mining result storage unit 111 assigns a score to each document to be stored and stores it.
[0070]
The score is the degree of association between the document and the keyword specified in the search or mining. For example, in the case of the search processing shown in FIG. 15, the score is the similarity (relevance) with the search condition specified in step ST61.
[0071]
In the case of the mining process of analyzing the relevance between the attribute and the keyword in FIGS. 17 and 18, the relevance with the keyword specified by the mining condition is used as the score. For example, if the attribute value of the attribute name “model” is “model 4” and a document set having a high degree of relevance to the keyword “screen” is a document set to be saved, the degree of relevance to “screen” is defined as a score. I do.
[0072]
Further, in the example of the mining process for analyzing the correlation between the attributes described in FIGS. 19 and 20, the degree of relevance with the keyword specified by the mining condition is used as the score. For example, when the attribute value of the attribute name “sex” is “female” and a set of documents having a high keyword “character” is set as a document set to be stored, the degree of association with “character” is set as the score.
[0073]
Further, in the example of the mining process for analyzing the correlation between the keywords described in FIGS. 21, 22, and 23, the degree of relevance with the keyword specified on the coordinate axis is used as the score. For example, the degree of relevance between each document and one of the keyword “melody” and the keyword “song” or the degree of relevance to a vector obtained by combining both word vectors is used as a score.
[0074]
FIG. 25 is a diagram illustrating an example of a search / mining result stored in the search / mining result storage unit 107. For example, a search / mining result named “fashion” by the user is obtained by specifying a keyword such as “design, character, color, etc.”, and a search named “operability” by the user is given. -The mining result is obtained from the search result by specifying keywords such as "key, button, operation ...", and the search named "function" and the user-The mining result is "kanji conversion, mail, The search / mining results named by the user as "business use" can be obtained by specifying keywords such as "boss, business partner, report ...". The search / mining result named "Private Use" by the user is assumed to be obtained from the search result by specifying keywords such as "Mel friend, child, travel ...". .
[0075]
Next, returning to step ST52 after the processing of step ST58 in FIG. 14, the search / mining result reading unit 109 reads the search / mining result stored in the search / mining result storage unit 107 specified by the user. In the designated search / mining result, a document set to which a score is given is set as the next search / mining target document set.
[0076]
For example, when “fashion” is specified in the search / mining result of FIG. 25, documents 1, document 3, document 5,... Are a set of documents to be searched / mined. In addition, a new set of documents to be searched and mined can be defined by specifying AND, OR, and NOT relationships for a plurality of search and mining results.
[0077]
For example, in the processing of FIG. 24, by specifying the search / mining result named “fashion” and the search / mining result named “private use” by AND, “fashion” and “private use” are designated. Are defined as a document set to be searched / mined. In addition, by specifying the search / mining result named “operability” and the search / mining result named “function” by OR, a score is assigned to either “operability” or “function”. The set of documents being searched becomes a set of documents to be searched and mined.
[0078]
In this way, by specifying the combination of the search and mining results, for example, the combination of the AND of “fashion” and “private use” becomes the “set of questionnaire results of those who are interested in fashion in private use”. A set of documents to be analyzed with such a meaning can be created, and the combination of OR of “operability” and “function” has a meaning like “set of questionnaire results of people who are interested in practicality”. A set of documents to be analyzed can be created.
[0079]
Thereafter, when a search / mining condition is input in step ST53 for the document set created by the search / mining result specified in step ST52 of FIG. 14, the search / mining condition is changed in step ST54. The search and mining process is executed in step ST55, and the result is displayed in step ST56.
[0080]
For example, when a mining process for analyzing the degree of association between an attribute and a keyword described in FIGS. 17 and 18 is performed on a document set in which “private use” and “fashion” are combined by AND, By performing “model” and “screen” and “ringtone” as keywords, “people who are interested in fashion in private use” can analyze for each model how much they are interested in screens and ringtones. .
[0081]
Next, the processing of the search / mining result editing unit 112 will be described.
FIG. 26 is a flowchart showing the flow of processing of the search / mining result editing unit 112. In step ST111, the search / mining result editing unit 112 receives the search / mining result of the target specified by the user and the attribute name to be given. For example, it is assumed that “fashion”, “operability”, and “function” in FIG. 25 are specified as the target search / mining result, and “point to be emphasized” is specified as the attribute name to be given. In step ST112, the search / mining result editing unit 112 reads the search / mining result specified by the user from the search / mining result storage unit 107.
[0082]
In step ST113, the search / mining result editing unit 112 selects a search / mining result having the highest score for each document. For example, regarding the document 1 in FIG. 25, “fashion” has the highest score among “fashion”, “operability”, and “function”. Is selected. Similarly, “function” is set for document 2, “operability” is set for document 3, “fashion” is set for document 5, and “function” is set for document 6. Selected. Since no score is assigned to any of the search and mining results for the document 4, no search or mining result is selected.
[0083]
In step ST114, the search / mining result editing unit 112 assigns the name of the search / mining result selected in step ST113 to each document as an attribute value for the attribute name specified in step ST111.
[0084]
FIG. 27 is a diagram showing an editing result by the search / mining result editing unit 112. As a result of the process of step ST114 in FIG. 26, an attribute indicated by “attribute name: important point” in FIG. 27 is generated. Regarding the document 4, since no search / mining result has been selected in step ST113, no attribute value is defined.
[0085]
Similarly, in the process of FIG. 26, when “business use” and “private use” are specified as search / mining results and “use field” is specified as an attribute name, “attribute name: use field” in FIG. 27 is used. Is generated.
[0086]
FIG. 28 is a flowchart showing a flow of processing of another implementation method of the search / mining result editing unit 112. FIG. 26 shows an example of the process of assigning one attribute value to one document, but FIG. 28 shows an example of the process of granting two or more attribute values to one document. The processing of step ST121 and step ST122 is the same as the processing of step ST111 and step ST112 in FIG. 26, respectively. As an example, a case will be described in which “fashion”, “operability”, and “function” in FIG. 25 are specified as the search / mining result to be read, and “point to be emphasized” is specified as the attribute name to be given.
[0087]
In step ST123 of FIG. 28, the search / mining result editing unit 112 selects a search / mining result whose score is equal to or larger than a threshold for each document. For example, if the threshold is set to 0.5, “fashion” and “function” are selected for document 1 because “fashion” and “function” have values equal to or greater than the threshold. Similarly, “function” is selected for document 2, “operability” is selected for document 3, and “fashion” is selected for document 5.
[0088]
In step ST124, the search / mining result editing unit 112 assigns the name of the search / mining result selected in step ST123 as an attribute value for the attribute name specified in step ST121 to each document according to a user instruction.
[0089]
FIG. 29 is a diagram showing an editing result by the search / mining result editing unit 112. As a result of the process of step ST124 in FIG. 28, an attribute indicated by “attribute name: important point” in FIG. 29 is generated. Similarly, when “business field” and “private use” are specified as “attribute fields” as attribute names as search / mining results, an attribute represented by “attribute name: field of use” in FIG. 29 is generated.
[0090]
As described above, the attribute generated by the processing in FIG. 26 or FIG. 28 can be handled in the mining process in the mining unit 208 in the same manner as the attribute originally provided in the document. That is, in the example of the target document set 101 in FIG. 3, in addition to the attributes “gender”, “age”, “model”, “region”, and “date” originally provided, a new attribute “point to be emphasized” And "applications" can be used for analysis.
[0091]
For example, FIG. 30 is a diagram showing a mining result by the mining unit 208. The attribute value determined by the search / mining result editing unit 112 is used as an axis, and the degree of association between each attribute value and a keyword input as a mining condition is shown. Is represented using a line graph. That is, in the analysis of the degree of association between the attribute and the keyword shown in FIGS. 17 and 18, "attribute point" is designated for the "attribute name", and "work", "price", and "screen" are designated for the keyword. It is a figure which shows the example performed. "Work" has low relevance to "fashion" and high relevance to "function". "Price" indicates relevance to any of "fashion", "operability" and "function". Can be read that does not change much.
As described above, the mining unit 208 can analyze the degree of association between the designated mining condition and the attribute value determined by the search / mining result editing unit 112.
[0092]
Also, in the example in which the correlation between the attribute values of the document and the attribute values shown in FIGS. 19 and 20 is analyzed based on the distribution of the specified words, the mining unit 208 similarly performs the search / mining result editing unit 112. The correlation between attribute values determined by the above can be analyzed by the distribution of the specified mining conditions.
[0093]
Further, in the example in which the correlation between words and the words shown in FIGS. 21, 22 and 23 is analyzed based on the distribution of the documents, the mining unit 208 similarly searches and mines the correlation between words and words. The analysis can also be performed based on the distribution of the documents stored in the result storage unit 107.
[0094]
Further, it is also possible to analyze the degree of association between the attributes created by the search / mining result editing unit 112.
FIG. 31 is a flowchart showing, as an example of the mining process in the mining unit 208, the flow of the process of analyzing the degree of association between the attributes created by the search / mining result editing unit 112. An example in which the relationship between the "point to be emphasized" and the "use field" shown in FIG. 27 or FIG. 29 will be described.
[0095]
In step ST131, the mining unit 208 acquires a document set for each attribute value for the attribute name 1 specified by the user. When "point to be emphasized" is specified for the attribute name 1, a document set including "fashion" in the attribute value, a document set including "operability" in the attribute value, and a document set including "function" in the attribute value are obtained. You. In step ST132, the mining unit 208 creates an attribute vector from each document set acquired in step ST131. The attribute vector is created by averaging the document vectors of each document in the document set.
[0096]
In step ST133, the mining unit 208 acquires a document set for each attribute value for the specified attribute name 2. When "Usage field" is specified for the attribute name 2, a document set including "business use" in the attribute value and a document set including "private use" in the attribute value are obtained. In step ST134, the mining unit 208 creates an attribute vector from each document set acquired in step ST133.
[0097]
In step ST135, the mining unit 208 calculates a cosine value between each attribute vector created in step ST132 and each attribute vector created in step ST134, and sets the cosine value as the degree of association. In step ST136, the mining unit 208 displays the relevance of each attribute value of the attribute name 2 to each attribute value of the attribute name 1 on a graph and outputs the result as a mining result.
[0098]
FIG. 32 is a diagram illustrating a mining result by the mining unit 208, and is a diagram illustrating an example in which a result of the processing in FIG. 31 is displayed in a graph. From the graph of FIG. 32, it can be seen that, for example, the correlation between “fashion” and “private use” is high, and the correlation between “function” and “business use” is high.
[0099]
As described above, the mining unit 208 can analyze the degree of association between attribute values determined for a plurality of attribute names by the search / mining result editing unit 112. In this case, the attribute values to be analyzed include not only the attribute values determined by the search / mining result editing unit 112 but also the other attribute values input using the attribute values determined by the search / mining result editing unit 112. May be analyzed.
[0100]
As described above, according to the first embodiment, the results obtained by the search and the text mining are saved, and the saved search and mining results are associated with each other to create a new analysis target and a new attribute. And analysis based on a user-defined criterion (analysis axis) that is not defined in the original document is obtained.
[0101]
In addition, according to the first embodiment, by using the concept dictionary 204 and the document vector index 205, even for words having different notations, words and words, and words and documents Since the degree of association between a word and an attribute can be determined, it is possible to obtain the effect that search, mining, and association of search and mining results can be performed based on semantic proximity even for expressions having different notations.
[0102]
Furthermore, according to the first embodiment, the concept dictionary generation unit 201 extracts a compound word, so that a word having higher specialty is extracted than a word combining general words, and a concept index for the compound word is extracted. Since it is generated, it is possible to perform a mining process based on a more appropriate word, and it is possible to obtain an effect that the accuracy of the mining process can be improved.
[0103]
Each of the above processes can be realized on a computer by a text mining program installed on the computer.
[0104]
【The invention's effect】
As described above, according to the present invention, the index storage unit storing the index information for searching and mining the target document set and the index information stored according to the specified search and mining conditions are referred to. A search / mining execution unit that executes the search / mining process of the target document set, and the search / mining result by the search / mining execution unit is assigned to each document with the degree of association between each document and the search / mining condition. A search / mining result storage unit to be stored in the search / mining result storage unit, and a plurality of specified search / mining results are read from the search / mining result storage unit, and each document is read according to the specified attribute name. A search / mining result editor that selects attribute values by selecting search / mining results read in To allow creation of attributes, there is an effect that users who are not defined in the original document can be analyzed based on the criteria defined.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a text mining device according to Embodiment 1 of the present invention.
FIG. 2 is a detailed block diagram showing a configuration of a text mining device according to Embodiment 1 of the present invention.
FIG. 3 is a diagram illustrating an example of a document in a target document set.
FIG. 4 is a flowchart showing a processing flow of a concept dictionary generation unit of the text mining device according to the first embodiment of the present invention.
FIG. 5 is a diagram for explaining processing of a concept dictionary generation unit of the text mining device according to Embodiment 1 of the present invention;
FIG. 6 is a flowchart showing a flow of compound word extraction processing of a concept dictionary generation unit of the text mining device according to Embodiment 1 of the present invention.
FIG. 7 is a diagram showing an example of a compound word candidate extraction dictionary held by the concept dictionary generation unit of the text mining device according to the first embodiment of the present invention.
FIG. 8 is a diagram showing a compound word candidate extraction result of the concept dictionary generating unit of the text mining device according to the first embodiment of the present invention.
FIG. 9 is a diagram illustrating generation of a document vector index by a document vector index generation unit of the text mining device according to the first embodiment of the present invention.
FIG. 10 is a flowchart showing a processing flow of a document vector index generation unit of the text mining device according to the first embodiment of the present invention.
FIG. 11 is a diagram showing a document vector index of the text mining device according to the first embodiment of the present invention.
FIG. 12 is a flowchart showing a processing flow of a mining index generation unit of the text mining device according to the first embodiment of the present invention.
FIG. 13 is a diagram illustrating an example of a mining index generated by a mining index generation unit of the text mining device according to the first embodiment of the present invention.
FIG. 14 is a flowchart showing a processing flow of an execution unit of the text mining device according to the first embodiment of the present invention.
FIG. 15 is a flowchart showing a flow of a search process executed by the search unit of the text mining device according to the first embodiment of the present invention.
FIG. 16 is a diagram showing an example of a search result by a search unit of the text mining device according to the first embodiment of the present invention.
FIG. 17 is a flowchart showing a flow of a mining process executed by the mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 18 is a diagram showing a mining result by a mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 19 is a flowchart showing a flow of a mining process executed by the mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 20 is a diagram showing a mining result by the mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 21 is a flowchart showing a flow of a mining process executed by the mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 22 is a diagram illustrating a mining result by a mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 23 is a diagram illustrating a mining result by a mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 24 is a flowchart showing a processing flow of a search / mining result storage unit according to the first embodiment of the present invention.
FIG. 25 is a diagram illustrating an example of a search / mining result stored in a search / mining result storage unit of the text mining device according to the first embodiment of the present invention.
FIG. 26 is a flowchart showing a processing flow of a search / mining result editing unit of the text mining device according to the first embodiment of the present invention.
FIG. 27 is a diagram showing an editing result by a search / mining result editing unit of the text mining device according to the first embodiment of the present invention.
FIG. 28 is a flowchart showing a processing flow of a search / mining result editing unit of the text mining apparatus according to Embodiment 1 of the present invention.
FIG. 29 is a diagram showing an editing result by a search / mining result editing unit of the text mining device according to the first embodiment of the present invention.
FIG. 30 is a diagram showing an editing result by a search / mining result editing unit of the text mining device according to the first embodiment of the present invention.
FIG. 31 is a flowchart showing a processing flow of a mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 32 is a diagram illustrating a mining result by a mining unit of the text mining device according to the first embodiment of the present invention.
FIG. 33 is a diagram illustrating a conventional text mining analysis process.
FIG. 34 is a diagram for explaining text mining analysis processing according to Embodiment 1 of the present invention;
[Explanation of symbols]
101 target document set, 102 index generation unit, 103 index storage unit, 104 execution unit, 105 input unit, 106 display unit, 107 search / mining result storage unit, 108 index reading unit, 109 search / mining result reading unit, 110 search Mining execution unit, 111 search / mining result storage unit, 112 search / mining result editing unit, 201 concept dictionary generation unit, 202 document vector index generation unit, 203 mining index generation unit, 204 concept dictionary, 205 document vector index, 206 Mining index, 207 search unit, 208 mining unit.

Claims (10)

An index storage unit that stores index information for searching and mining a set of target documents;
A search / mining execution unit that executes a search / mining process of the target document set by referring to the index information stored in the index storage unit according to a specified search / mining condition;
A search / mining result storage unit for assigning the degree of relevance between each document and the search / mining condition to each of the documents and storing the search / mining result by the search / mining execution unit in the search / mining result storage unit;
A plurality of specified search / mining results are read from the search / mining result storage unit, and the attribute values are determined by selecting the read / mining results read for each document in accordance with the specified attribute name. A text mining device equipped with a search / mining result editing unit. 2. The text mining apparatus according to claim 1, wherein the search / mining result editing unit selects a search / mining result having the highest relevance for each document and determines an attribute value of each document with respect to the attribute name. . 2. The text according to claim 1, wherein the search / mining result editing unit selects a search / mining result whose relevance is equal to or more than a predetermined threshold for each document, and determines an attribute value of each document with respect to the attribute name. Mining equipment. 2. The text mining apparatus according to claim 1, wherein the search / mining execution unit analyzes the degree of association between the designated mining condition and the attribute value determined by the search / mining result editing unit. 2. The text mining apparatus according to claim 1, wherein the search / mining execution unit analyzes the correlation between the attribute values determined by the search / mining result editing unit and the distribution of the specified mining conditions. . The text mining apparatus according to claim 1, wherein the search / mining execution unit analyzes the correlation between the words based on the distribution of the documents stored in the search / mining result storage unit. The text mining apparatus according to claim 1, wherein the search / mining execution unit analyzes the degree of association between the attribute values determined for the plurality of attribute names by the search / mining result editing unit. The index information stored in the index storage unit includes a concept dictionary that describes a concept vector of a word in each document generated by performing morphological analysis on text included in the target document set, and a document included in the target document set. 2. The text mining apparatus according to claim 1, further comprising a document vector index that describes a concept vector of a document generated by morphologically analyzing the document and referring to the concept dictionary. 9. The text mining apparatus according to claim 8, wherein a concept vector of a compound word extracted from a morphological analysis result of the text included in the target document set is described in a concept dictionary. A first function of executing a search / mining process of a target document set by referring to index information stored in an index storage unit according to a specified search / mining condition;
A second function of adding the search / mining result obtained by the first function to each document and the relevance between each document and the search / mining condition, and storing the result in the search / mining result storage unit;
A plurality of specified search / mining results are read from the search / mining result storage unit, and the attribute values are determined by selecting the read / mining results read for each document in accordance with the specified attribute name. A text mining program that makes a computer realize the third function.

Images