JP4423004B2 - Text mining device, text mining method, and text mining program - Google Patents

Description

  The present invention relates to a text mining apparatus and text mining that can extract important information necessary for business such as product planning and quality control from a large amount of accumulated text and use it for business improvement. The present invention relates to a method and a text mining program.

  With the progress of computerization of documents, the importance of text mining devices for extracting important information required for product planning and quality control from stored documents is increasing. Among such text mining devices, as a text mining device capable of searching for a document having similar contents even if the expression is different from that of the input document, Document 1 (text mining for keeping customers' voices awake, Katsuya Mimuro) , Intellectual Asset Creation September 2002 issue). In addition, Reference 2 (TRUE TELLER Ver.2.0), a text mining tool that analyzes “customer voice”, was released. NRI Nomura Research Institute News Release, February 18, 2002. ) And Document 3 (announced “TRUE TELLER Ver.3.0”, a text mining tool that analyzes “customer voice”, NRI Nomura Research Institute News Release, December 19, 2002) There are things. The text mining method disclosed by the above Literature 1, Literature 2, and Literature 3 will be described with reference to FIG.

  In the figure, word dividing means 1801 analyzes text included in the analysis target document 1821 and divides it into words. The word unifying unit 1802 refers to the synonym dictionary 1803 and absorbs the fluctuation of the word division result by the word dividing unit 1801. The syntax analysis means 1804 extracts the dependency relationship between words from the word division result by the word division means 1801 and stores it in the analysis database 1806. At this time, information indicating which word appears in which document is also stored in the analysis database 1806. Also, the dependency synonym is processed by the dependency synonym dictionary 1805 and stored in the analysis database 1806.

  The parsing unit 1804 further stores the score information calculated for each document in the analysis database 1806 with reference to the score definition information 1808 that defines the degree of dissatisfaction and the degree of demand for the word. (“Scoring function” in Document 2). The analysis unit 1809 generates an analysis result 1823 for the analysis input 1822 with reference to the analysis database 1806 and the customer attribute database 1807. With such a text mining device, for example, analysis support as shown in FIG. 19 can be performed for a free description sentence of a questionnaire regarding cosmetics (Reference 1).

Katsuya Mimuro “Text Mining to Keep Customer Voices Awake” Intellectual Asset Creation, September 2002 pp44-pp53 "Announced" TRUE TELLER Ver.2.0 ", a text mining tool for analyzing" customer voice "", News Release, NRI Nomura Research Institute, February 18, 2002 "Announced" TRUE TELLER Ver. 3.0 ", a text mining tool for analyzing" customer voice "", News Release, NRI Nomura Research Institute, December 19, 2002

  However, the techniques disclosed in Document 1, Document 2, and Document 3 have the following problems.

  First, although a score for a word can be defined for each piece of information to be extracted, such as popular, unpopular, and desired, there is a problem that it is not possible to extract sufficient information by itself. For example, considering the word “high” in the free description response to a product questionnaire, a negative opinion is given if the price is high, and a positive opinion if the reliability is high. Therefore, it may not be possible to distinguish between positive opinions and negative opinions simply by defining a score for a word.

In addition, in order to perform analysis support corresponding to various expressions of freely described texts, it is necessary to manually create a large number of synonym dictionaries, which is problematic. Reference 4 (How to use text mining, Satoshi Ishii, Rick Telecom, November 2002) discloses a method that can automatically calculate similarity by statistically processing words in a learning document. , Because it does not take into account functional expressions that are important in the analysis of free description text, such as adjectives such as “good” and “bad”, and desired expressions such as “I want to”, so sufficient analysis support functions are provided. There wasn't.

  The present invention has been made in view of the above-described problems. A collation pattern that defines a correspondence relationship between information in an analysis target text and information to be extracted, and a text obtained by collating information required for analysis with a collation pattern. It is intended to be able to discriminate free description answers in consideration of multiple words and utilize it for analysis support by providing information extraction means for extracting from the information and extraction information storage means for storing the extracted information .

The text mining device according to the present invention is:
Registration means, text analysis means, attribute database, concept dictionary creation means, concept dictionary, collation pattern storage means , information extraction means, extraction information index storage means , document index storage means , analysis means,
The registration means reads the analysis target document,
The text analysis means analyzes the text of the read analysis target document by the registration means, divides it into words, extracts the co-occurrence frequency of words and the dependency relationship between words ,
The attribute database is attribute information consisting of items that can narrow down the analysis target document by customer information such as age and gender given to questionnaire documents, and analysis processing contents such as model name and failure derivation date given to failure case documents Is stored,
The concept dictionary creation means creates concept dictionary data by calculating the concept vector of each word by singular value decomposition from the co-occurrence frequencies of the words divided by the text analysis means,
The concept dictionary stores concept dictionary data created by the concept dictionary creating means,
The registration means further reads out a concept vector corresponding to each word divided by the text analysis means from the concept dictionary and synthesizes these concept vectors for each sentence to generate a document vector which is one of the document index information. The document index storage means stores the document vector generated by the registration means,
The collation pattern storage means stores a collation pattern that is created in advance and has an analysis type as an analysis condition, a word related to the analysis type, and an evaluation value for the analysis type of the word ,
The information extraction unit extracts an evaluation value for the word, the analysis type, and the analysis type when the word obtained as a result of analyzing the analysis target document by the text analysis unit is in the collation pattern stored in the collation pattern storage unit .
The extracted information index storage means stores the word extracted by the information extracting means , the analysis type, the evaluation value for the analysis type, and the document id of the corresponding document ,
Analytical means
An analysis condition input means for an analysis worker to input analysis conditions;
A frequency totaling unit that obtains a word that matches the analysis condition input by the analysis condition input unit and a word that easily co-occurs with the word from the document index storage unit with reference to the attribute database and the concept dictionary ;
Relevance calculating means for extracting and summing the evaluation value of each word from the extracted information index storage means for words that match the analysis conditions acquired by the frequency counting means and words that are likely to co-occur with the words ;
It has an output means for shaping and outputting the information obtained by the relevance calculation means into a graph or table format .

Further, the text mining method according to the present invention includes:
A registration step of reading the analysis target document by a registration means;
In the registration step, the text of the read analysis target document is analyzed by a text analysis means, divided into words, and a text analysis step for extracting the co-occurrence frequency of words and the dependency relationship between words ;
A concept dictionary creation step of calculating a concept vector of each word by singular value decomposition by means of singular value decomposition from the co-occurrence frequency of the words divided by the text analysis step, and storing the concept dictionary data in the concept dictionary;
A concept vector corresponding to each word divided by the text analysis step by the document vector generation means is read from the concept dictionary, and these concept vectors are synthesized for each sentence to generate a document vector, which is one of the document index information. Generating a document vector to be stored in the index;
An analysis target document is analyzed in a text analysis step using a collation pattern that is created in advance and stores an analysis type that is an analysis condition, a word related to the analysis type, and a collation pattern having an evaluation value for the analysis type of the word. If the analyzed word is in the collation pattern, the information extraction step extracts the word, the analysis type, and the evaluation value for the analysis type by the information extraction unit, and stores them in the extracted information index storage unit together with the document id of the corresponding document When,
In addition to the customer information in the questionnaire document, the attribute database that stores the attribute information given to the analysis target document of the model name and failure derivation date in the failure case document, and the document vector stored in the document index storage means, the information extraction step An analysis step for obtaining an analysis result by an analysis means with reference to the extracted information is provided,
The analysis step refers to a word that matches the analysis condition input by the analysis operator using the analysis condition input means and a word that easily co-occurs with the word from the document index storage means by referring to the attribute database and the concept dictionary. A frequency aggregation step to retrieve,
Relevance calculation step of extracting the evaluation value of each word from the extracted information index storage means for the word that matches the analysis condition acquired in the frequency counting step and the word that is likely to co-occur with the word ,
It has an output step of shaping and outputting the information obtained in the relevance calculation step into a graph or table format .

Further, the text mining program according to the present invention is:
A registration step of reading the analysis target document by a registration means;
In the registration step, the text of the read analysis target document is analyzed by a text analysis means, divided into words, and a text analysis step for extracting the co-occurrence frequency of words and the dependency relationship between words ;
A concept dictionary creation step of calculating a concept vector of each word by singular value decomposition by means of singular value decomposition from the co-occurrence frequency of the words divided by the text analysis step, and storing the concept dictionary data in the concept dictionary;
A concept vector corresponding to each word divided by the text analysis step by the document vector generation means is read from the concept dictionary, and these concept vectors are synthesized for each sentence to generate a document vector, which is one of the document index information. Generating a document vector to be stored in the index;
An analysis target document is analyzed in a text analysis step using a collation pattern that is created in advance and stores an analysis type that is an analysis condition, a word related to the analysis type, and a collation pattern having an evaluation value for the analysis type of the word. If the analyzed word is in the collation pattern, the information extraction step extracts the word, the analysis type, and the evaluation value for the analysis type by the information extraction unit, and stores them in the extracted information index storage unit together with the document id of the corresponding document When,
In addition to the customer information in the questionnaire document, the attribute database that stores the attribute information given to the analysis target document of the model name and failure derivation date in the failure case document, and the document vector stored in the document index storage means, the information extraction step An analysis step for obtaining an analysis result by an analysis means with reference to the extracted information is provided,
The analysis step refers to a word that matches the analysis condition input by the analysis operator using the analysis condition input means and a word that easily co-occurs with the word from the document index storage means by referring to the attribute database and the concept dictionary. A frequency aggregation step to retrieve,
Relevance calculation step of extracting the evaluation value of each word from the extracted information index storage means for the word that matches the analysis condition acquired in the frequency counting step and the word that is likely to co-occur with the word ,
The computer executes each step of the output step of shaping and outputting the information obtained in the relevance calculation step into a graph or table format .

The present invention collates a collation pattern that defines information to be extracted with a result obtained by analyzing text in an analysis target document by a text analysis means, extracts information necessary for analysis from the collation result, and stores it in an extracted information index The information extraction means includes an attribute database for storing attribute information and a document vector generated from the analysis target document, and obtains an analysis result by referring to the registered document index and the extraction information of the information extraction means. Thus, it is possible to discriminate a free description answer in consideration of a plurality of words and use it for analysis support.
In addition, many similar expressions can be covered, and there is an advantage that it is possible to reduce the trouble of constructing a synonym dictionary during analysis work.

Embodiment 1.
FIG. 1 shows a configuration diagram of Embodiment 1 of a text mining apparatus according to the present invention. The text analysis means 101 analyzes the text contained in the document 121, divides it into words, and extracts the relationship between the words. The concept dictionary creation means 102 calculates the concept vector of each word from the appearance tendency of the words divided by the text analysis means 101 and stores it in the concept dictionary 103. The registration means 104 converts the text contained in the document 121 into vector information based on the concept vector registered in the concept dictionary 103 and registers it in the document index 105. The information extraction means 106 refers to the collation pattern 107 defined in advance, extracts information necessary for the analysis from the result of text analysis by the text analysis means 101, and stores it in the extracted information index 108 via the registration means 104. sign up. The attribute database 109 stores customer information such as gender and age in questionnaire analysis, and stores attribute information such as model name and failure derivation date in failure case analysis. The analysis means 110 reads the analysis input 122 input by the user, refers to the concept dictionary 103, the document index 105, the extracted information index 108, and the attribute database 109, and performs analysis for analysis support corresponding to the analysis input 122 Output information 123.

FIG. 2 is a processing flow showing an outline of the document analysis processing. The outline of the analysis process will be described below with reference to FIGS. 1 to 4 as appropriate.
First, the concept dictionary creation process in step S201 will be described. In step S201, first, the document 121 is read by the registration unit 104, and the text included in the document 121 is divided into words by the text analysis unit 101. Next, the registration means 104 calls the concept dictionary creation means 102 and uses the algebraic operation called singular value decomposition from the appearance tendency of the words divided by the text analysis means 101 (the frequency of words that appear at the same time (co-occurs)). The concept vector is calculated to create concept dictionary data as shown in FIG.
For the calculation of the concept vector, for example, the method disclosed in Reference 5 (“Information search system InfoMAP based on word associations, Takayama et al., Informatics Fundamentals 53-1, 1999-3”) is used.
Moreover, since there are many well-known documents about the method of dividing a word and extracting the dependency relation between words, description is abbreviate | omitted.

  In FIG. 1, only one concept dictionary 103 is shown to avoid complexity, but a concept dictionary 103 is created for each field of the analysis target document. For example, in order to analyze a mobile phone questionnaire result, a mobile phone questionnaire analysis concept dictionary is created. If an inquiry mail related to a washing machine is analyzed, a concept dictionary for the washing machine is created. These concept dictionaries only need to be learned from text having contents similar to the registration target document. Therefore, the concept dictionary 103 learned from a certain questionnaire result can be used for analysis of another questionnaire result having similar contents.

In step S202, document index creation processing is performed. In this document index creation process, the document 121 read by the registration unit 104 is divided into words by the text analysis unit 101, concept vectors corresponding to the divided words are read from the concept dictionary 103, and these concept vectors are synthesized. A document vector that is one of the document index information is generated and stored in the document index 105. FIG. 4 shows an example of a document vector stored in the document index 105.
In the above description, an example of a document vector is described as the document index information. However, as the document index information, the concept dictionary 103 is not used, and a reference table that associates a word that appears in the document with the document can be used. Good.

  Next, in step S203, pattern extraction processing is performed. In the pattern extraction process, the document read by the registration unit 104 is analyzed by the text analysis unit 101, and whether the analyzed result matches the verification pattern 107 is verified by the information extraction unit 106, and necessary information is extracted from the verification result And stored in the extracted information index 108. Details of this pattern extraction processing will be described later.

  Finally, in step S204, text analysis processing is performed. In the text analysis process, an analysis input 122 input by an analysis operator is read by the analysis means 110, and an analysis input 122 for analysis support is referred to by referring to the concept dictionary 103, the document index 105, the extracted information index 108, and the attribute database 109. Output analysis information 123 for. Details of this text analysis processing will also be described later.

  The details of the pattern extraction process (collation process) in step S203 will be described below with reference to FIG. 1 and FIGS. 5 to 13 as appropriate. In the following description, it is assumed that the analysis target data is a questionnaire result regarding an air conditioner.

  FIG. 5 is a detailed configuration diagram of the information extraction means 106. The in-phrase pattern collating unit 501 extracts patterns that can be collated with respect to each phrase in the text analysis result read by the registering unit 104 and analyzed by the text analyzing unit 101, and collates with the phrase collation pattern of the collation pattern 107. To do. The in-sentence pattern matching unit 502 extracts patterns that can be matched against each sentence in the text analysis result that is also read by the registration unit 104 and analyzed by the text analysis unit 101, and extracts a plurality of words in the matching pattern 107. Match with in-sentence collation pattern. The dependency pattern matching unit 503 extracts a pattern that can be collated with respect to each dependency between words (relationship between two phrases) read by the registration unit 104 and analyzed by the text analysis unit 101. Collation with a dependency collation pattern that defines the dependency between 107 words.

  FIG. 6 is a processing flow of pattern extraction processing (information extraction processing) in the information extraction means 106. First, in step S601, the phrase pattern matching unit 501 performs the phrase pattern matching process. The in-phrase pattern matching means 501 refers to the phrase matching pattern of the matching pattern 107 and extracts a pattern that can be verified from the text analysis result by the text analysis means 101 input from the registration means 104. An example of the text analysis result is shown in FIG. The analysis result consists of a list of sentences (sentence), and each sentence consists of a list of clauses (pp). Each clause has a morpheme list, and the morpheme list consists of a list of morphemes. Also, the dependency relationship between each phrase is indicated by a pp-rel tag. When this text analysis result matches the collation pattern described below, necessary information is extracted according to the contents described in the collation pattern and stored in the extracted information index 108.

  FIG. 7 is an example of a collation pattern used in the phrase pattern collation process. In this embodiment, it is described in the xml format. Note that a description format other than the xml format may be used as long as it can hold the information described below. In FIG. 7, two patterns are shown. Each pattern is bounded by <pattern>-</ pattern>.

  Next, information described inside each pattern in FIG. 7 will be described. The first <extract-object> tag indicates the information to be extracted. Here, information whose attribute name is “favorability” and whose value is “3” is extracted. The next <co-region> tag is a tag for specifying a collation range. Here, “pp” indicating the inside of a phrase is specified. From <pp id = "0" negative = "false"> to </ pp> is pattern information used for collation with the text analysis result.

  The <pp> tag has two attribute values. The id attribute is an integer value indicating the position where the clause appears in the sentence. The negative attribute is flag information indicating whether or not the clause includes a negative expression, and “true” is specified if negative information is included, and “false” is specified if the negative information is not included. "False" is specified for clauses containing double negation. With this negative attribute, it is possible to extract a value of likability 3 by collating against the negative expression “not loud” in the second pattern shown in FIG.

  The child element of the <pp> tag is the <morph-list> tag. The <morph-list> tag has an order attribute. When the value is "false", matching is performed without considering the order. Also, morphemes that are not described in the verification pattern are ignored during verification. If the order attribute is "true", matching is performed considering the order. If a morpheme that is not described in the verification pattern appears, the verification fails. Usually, the description is simplified by collating without considering the order. For example, in the first pattern shown in FIG. 7, the matching condition is that the adjective verb "quiet" appears in <morph-list>. It is checked against various expressions such as “if it is quiet”.

  The child element of the <morph-list> tag is the <morph> tag. This corresponds to the minimum unit (morpheme) of word division. The <morph> tag has, as child elements, a <hyouki> tag indicating the heading notation of each morpheme and a <pos> tag indicating the part of speech. If the attribute is match and "strict" is specified, the notation and the part of speech are matched exactly. If "ambiguous" is specified, refer to the concept dictionary 103, Matching is also attempted with expressions such as “silent” and “low noise” similar to “quiet”, and if the degree of similarity is equal to or higher than a predetermined value, information necessary as “quiet” for successful matching (in this case, favorable sensitivity 3) To extract.

  In summary, the first pattern shown in FIG. 7 includes words similar to “quiet” and “quiet” in the free description, and the words are not “quiet”. If it is not denied, it means that a favorable opinion 3 regarding the air conditioner is given. Similarly, if the second pattern shown in FIG. 7 includes the word “noisy” and a negative expression such as “not noisy”, it is a positive opinion about the air conditioner. This means giving a preference of 2.

  The in-phrase pattern matching means 501 sequentially applies the matching pattern of FIG. 7 to each clause (pp) in the text analysis result (FIG. 8), and if the matching is successful, the extra- Information described in the object is extracted and stored in the extracted information index 108 via the registration unit 104. FIG. 9 shows an example of extracted information extracted by the phrase pattern matching process and stored in the extracted information index 108. The analysis result using this extracted information will be described later.

  Next, in step S602 of FIG. 6, the in-sentence pattern matching unit 502 performs the in-sentence pattern matching process. The in-sentence pattern matching unit 502 refers to the in-sentence matching pattern of the matching pattern 107 and extracts a pattern that can be verified from the text analysis result by the text analysis unit 101 input from the registration unit 104.

  FIG. 10 is an example of a collation pattern used in the in-sentence pattern collation process. Similar to FIG. 7, it is described in the xml format. Below, it demonstrates centering on a different part from FIG. The extract-object is the same as in FIG. First, the content of the co-region tag is sentence. The next element of the co-region tag is a sentence tag to be collated. The sentence tag is composed of a list of phrases as in the text analysis result shown in FIG. The sentence tag has an order attribute. When "true" is specified, collation is performed in consideration of the order matching of the clause list of elements. If "false" is specified, the order relation of clauses is ignored, and if the elements in the sentence match, the verification is successful.

  FIG. 10 shows two patterns as an example. The first pattern succeeds when the words “sound” and “small” appear in the sentence. For the matching of each of the “sound” and “small” clauses, the same processing as in the above-mentioned clause matching is performed. The second pattern shown in FIG. 10 succeeds when the words “display” and “small” appear in the sentence.

  The in-sentence pattern matching unit 502 sequentially applies the matching pattern of FIG. 10 to each sentence (sentence) in the text analysis result shown in FIG. 8, and when the matching is successful, the extra-object of each pattern. Are extracted and stored in the extracted information index 108 via the registration means 104. FIG. 11 shows an example of extracted information extracted by the in-sentence pattern matching process and stored in the extracted information index 108. The analysis result using this extracted information will be described later.

  Next, in step S603 of FIG. 6, the dependency pattern matching unit 503 performs dependency pattern matching processing. The dependency pattern collating unit 503 refers to the dependency collation pattern of the collation pattern 107 and extracts a collable pattern from the text analysis result by the text analyzing unit 101 input from the registration unit 104.

  FIG. 12 is an example of a collation pattern used in the dependency pattern collation process. As in FIG. 10, it is described in the xml format. 12 is different from FIG. 10 in the <pp-rel> tag inside the sentence tag. This indicates that the phrase with id = "0" (the phrase containing "belly" or "belly") relates to the phrase with id = "1" (the phrase containing "standing") Yes. At the time of collation, the collation is performed in consideration of this dependency relationship. That is, the sentence “I feel cold when I stand” includes the words “belly” and “stand” in the sentence, but there is no dependency relationship between “belly” and “stand”. Verification fails with the pattern shown.

  The dependency pattern matching unit 503 sequentially applies the matching pattern of FIG. 12 to each sentence (sentence) in the text analysis result (FIG. 8), and when the matching is successful, the extra-object of each pattern. Are extracted and stored in the extracted information index 108. FIG. 13 shows an example of extracted information extracted by the dependency pattern matching process and stored in the extracted information index 108. The analysis result using this extracted information will be described later.

  This is the end of the detailed description of the pattern collating process from steps S601 to S603 in FIG.

Next, details of the text analysis processing in step S204 of FIG. 2 will be described with reference to FIGS.
FIG. 14 is a detailed configuration diagram of the analysis unit 110. The analysis condition input means 1401 is a GUI (Graphical User Interface) for the analysis operator to interactively input the conditions of the analysis input 122. The frequency counting unit 1402 obtains the frequency of documents and words that match the analysis condition input by the analysis condition input unit 1401 with reference to the concept dictionary 103, the document index 105, the extracted information index 108, and the attribute database 109. To do. The relevance calculation means 1403 refers to the concept dictionary 103, the document index 105, the extracted information index 108, and the attribute database 109. From the document set classified by the analysis condition designation input by the analysis condition input means 1401 The similarity between the synthesized concept vector and the concept vector such as text designated by the analysis condition input by the analysis condition input means 1401 is calculated. The output means 1404 shapes the frequency and the degree of association obtained above in a form that makes it easy for the analysis operator to grasp the trend, and outputs the analysis information 123 in the form of a table or a graph.

  FIG. 15 is a process flow showing details of the text analysis process. First, in step S1501, the analysis operator inputs analysis conditions using the analysis condition input means 1401. FIG. 16 shows an example of the analysis condition input screen. The analysis target is limited to the 50 to 80-year-old age group, “Sound” and “Display” are selected as the topic of interest, and the favorable / unpopular analysis type is designated to grasp the reputation of these topics.

  Next, in step S1502, the frequency counting means 1402 refers to the word-document relationship table recorded in the document index 105 for words that are likely to co-occur with “sound” and “display” designated as the topic of interest. Get. At that time, referring to the attribute database 109, the following processing is performed after narrowing down the responses from customers aged 50 to 80. Here, “small”, “quiet”, “noisy”, etc. are obtained as co-occurrence words for “sound”, and “easy to see”, “small”, “difficult to see”, etc. are obtained as co-occurrence words for “display”.

  Next, in step S1503, the relevance calculation means 1403 obtains the information shown in FIGS. 9, 11, and 13 for each expression “sound-small”, “sound-quiet”, “display-small”, and the like acquired above. Based on the document id information of the extraction result, the likability and dissatisfaction of the documents including each expression are summed.

  Further, in step S1504, the output means 1404 graphs the favorability and the degree of dissatisfaction acquired as shown in FIG. This graph shows that, as an opinion of elderly people, the sound is popular but the display is unpopular.

  Here, the information shown in FIG. 11 is extracted by the in-sentence pattern matching process using the matching pattern shown in FIG. 10, so that the same “small” adjective is assigned to each of favorable and unfavorable cases. It becomes possible to analyze. ("Sound-Small" and "Display-Small" in FIG. 17)

  In addition, by the intra-phrase pattern matching process using the matching pattern shown in FIG. 7, accurate information as shown in FIG. 9 is extracted from the expression including the negative form. Even if it contains adjectives that contain, it can be recognized and analyzed.

  Each step described above can be processed by operating a computer by a program.

  As described above, according to the present embodiment, since the in-sentence pattern matching unit and the dependency pattern matching unit are provided, it is possible to extract the necessary information in consideration of a plurality of words. .

  Further, according to the present embodiment, by using the automatically generated concept dictionary 103 in the phrase pattern matching process, a single pattern description can cover a large number of similar expressions. There is an advantage that it is possible to reduce the trouble of dictionary construction.

  Furthermore, since the phrase pattern matching means is provided, information can be accurately extracted from text including negative expressions.

  Furthermore, since the in-sentence pattern matching means that can be collated in any word order is provided, the amount of pattern description can be reduced.

  Furthermore, since the dependency pattern collating means capable of collating by specifying a strict dependency relationship is provided, it is possible to accurately analyze an idiomatic expression in which the word order has an important meaning.

  Because it is possible to collate ambiguous expressions and to extract information such as popular, unpopular and desired from the text, it supports the questionnaire analysis work and can be applied to the questionnaire analysis service business.

It is a block diagram in Embodiment 1 of this invention. It is a processing flowchart which shows the outline | summary of an analysis process. It is explanatory drawing of the concept dictionary data created by the concept dictionary creation means. It is explanatory drawing of the document vector stored in a document index. It is a detailed block diagram of an information extraction means. It is a processing flowchart which shows the detail of an information extraction process. It is explanatory drawing of the collation pattern used by phrase pattern collation processing. It is explanatory drawing of the text analysis result by a text analysis means. It is explanatory drawing of the extraction information extracted by the pattern matching process in a phrase. It is explanatory drawing of the collation pattern used by the in-sentence pattern collation process. It is explanatory drawing of the extraction information extracted by the pattern verification process in a sentence. It is explanatory drawing of the collation pattern used by the dependency pattern collation process. It is explanatory drawing of the extraction information extracted by a dependency pattern collation process. It is a detailed block diagram of an analysis means. It is a processing flowchart which shows the detail of a text analysis process. It is explanatory drawing of the example of an analysis condition input screen. It is explanatory drawing of the example of an output graph processed by an output means. It is a block diagram of the conventional text mining apparatus. It is explanatory drawing of the example of an analysis assistance function by the conventional apparatus.

Explanation of symbols

101: Text analysis means,
102: Concept dictionary creation means,
103: Concept dictionary
104: Registration means
105: Document index,
106: Information extraction means,
107: Verification pattern
108: Extraction information index,
109: Attribute database,
110: Analytical means
121: Document,
122: Analysis input,
123: Analysis information
501: In-phrase pattern matching means,
502: In-sentence pattern matching means,
503: dependency pattern matching means,
1401: Analytical condition input means,
1402: Frequency counting means,
1403: Relevance calculation means
1404: Output means.

Claims (6)

Registration means, text analysis means, attribute database, concept dictionary creation means, concept dictionary, collation pattern storage means , information extraction means, extraction information index storage means , document index storage means , analysis means,
The registration means reads the analysis target document,
The text analysis means analyzes the text of the read analysis target document by the registration means, divides it into words, extracts the co-occurrence frequency of words and the dependency relationship between words ,
The attribute database is attribute information consisting of items that can narrow down the analysis target document by customer information such as age and gender given to questionnaire documents, and analysis processing contents such as model name and failure derivation date given to failure case documents Is stored,
The concept dictionary creation means creates concept dictionary data by calculating the concept vector of each word by singular value decomposition from the co-occurrence frequencies of the words divided by the text analysis means,
The concept dictionary stores concept dictionary data created by the concept dictionary creating means,
The registration means further reads out a concept vector corresponding to each word divided by the text analysis means from the concept dictionary and synthesizes these concept vectors for each sentence to generate a document vector which is one of the document index information. The document index storage means stores the document vector generated by the registration means,
The collation pattern storage means stores a collation pattern that is created in advance and has an analysis type as an analysis condition, a word related to the analysis type, and an evaluation value for the analysis type of the word ,
The information extraction unit extracts an evaluation value for the word, the analysis type, and the analysis type when the word obtained as a result of analyzing the analysis target document by the text analysis unit is in the collation pattern stored in the collation pattern storage unit .
The extracted information index storage means stores the word extracted by the information extracting means , the analysis type, the evaluation value for the analysis type, and the document id of the corresponding document ,
Analytical means
An analysis condition input means for an analysis worker to input analysis conditions;
A frequency totaling unit that obtains a word that matches the analysis condition input by the analysis condition input unit and a word that easily co-occurs with the word from the document index storage unit with reference to the attribute database and the concept dictionary ;
Relevance calculating means for extracting and summing the evaluation value of each word from the extracted information index storage means for words that match the analysis conditions acquired by the frequency counting means and words that are likely to co-occur with the words ;
A text mining apparatus comprising output means for shaping and outputting the information obtained by the relevance calculation means into a graph or table format . The collation pattern created in advance stored in the collation pattern storage means includes an analysis type as an analysis condition, a word related to the analysis type, an evaluation value for the analysis type of the word, and a search target range within a phrase. An in-phrase collation pattern having
The information extraction means uses the intra-phrase collation pattern stored in the collation pattern storage means , uses the words in the clause of the text analysis result by the text analysis means , and the analysis type that is the analysis condition, and relates to the analysis type 2. The text mining device according to claim 1, further comprising a phrase pattern matching unit that extracts a word and an evaluation value for the analysis type of the word . The collation pattern created in advance stored in the collation pattern storage means includes an analysis type as an analysis condition, a plurality of words related to the analysis type, an evaluation value for the analysis type by the plurality of words, and a search target An in-sentence matching pattern whose range is in a sentence ,
The information extraction means uses the in- sentence collation pattern stored in the collation pattern storage means , uses the words in the sentence of the text analysis result by the text analysis means , and the analysis type that is the analysis condition, and relates to the analysis type 3. The text mining apparatus according to claim 1, further comprising an in-sentence pattern matching unit that extracts a word and an evaluation value for the analysis type of the word . The collation pattern created in advance stored in the collation pattern storage means includes an analysis type as an analysis condition, a plurality of words related to the analysis type, a dependency relationship between the plurality of words, and a plurality of words. An evaluation value for the analysis type based on the dependency relationship between the two, and a dependency collation pattern having the search target range within the sentence ,
The information extraction means uses the dependency collation pattern stored in the collation pattern storage means , uses the words in the sentence of the text analysis result by the text analysis means , and the analysis type as the analysis condition, and relates to the analysis type a modification relation between words, according to any one of claims 1 to 3, characterized in that it has a dependency pattern checking means for extracting an evaluation value for analysis classification by modification relation between the word Text mining equipment. A registration step of reading the analysis target document by a registration means;
In the registration step, the text of the read analysis target document is analyzed by a text analysis means, divided into words, and a text analysis step for extracting the co-occurrence frequency of words and the dependency relationship between words ;
A concept dictionary creation step of calculating a concept vector of each word by singular value decomposition by means of singular value decomposition from the co-occurrence frequency of the words divided by the text analysis step, and storing the concept dictionary data in the concept dictionary;
A concept vector corresponding to each word divided by the text analysis step by the document vector generation means is read from the concept dictionary, and these concept vectors are synthesized for each sentence to generate a document vector, which is one of the document index information. Generating a document vector to be stored in the index;
An analysis target document is analyzed in a text analysis step using a collation pattern that is created in advance and stores an analysis type that is an analysis condition, a word related to the analysis type, and a collation pattern having an evaluation value for the analysis type of the word. If the analyzed word is in the collation pattern, the information extraction step extracts the word, the analysis type, and the evaluation value for the analysis type by the information extraction unit, and stores them in the extracted information index storage unit together with the document id of the corresponding document When,
In addition to the customer information in the questionnaire document, the attribute database that stores the attribute information given to the analysis target document of the model name and failure derivation date in the failure case document, and the document vector stored in the document index storage means, the information extraction step An analysis step for obtaining an analysis result by an analysis means with reference to the extracted information is provided,
The analysis step refers to a word that matches the analysis condition input by the analysis operator using the analysis condition input means and a word that easily co-occurs with the word from the document index storage means by referring to the attribute database and the concept dictionary. A frequency aggregation step to retrieve,
Relevance calculation step of extracting the evaluation value of each word from the extracted information index storage means for the word that matches the analysis condition acquired in the frequency counting step and the word that is likely to co-occur with the word ,
Text mining method characterized by having an output step of the information obtained by relevance calculating step, and outputs shaped into the form of a graph or table. A registration step of reading the analysis target document by a registration means;
The registration step, the text analysis step of the text of the loaded analyte document by analyzing a text analyzing means divides words, to extract a dependency relationship between words of co-occurrence frequency and word,
A concept dictionary creation step of calculating a concept vector of each word by singular value decomposition by means of singular value decomposition from the co-occurrence frequency of the words divided by the text analysis step, and storing the concept dictionary data in the concept dictionary;
A concept vector corresponding to each word divided by the text analysis step by the document vector generation means is read from the concept dictionary, and these concept vectors are synthesized for each sentence to generate a document vector, which is one of the document index information. Generating a document vector to be stored in the index;
An analysis target document is analyzed in a text analysis step using a collation pattern that is created in advance and stores an analysis type that is an analysis condition, a word related to the analysis type, and a collation pattern having an evaluation value for the analysis type of the word. If the analyzed word is in the collation pattern, the information extraction step extracts the word, the analysis type, and the evaluation value for the analysis type by the information extraction unit, and stores them in the extracted information index storage unit together with the document id of the corresponding document When,
In addition to the customer information in the questionnaire document, the attribute database that stores the attribute information given to the analysis target document of the model name and failure derivation date in the failure case document, and the document vector stored in the document index storage means, the information extraction step An analysis step for obtaining an analysis result by an analysis means with reference to the extracted information is provided,
The analysis step refers to a word that matches the analysis condition input by the analysis operator using the analysis condition input means and a word that easily co-occurs with the word from the document index storage means by referring to the attribute database and the concept dictionary. A frequency aggregation step to retrieve,
Relevance calculation step of extracting the evaluation value of each word from the extracted information index storage means for the word that matches the analysis condition acquired in the frequency counting step and the word that is likely to co-occur with the word ,
A text mining program that causes a computer to execute each step of an output step of shaping and outputting the information obtained in the relevance calculation step into a graph or table format .

Images