JP2016057652A - Word list generation device and word list generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the number of words required to be input by a user from among lacking words not described in a document described in natural language.SOLUTION: Clustered important words generated by using other documents and a causal relationship among these important words are formed into a database, when important words are extracted from a new document, and those among the important words of the database not included in the important words of the new document and the upstream words of the causal relationship included in the cluster of the important words are words not included in the important words, the words are included in a word list presented to require a user to input, and when those among the important words of the database not included in the important words of the new document and the upstream words of the causal relationship included in the cluster of the important words are words included in the important words, the words are not included in the word list presented to require a user to input, and thus a word list is created.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing technique for generating a word list that is lacking in a document written in a natural language.

  JP 2004-302506 A (Patent Document 1) [0024] states that “the text mining means 122 extracts necessary information from the text data converted by the speech recognition means 121. Although it may be used when text data is aggregated, it is possible to extract conceptual information that is useful for mining from text based on language processing.This function enables user information (names) to be extracted from text data (Including contact information), repair site, target device, failure location, failure status, etc. Necessary information can be extracted, and the text mining means 122 can extract necessary data from words that the user 200 speaks freely. In order to improve the accuracy of text mining, in order to get answers on necessary items, In some cases, the request receiving system 101 asks a predetermined question by voice, specifically, the voice transmission / reception unit 110 asks a name, address, telephone number, target device, faulty part, and fault status by a synthesized voice or the like. This is exemplified by a method for obtaining an answer, since it is an answer to a question, the range of the answer content can be predicted, and the accuracy of text mining is improved.

JP2004-302506

  In Patent Document 1, even if necessary items can be obtained from other answers, if the necessary items themselves cannot be obtained, all questions are asked.

  However, given the co-occurrence and causal relationships of words, there should be some words that can be derived from other answers that have already been answered.

  An object of the present invention is to collect necessary information in a short time by reducing the amount of information that the user is required to input among information that is lacking in a document described in a natural language.

  The present application includes a plurality of means for solving the above-described problems. An example of the means is as follows.

A word list generation method for inputting a new document described in a natural language into a computer, and causing the computer to generate a word list that requests a user to input from words lacking in the new document,
Cluster multiple important words extracted from other documents written in natural language, store the important words in the database together with the causal relationship between the important words in each cluster,
On the computer,
Extract important words from a new document for which you want to create a word list,
Of important words extracted from the other document, important words that are not included in the extracted important word of the new document, and an upstream word of causal relationship included in the cluster of the important word becomes an upstream word of the new document If the word is not included, include it in the word list presented to the user for input,
Of the important words extracted from the other documents, the words that are not included in the important words of the extracted new document, and the causal upstream words included in the cluster of the words are included in the upstream words of the new document A word list generation method characterized by causing a word list to be created when a word is not included in a word list to be presented to the user for input when the word is a word.

  According to the present invention, it is possible to collect necessary information in a short time by reducing the amount of information that the user is requested to input among information that is lacking in a document described in a natural language.

It is a block diagram of a word list production | generation apparatus. It is a figure which shows the processing flow of a word list production apparatus. It is a processing flowchart of an important word selection step. It is a figure which shows the data table of the document for DB creation. It is a figure which shows the word list produced from the document for DB creation of FIG. It is a figure which shows a topic estimation model. It is a figure which shows word distribution. It is a figure which shows the processing flow of a co-occurrence / causal relationship analysis step. It is a figure which shows the use performance data of each important word. It is a figure which shows the word cluster information of an important word. It is a figure which shows the causal relationship between each word. It is a figure which shows the processing flow of a word list preparation step. It is a figure which shows the screen image which presents a word list.

  Examples of the present invention will be described below, but the present invention is not limited thereto.

Hereinafter, embodiments will be described with reference to the drawings.
[Basic block configuration]
FIG. 1 is a block diagram of a word list generation device.

  The word list generation apparatus according to the present embodiment includes a database unit 1510, a word list generation engine unit 1520 connected to the database unit 1510 via a network, and a user interface unit 1530 connected to the word list generation engine unit 1520 via a network. And.

  The database unit 1510 includes a DB creation document DB 1511, an important word DB 1512, a co-occurrence / causal relationship DB 1513, and a new document DB 1514.

  The DB creation document DB 1511 stores a document described in a natural language used to construct the important word DB 1512 and the co-occurrence / causal relationship DB 1513.

  The important word DB 1512 stores important words extracted from the DB creation document DB 1511.

  The co-occurrence / causal relationship DB 1513 stores co-occurrence relationships and causal relationships between important words stored in the important word DB 1512.

  The new document DB 1514 stores a document described in a natural language, which is a word list generation target.

  The word list generation engine unit 1520 includes a morphological analysis engine 1521, a topic estimation model construction engine 1522, a word distribution estimation engine 1523, a threshold setting engine 1524, an important word determination engine 1525, and a word cluster creation engine 1526, The causal relationship analysis engine 1527, the candidate word extraction engine 1528, and the output word selection engine 1529 are included.

  The user interface 1530 includes a new document input screen 1531 and a word list confirmation screen 1532.

These engines and user interfaces will be described together with the following processing flow.
[Flow explanation]
FIG. 2 is a process flow diagram of the word list generation device of this embodiment.

  The word list generation apparatus according to the present embodiment targets a document described in a natural language, and acquires another document (for example, a document created in the past) that is not a word list generation target as a DB creation document. Document creation step 101, important word selection step 102 for selecting important words from the acquired DB creation document, and clustering the selected important words in a co-occurrence relationship, and the causal relationship of the important words in the cluster A co-occurrence / causal relationship extraction step 103 for analyzing a new document, a new document acquisition step 104 for acquiring a new document as a word list generation target, an important word selection step 105 for selecting an important word from the acquired new document, and a database A word list creating step 106 for generating a word list to be presented to the user from the important words clustered and stored in the part and the causal relationship in the cluster; And a processing flow from the word list presentation step 107 for presenting a word lists that form the user.

In the processing flow, steps 101 to 103 are DB creation steps, and steps 104 to 107 are word list creation steps.
[DB creation document acquisition step]
In a DB creation document acquisition step 101, a DB creation document is obtained from the DB creation document DB 1511.

A data table of the DB creation document DB 1511 is shown in FIG. In the DB creation document DB 1511, a case number, an issue date, and a document are recorded. The new document DB described later has the same format. In this embodiment, all the documents are acquired. However, if necessary, only a part of the documents may be acquired by using the issue date / time column or by setting other conditions. Good.
[Important word selection step]
Next, in the important word selection step 102, the DB creation document acquired in the DB creation document acquisition step 101 is input, and an important word is selected.

In this embodiment, a word affecting the cost or man-hour is selected as an important word for a design document. In the manufacturing industry, a design document is received from a customer, or a manufacturer creates a design document (such as a product specification) according to a customer's request. Words used in design documents are limited in terms of additional costs and man-hours, and the co-occurrence and causal relationships between words are easier to extract than other documents. is there. And it is a great merit for the manufacturing industry that a word list that can grasp cost and man-hour estimation from these design documents can be automatically generated by information processing by a computer.
[Detailed flow of important word selection step]
A detailed flow of the important word selection step 102 is shown in FIG.

The important word selection step 102 has a processing flow including a morphological analysis step 301, a topic estimation model construction step 302, a word distribution analysis step 303, a threshold setting step 304, and an important word determination step 305. .
<Morphological analysis step>
In the morpheme analysis step 301, the DB creation document acquired in the DB creation document acquisition step 101 is input to the morpheme analysis engine 1521. Since the morphological analysis engine 1521 is an existing technology, the details are omitted, but the morphological analysis engine 1521 outputs a word list obtained by dividing each DB creation document into words.

FIG. 5 shows a word list generated in the morphological analysis step 301 from the DB creation document of FIG. The document for creating a DB in Item No. 1 “Placement of device A and device B has changed, so please extend the length of the wiring.” Is “device A” “and” “device B” “Of” “placement” “ga” “change” “to” “now” “so” “,” “wiring” “of” “length” “to” “extend” “do” “please” “ . ”Is divided into a word list.
<Topic estimation model construction step>
In the next topic estimation model construction step 302, a word list obtained by dividing the document into words in the morpheme analysis step 301 is input to the topic estimation model construction engine 1522. The topic estimation model construction engine 1522 outputs a model for estimating what topic each DB creation document handles. Since this topic estimation model construction engine 1522 is an existing technology, it will not be described in detail, but usable models include Naive Bayes, Probabilistic Latent Semantic Indexing, and Latent Dirichlet Allocation. In this embodiment, as shown in FIG. 6, a model using Latent Dirichlet Allocation is constructed.

<Word distribution estimation step>
In the next word distribution analysis step 303, the word distribution engine 1523 receives the word list output from the morpheme analysis step 301, the DB creation document, the topic estimation model constructed in the topic estimation model construction step 302, and the morpheme The word list that is the output of the analysis step 301 is input. The word distribution engine 1523 estimates the topic of the DB creation document based on the topic estimation model, analyzes the appearance frequency of each word for each topic, and outputs the word distribution (appearance frequency). FIG. 7 shows an example of word distribution. FIG. 7 shows a histogram of the appearance frequency of words for topic A. The horizontal axis indicates a list of words divided by the morphological analysis engine 1523, and the vertical axis indicates the appearance frequency of each word.

<Threshold setting step>
In the next threshold setting step 304, the appearance frequency of each word analyzed in the word distribution analysis step 303 is input to the threshold setting engine 1524. The threshold setting engine 1524 outputs a threshold for determining whether each word is an important word. In the present embodiment, as a threshold setting method, a threshold for determining whether the appearance frequency of each word is significantly higher than other words is set for each topic. Specifically, the threshold value is a value obtained by adding a constant multiple of the difference between the 75% tile and the 25% tile point to the 75% tile point of the appearance frequency of each word.

  In this embodiment, the user can arbitrarily select whether to set the threshold for each topic or to set a common threshold for all topics.

  Further, in this embodiment, only important nouns and verbs excluding pronouns are extracted as important words, and other parts of speech are excluded because the amount of information is low. In the case of the present embodiment, a word affecting the cost or man-hour is selected as an important word for the design document, but other parts of speech hardly influence the cost or the man-hour.

<Important word determination step>
In the next important word determination step 305, the threshold set in the threshold setting step 304 and the appearance frequency of each word analyzed in the word distribution estimation step 303 are input to the important word determination engine 1525. The important word determination engine 1525 outputs a list of words having an appearance frequency greater than the threshold as important words. This important word is stored in the important word DB 1512.

  Next, a detailed flow of the co-occurrence / causal relationship extraction step 103 will be described with reference to FIG.

[Co-occurrence / causal relationship extraction step]
The co-occurrence / causal relationship extraction step 103 includes a word cluster creation step 801 and a causal relationship analysis step 802.

<Word cluster creation step>
In the word cluster creation step 801, the DB creation document of the DB creation document DB 1511 and the important word selected in the important word determination step 305 are input to the word cluster creation engine 1526. The word cluster creation engine 1526 extracts the use results of the important words selected in the important word determination step 305 in the DB creation document. Then, the word cluster creation engine 1526 analyzes the co-occurrence relationship of the important words for each topic selected in the important word determination step 305, and extracts words having a certain co-occurrence relationship as one cluster.

  An example of usage record data of each important word in the DB creation document is shown in FIG. Each row of the table shown in FIG. 9 corresponds to each DB creation document. Each column corresponds to each important word. When the column of the important word 1 in the row of the document 1 is 1, it means that the important word 1 is used in the DB creation document 1. On the other hand, the column of the important word 2 in the row of the DB creation document 1 is 0, which means that the important word 2 is not used in the DB creation document 1.

  FIG. 10 shows an example of word cluster information of important words. Each line corresponds to an important word. The second column of each row represents a word cluster. In FIG. 10, the fact that both the important word 1 and the important word 2 have a word cluster of 1 means that the important word 1 and the important word 2 belong to the same word cluster. As an example of the cluster division method, there is a method of applying a cluster analysis method to a matrix composed of the frequency with which words co-occur. Cluster analysis methods include k-means method and EM algorithm method.

<Causality analysis step>
In the causal relationship analysis step 802, the word cluster information created in the word cluster creation step 801 and the usage record data of each word are input to the causal relationship analysis engine 1527. The causal relationship analysis engine 1527 evaluates the causal relationship of words in the word cluster, determines which word is the upstream word, and determines the causal relationship between the determined words as a co-occurrence / causal relationship DB 1512. To remember.

  An example of the causal relationship between words is shown in FIG. Each row and column in the table shown in FIG. 11 corresponds to an important word. The column of important word 2 in the row of important word 1 is 1. The causal relationship that important word 1 is used when important word 1 is used (important word 1 is upstream, important word 2 is Means downstream). In addition, the column of important word 1 in the row of important word 2 is 0. When important word 2 is used, there is no causal relationship that important word 1 is used (important word 2 is important). Means upstream of word 1).

  Another method of determining which word is upstream is a method of using the number of documents created in the past using each word. For example, word A is used in 100 documents among past documents, word B is used in 50 documents among documents created in the past, and word A and word B are used simultaneously. Suppose that there were 10 documents. At this time, the probability that the word B co-occurs when the word A is used is 10%, and the probability that the word A co-occurs when the word B is used is 20%. The probability that the word A co-occurs is higher and it is determined that there is a causal relationship with the word B upstream.

[New document acquisition step and important word selection step]
The new document acquisition step 104 and the important word selection step 105 are the same as the DB creation document acquisition step 101 and the important word selection step 102 except that the target document is a new document. Note that a new document is registered in the new document DB by the user using the new document input screen 1531.

[Word list creation step]
Next, a detailed flow of the word list creation step 106 is shown in FIG.

  The word list creation step 106 is based on the important words in the important word DB 1512, the causal relationship between words in each word cluster of the co-occurrence / causal relationship DB 1513, and the important words in the new document selected in the important word selection step 105. Candidate words are extracted, words in the word list to be presented to the user are selected and output.

  The word list creation step 104 includes a candidate word extraction step 1201 and an output word selection step 1202.

<Candidate word extraction step>
Candidate word extraction step 1201 provides the candidate word extraction engine 1528 with important words in the important word DB 1512, causal relationships between words in each word cluster in the co-occurrence / causal relation DB 1513, and important words in the important word selection step 105. The important word in the new document selected by the determination engine 1525 is input. Of the important words in the important word DB 1512, important words that are not included in the important words in the new document are extracted as a first candidate word list. Then, the candidate word extraction engine 1528 refers to the causal relationship between the words in the co-occurrence / causal relationship DB 1513, and when the upstream word of the first candidate word is not included in the important word in the new document, Create the second candidate word list as it is without excluding the word, and generate the second candidate word list by excluding the word if the upstream word of the first candidate word is included in the important word in the new document To do.

<Output word selection step>
The output word selection step 1202 inputs the causal relationship between the second candidate word list extracted in the candidate word extraction step 1201 and each word of the co-occurrence / causal relationship DB 1513 to the output word selection engine 1529. The output word selection engine 1529 outputs, as a final word list, the word at the top of the category that includes each word in the second candidate word list by referring to the causal relationship between the words. As a simpler method, the second candidate word list may be used as it is as the final word list.

[Word list presentation step]
In the word list presentation step 105, the final word list created in the word list creation step 104 is presented to the user, and an input relating to the necessity of supplementation is requested. A screen image presented to the user is shown in FIG.

101 ... DB creation document acquisition step, 102 ... important word selection step, 103 ... co-occurrence / causal relationship extraction step, 104 ... new document acquisition step, 105 ... important word selection step, 106 ... Word list creation step, 107 ... Word list presentation step, 301 ... Morphological analysis step, 302 ... Topic estimation model construction step, 303 ... Word distribution analysis step, 304 ... Threshold value setting step, 305 ... Important word determination step, 801 ... Word cluster creation step, 802 ... Causal relationship analysis step, 1201 ... Candidate word extraction step, 1202 ... Output word selection step, 1510 ... Database part, 1511 ... Document DB for DB creation, 1512 ... Important word DB, 1513 ... Co-occurrence / causal relation DB, 1520 ... Word list generation engine part, 1521 ... Morphological analysis engine, 1522 ... Topic estimation model construction engine, 1523 ... Word distribution estimation engine, 1524 ... Threshold setting engine, 1525 ... Important word judgment engine, 1526 ... Word cluster creation engine, 1527 ... Causal relationship Analysis engine, 1528 ... candidate word extraction engine, 1529 ... output word selection engine, 1530 ... user interface, 1531 ... new document input screen, 1532 ... word list confirmation screen

Claims (8)

A word list generation device that generates a word list that requests a user to input from words that are lacking in a target document described in a natural language,
A database section and a word list generation engine section;
In the database unit, a plurality of important words extracted from other documents described in a natural language are clustered, and the important words are stored together with a causal relationship between words in each cluster,
The word list generation engine unit
An engine that extracts important words from a new document for which a word list is to be created;
Of the important words extracted from the other documents, the words that are not included in the important words of the extracted new document, and the causal upstream words included in the important word cluster are also included in the important words of the new document. An engine that is included in the word list that is presented to the user if the word is not included,
Of the important words extracted from the other documents, the words that are not included in the important words of the extracted new document, and the causal upstream words included in the cluster of the words are also included in the important words of the new document A word list generation device, comprising: an engine that creates a word list by not including the word in a word list presented to a user. In claim 1,
The word list generation engine unit
An engine that extracts a plurality of important words from other documents described in a natural language and stores them in a database unit;
An engine for extracting and clustering co-occurrence relationships between the plurality of important words and storing them in a database unit;
An engine that analyzes causal relationships between clustered important words and stores them in a database section;
A word list generation device comprising: In claim 1 or 2,
The other document and the new document are design documents. In claim 2,
A word list generation apparatus, wherein a causal relationship between words in the cluster is determined by the number of documents using each word. A word list generation method for inputting a new document described in a natural language into a computer, and causing the computer to generate a word list that requests a user to input from words lacking in the new document,
Cluster multiple important words extracted from other documents written in natural language, store the important words in the database together with the causal relationship between the important words in each cluster,
On the computer,
Extract important words from a new document for which you want to create a word list,
Of the important words extracted from the other document, important words that are not included in the extracted important word of the new document, and the causal upstream word included in the important word cluster becomes the important word of the new document. If the word is not included, include it in the word list presented to the user for input,
Of the important words extracted from the other documents, the words that are not included in the important words of the extracted new document and that are included in the cluster of the words are included in the important words of the new document. A word list generation method characterized by causing a word list to be created when a word is not included in a word list to be presented to the user for input when the word is a word. In claim 5,
Extract multiple important words from other documents written in natural language, store them in the database section,
Extracting and clustering co-occurrence relationships between the plurality of important words, storing them in a database unit,
A word list generation method, characterized in that a causal relationship between clustered important words is analyzed and stored in a database unit. In claim 5 or 6,
The word list generation method, wherein the other document and the new document are design documents. In claim 6,
A word list generation method, wherein the causal relationship between words in the cluster is determined by the number of documents using each word.

Images