JP2001084250A - Method and device for extracting knowledge from enormous document data and medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically extract a document satisfying a pattern from enormous amount of documents, to extract useful knowledge and to reduce time required for a response by generating a field-dependent dictionary from document data, generating a syntax tree considering modification, by means of a language analysis device and extracting/outputting a frequentlyappearing pattern by means of a pattern extraction device. SOLUTION: A language feature analysis device generates an analysis- dependent dictionary. A language analysis device needs to prepare a field- dependent dictionary for requiring an attribute adjusted to data to be analyzed. A word having the specified attribute is to be generated by each field. The language feature analysis device checks the word from actual data and registers it in the field-dependent dictionary. A pattern extraction device obtains a pattern, which frequently appears by using document data which is structure- analyzed by the device and takes out an original document having a syntax which is matched with the pattern. A frequently-appearing pattern device displays the document, having the detected frequently-appearing pattern and a syntax tree matched with it.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technology for automatically extracting a pattern in order to extract knowledge from a large amount of documents in a specific field. Is a technique for extracting knowledge from a large amount of document data by extracting useful knowledge by extracting documents satisfying the following from a large number of documents, an apparatus and a medium therefor.

[0002]

2. Description of the Related Art With the development and spread of computers and network environments, enormous amounts of data have been digitized and stored, and can be referenced online. In order to effectively use this data, data mining technology has been actively researched and developed. However, the conventional data mining technology targets only fixed data that can be tabulated based on numerical values. However, most data includes a text portion (that is, document data), and the document data is basically not a fixed form. Therefore, unlike the fixed form data centered on numerical values, it is difficult to perform the aggregation. Therefore, the document data basically needs to be passed one by one, which is extremely troublesome. That is, there is a limit to the amount of document data that can be analyzed manually, and there is a problem that a huge amount of document data that has been accumulated is left behind. Such a technique for extracting knowledge from an atypical text document is called “text mining” and has recently attracted attention. This text mining is the technology most anticipated because it can be applied not only to record in call centers, totaling of questionnaire results, etc. but also to analysis of any atypical documents such as patent documents, business reports, etc. is there.

[0003] As a means of analyzing the contents of a large number of documents,
There is a method of finding documents having similar contents and classifying them by category. For example, as a method currently used in web search sites and the like, there is a method in which a category is prepared in advance, and a person reads a document, determines a category to which the document corresponds, and classifies the document. In addition, this operation can be automatically performed based on the rule that a document including a specific keyword belongs to a certain category. For example, if there is a document including the keywords “ABS” and “airbag”, it can be determined that the document belongs to the category of car.
This is suitable for large classification of a large number of documents, but it is difficult to make a finer classification.

For example, in a call center business,
There is a demand for improving call center operations by analyzing what requirements are high in telephone contents from customers. This requirement can be fulfilled by roughly classifying the contents of recorded telephone calls manually, carefully reading the documents from the classified results, and counting documents having substantially the same contents. However, in the case of a call center receiving tens of thousands of inquiries every month, it is very labor-intensive to do this manually, and it is difficult in practice. The stored documents are documents for a specific field, and the categories need to be very finely divided. However, it is also very difficult to predict the contents and prepare the categories in advance. For example, it is required to categorize not only the simple category of "car" but also the category of "occurrence of abnormal noise of engine". In such a fine classification, the classifying person has to examine the contents of the document more carefully to perform the classification work, and the amount of work is enormous. In addition, there is a possibility that the judgment criteria of the category differ from person to person, and even if the same person makes a different judgment each time, it is difficult to obtain objective data.

In recent years, a document classification method (document clustering) using a computer has been developed. This method performs rough classification based on keywords appearing in a document. However, when targeting a specific field, a more detailed classification is required, and cannot be dealt with by the conventional method. Further, as a result of the clustering, there is a problem that it is impossible to understand what kind of documents are collected in one cluster unless a person reads the documents.

As described above, the conventional technique of extracting words as keywords from a large amount of documents and extracting a pair of co-occurring words is called "correlation rule extraction technique" in data mining. However, this method has the following problems. There may be no relation between the first word and the last word in a long document, but this is counted as co-occurrence or because the dependency of words is ignored. For example, "A is B
Then, the meaning is different between "C did D" and "A did D then C did B". However, if only the co-occurrence relationship is viewed, these two documents are processed as the same document. Therefore, it is often the case that a document with the same content is not correctly extracted.

In order to solve the inconvenience as described above, a method of extracting only words in which specific words appear in a specific order can be considered. This is called “time-series pattern extraction technology” in data mining.
For example, only documents in which words appear in the order of word A, word B, word C, and word D can be extracted. However, according to this rule, a document "A is B and C is D" can be extracted, but "C is D if A is B
There is a problem that the document "I did" cannot be extracted because the word order is different, although the meaning of the document is the same.
That is, in order to extract documents having the same contents, it is necessary to pay attention not only to the co-occurrence relation and the appearance order of words but also to the dependency relation between words.

[0008]

As described above, according to the present invention, a specific pattern is extracted from a large number of documents.
Another object of the present invention is to provide a method of extracting knowledge from a huge amount of document data, which realizes useful knowledge extraction by automatically extracting a document satisfying the pattern, an apparatus and a medium therefor.

[0009]

SUMMARY OF THE INVENTION The present invention is directed to a method for extracting knowledge from a large amount of document data. In this knowledge extraction method, a step of extracting words from one document by a morphological analysis technique, estimating a dependency relationship between words, and constructing a syntax tree from the dependency relationships, The method includes a step of finding a frequent pattern included in the parse tree based on a given pattern constraint, and a step of searching for a document that matches the assignment to the found frequent pattern. The present invention also includes a computer-readable medium storing a program for causing a computer to execute the steps of the above method.

Further, the present invention is directed to an apparatus for extracting knowledge from a large amount of document data. In this knowledge extraction device, a language feature analysis device that registers a vocabulary not included in the basic dictionary in the field-dependent dictionary, a language analysis device that performs natural language analysis, a pattern extraction device that finds data matching a specific pattern, and The apparatus includes a frequent pattern display device that displays frequent patterns, and includes a basic dictionary for a general field from document data, a generation rule for generating a clause, a generation rule for generating a syntax tree, and a field-dependent dictionary. . In the above configuration, knowledge extraction from a large number of documents can be suitably performed.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A language feature analysis device creates a field-dependent dictionary from document data to improve the accuracy of the language analysis device, creates a syntax tree in consideration of the dependency by the language analysis device, and creates a syntax tree by a pattern extraction device. Extract and output frequent patterns (ie, knowledge). Hereinafter, the present invention will be described in the form of a device, but it goes without saying that the present invention includes a method and a program medium. Specific functions include: 1. A function of extracting words from one document by a morphological analysis technique, estimating a dependency relationship between words, and constructing a syntax tree from the dependency relationships. In a large syntax tree constructed from a large number of documents,
2. A function of finding a frequent pattern included in many syntax trees based on a given pattern constraint. A function of outputting a document having a syntax tree of a found frequent pattern, and the like.

FIG. 1 is a schematic diagram showing a process of extracting a morpheme from a sentence, extracting a dependency relationship, and generating a syntax tree from the dependency relationship. As a result of the morphological analysis and the extraction of the dependency relation from the sentence “A becomes B, C becomes D” in FIG. 1, “A” becomes “B” and “C” becomes “D
Binary relations of "do", "do" and "do" are extracted. Here, the direction of the arrow is determined from the dependency relationship between the two words. From these relationships, a syntax tree in the figure is generated. The parse tree is expressed as a directed graph (a graph in which the branches connecting the nodes have directions). Nodes (referred to as nodes) on the directed graph are labeled with words extracted by morphological analysis (abbreviated as A to D in the figure). A branch (called an arc) connecting two nodes has a direction. The direction of the arc is determined by the dependency relationship between words, as described above. As shown in FIG. 2A, the pattern indicates a node existing in the syntax tree and its positional relationship.
The number of nodes, that is, the number of words is arbitrary. Here, a constraint can be given to each word (for example, verb, terminology, etc.). The positional relationship may be restricted to a certain one, but may be all possible positional relationships if the number of words is small. Here is an example of a pattern. Now
When there are two words A and B in one parse tree, A
By tracing the directed graph in the parse tree from the node having the label “B” to the node having the label “B”, the node can be reached. As shown in FIG. A- * → B can be described as a pattern. Further, similarly, there are other words C and D, and at the same time, A- * → B, A- * → C- * →
When the relationship of D is established, this is defined as a pattern consisting of four words and their positional relationship. Also, restrictions can be given to this pattern. For example, there is a restriction such as a verb or a technical term for A. The discovery of a frequent pattern is to find a frequent pattern among patterns representing a plurality of words and their positional relationship.

When the document is in Japanese or the like, not only a syntax tree but also a linear list can be constructed. Similarly, a given pattern can be found for a linear list. In this case, the processing speed is increased.

Regarding co-occurrence relations, in general, the greater the distance between words in a sentence, the smaller the relevance between the words and phrases is.
In the syntax tree, the concept of the number of branches (number of arcs) passing from a certain node to a certain node is introduced. For example, in the case where the distance is defined as 3, for example, a word having a distance of 4 or more is treated as having no co-occurrence relation between nodes where the word is separated from the word. This distance is set to an appropriate value according to the target document. FIG. 3 is a diagram showing the overall configuration of the present invention. FIG. 4 is a flowchart showing the flow of the processing of the present invention. FIG. 5 shows the details of the language analysis apparatus. The pattern extraction apparatus obtains a frequently occurring pattern using the document data subjected to the structure analysis by the apparatus, and extracts an original document having a syntax matching the pattern. . The frequent pattern display device displays a document having a found frequent pattern and a syntax tree that matches the frequent pattern. Here, the present invention is configured as follows. 1. Language feature analyzer, 2. language analyzer, 3. a pattern extraction device; The frequent pattern display device will be described.

1. About the language feature analyzer The language feature analyzer creates a field-dependent dictionary in order to improve the accuracy of the language analyzer. This adds a vocabulary for a specific field that is not included in the general dictionary and describes the attributes of that vocabulary. In addition, a field-dependent dictionary is created for vocabularies having different meanings and attributes depending on the field. Since the language analysis device needs an attribute suitable for the data to be analyzed, a field-dependent dictionary (for example, "device (19)"
It is necessary to prepare a dictionary for rewriting a “device (H / W)”. For general words such as "device" and "good-good", the first one can be used for any data, but words with specific attributes such as product names should be created for each field. There must be. It is the language feature analyzer that checks this from actual data and registers it in the field-dependent dictionary, and performs registration in the following procedure.

A. A sentence is divided into a word string with a part of speech using a morphological analysis device and a basic dictionary, which are conventional techniques. B. Those already registered in the field-dependent dictionary are deleted from the word string. C. Calculate the frequency of appearance of words, and sort the word strings in descending order of frequency of appearance. D. From this word string, a word corresponding to a preset attribute is found and additionally registered in a field-dependent dictionary. Here, if the structure of the entry in the field-dependent dictionary is changed from a word string with part-of-speech to a word string with part-of-speech or a word string with attributes, even if the morphological analyzer performs incorrect word division or incorrect part-of-speech assignment, And attributes can be extracted.

2. About the language analysis device The language analysis device includes a morphological analysis device, a phrase generation device, a dictionary application device, and a dependency analysis device, and each will be described below. (1) Morphological analysis device An input sentence is divided into words t by performing a morphological analysis, which is a conventional technique, on a sentence, and then, using a basic dictionary, a word string is labeled 1 (corresponding to a part of speech or an attribute name). Name). The distance d between words is added as a weight. Hereinafter, a set of morphemes w = [t, l, d].
In addition, using the synonym dictionary, the fluctuation of the expression and the synonyms are changed to one unified notation.

(2) Clause generation device In the order in which each word appears in a sentence (or a specific context), w ₁
, w ₂ ,,... w _n , the clauses are determined in order from w ₁ according to the generation rules. Separate clauses where w _n is an adjunct or where it can be clearly determined that the clause is broken. If a clause is separated by w _k , the next clause starts at w _{k + 1} and continues until the end of the sentence. Each clause is made up of a combination of an independent word and an adjunct word, which is a node of the syntax tree and an arc from the node. In addition, in the case of a phrase having a prefix such as “anti, non” and an auxiliary verb such as “not,” the sign of the label is inverted.

(3) Dictionary application apparatus Words and labels in a word string are rewritten by a field-dependent dictionary. If there is no corresponding attribute name, the part of speech remains as a label. Information such as part of speech and the like are added to each node in addition to words, and information about particles is added to the arc.

For example, from the sentence "The device is not bad", the following is output. In the morphological analyzer used here, the weight is always 1, and the display of the weight d is omitted. The numbers indicate the parts of speech.
For example, 19: noun, 75: case particle "ga", 17: stem of adjective, 42: adjective conjugation ending, etc. Period (.)
Is a simple but effective weighting method. (1). Output from morphological analyzer : [apparatus, 19]
[But, 75] [Good, 17] [C, 42] [No, 51] [Yes, 4
3] [translation, 94] [,, 56] [is, 85] [not, 51] [is, 4
3] (2). Output from phrase generator : ([device, 19]
[, 75]) ([good, 17] [ku, 42] [na-51,]
[I, 43]) ([translation, 94] [in, 56] [, 85]) ([na-i, 51] [i, 43]) The clauses are separated by parentheses. (3). Output from dictionary application device : ([device, H / W]
[, 75]) ([good, good] [ku, 42] [no, 51]
[I, 43]) ([translation, 94] [in, 56] [in, 85])
([Na-i, 51] [i, 43]) In this way, the input sentence is decomposed for each clause, and the syntax structure data of the linear list is created. Furthermore, by analyzing the dependency relationship between the clauses described later, it is possible to create the syntax structure data of the directed graph.

(4) Dependency generating device The grammar rules are that the independent word (R _sd ) and the auxiliary word (R _si ) of the dependency source node, the independent word (R _dd ) of the dependency destination and the auxiliary word (R
_di ) and the nature of the dependency (T), ｛R _sd , R
_It consists of _si , R _dd , R _di , T｝. This grammar rule is applied to the dependency source node N _n and the dependency destination node N _m (n
m), it is determined that there is a dependency relationship between N _n and N _m when the grammar rules are met, and a dependency relationship is _given from N _n to N _m . You can have any number of dependencies as long as they conform to the grammar rules. In addition, the weight of the arc can be weighted based on the properties of the accessory word and the dependency. The extracted dependency relationship is defined as an arc, and information extracted by the dictionary application device is added to each node to create a syntax tree.

3. About the pattern extraction device The pattern extraction device includes a frequent pattern extraction device and a specific pattern matching document extraction device, and each will be described below.

(1) Frequent Pattern Extraction Device Here, four words (tentatively V
a, Vb, Na, Nb) and their positional relationship as Va− * → Vb−
* → Nb, Va− * → Na In addition, Va and Vb are restricted as verbs, and Na and Nb are restricted as nouns. When such a pattern is given, the frequent pattern extraction device uses the words included in each syntax tree as Va and Na, Vb and Nb, and Va and
Has a dependency relationship of Vb, and Va Vb is a verb, Na
A set of words where Nb is a noun (Va-Vb-Na-Nb)
Search for and sum up this.

As an example for realizing, specifically, (1) A. Analyzing the parse tree, finding a verb node, examining the verb nodes that are close to that node,
Find a verb-verb pair that has a verb-to-verb dependency relationship. When there are a plurality of routes, the distance on the route having the shortest distance is set as an object of aggregation. For example, tracing the directed graph from node Va, node Vb within a certain distance
Exists, the pair of the node Va and the node Vb is targeted.
This is performed for all verb nodes on the syntax tree. For example, it is assumed that Va-Vb and Vb-Vc are found here. B. Similar to A, the parsing tree is analyzed, a noun node existing at a short distance from the verb node is examined, and a verb-noun pair having a dependency relationship is obtained. For example, here Va−
It is assumed that a pair of Na, Vb-Nb, and Vc-Nc is found. C. From the verb-verb dependency pair obtained in A and the verb-noun dependency pair obtained in B, a set of four words is obtained. For example, if Va−Vb is discovered in A and B
If Va−Na and Vb−Nb are found in FIG. 7, the set of these four words (Va−Na−Vb−Nb) is to be aggregated as shown in FIG. Similarly, (Va−Nb−Vc−Nc) is also a target of counting.

(2) The above A, B, and C are performed on all documents (syntax trees), and a frequently-used combination is output from a set of four words that are finally totaled. (3) Consider a case where a frequent pattern having a large number of elements is extracted. 6 words as pattern (Va, Vb, Vc, Na, Nb, Nc)
Consisting of Va− * → Vb− * → Vc− * → Nc, Va− * → Na,
Consider a positional relationship of Vb− * → Nb. Also, Va, Vb, Vc
Is a verb, and Na, Nb, and Nc are nouns. When such a pattern is given, similarly, in the verb-verb pair obtained in A,
It checks whether there is a pair of Va-Vb and Vb-Vc (Va has a dependency relationship with Vb and Vb has a dependency relationship with Vc), and uses the verb-noun dependency pair obtained in B to obtain a diagram. A set consisting of six words such as 8 is extracted.

(2) Specific pattern matching document extraction device A document satisfying a frequent pattern is extracted from a large number of documents and output. This means that the syntax analysis data (syntax tree data) includes all the words and attributes that make up a specific pattern, and if so, whether there is a dependency relationship between the words. Can be realized by examining

(3) Pattern Extraction from Linear List In the language analyzer, syntactic analysis data having a linear list structure is constructed as data to be applied to the dependency analyzer. Can be extracted. (1) For a column w ^{* of} a morpheme (element of a linear list) w including a weighted distance, a search range of the dependency is set from 0 to ∞. w is a set of a word t, a label 1 representing a part of speech or an attribute, and a weighted distance d to the right adjacent word (w =
[T, l, d]). At this time, if the value of the search range is 0, it means that only the word at the place where the search is started is searched, and if it is 1, it means that the preceding and following words are also candidates for the dependency. (2) The search pattern can be represented by P = <p ₁ , p ₂ ,..., _Pn >, p ₁ ,..., _Pn {{[t, l]}. Each _{p i (i = 1,2, ...} , n) is given to the set of labels l representing the term t and the part of speech or attributes, P is obtained by arranging the _{p 1, p 2, ....} , The p _n in order It is. At this time, p _i and the following p _{i + 1} must be within the dependency search range specified in (1). Also,
Patterns can also be described using regular expressions.
Sentences that match the pattern P [t, l, d] * searched from among the weighted distance d = sigma subset of linear list that matches to _{(d 1, ..., d n} ) (d _{1, ...,} d _n is the weighted distance of the word from the beginning that match the pattern to the end) is pick out what is minimized.

(3) Given a search range and a search pattern, an input word string [t, l, d] ^* (a word is composed of elements such as a name t, an attribute name l, and a distance d from the right adjacent word) ) Is the extracted information. For example, taking the sentence "The device is not bad", for example, a linear list constructed from this sentence (

), An element [device, H / W] that matches the attribute name “H / W” by the pattern P = <[*, H / W]>
W] (the distance is omitted). By P = <[*, H / W | S / W], [*, evaluation]>, the composite attribute [H / W]-[evaluation] is obtained from the text.
Alternatively, a set of elements matching [S / W]-[evaluation] is searched, and in this example, [apparatus, H / W]-[good, evaluation] can be extracted.

4. Frequent pattern display device Displays a document that has a frequent pattern found by the pattern extraction device and a syntax tree that matches it.

The effectiveness of this method was confirmed by processing 90,000 call data created in actual call center operations. Hereinafter, one specific example of the embodiment will be described. First, a morphological analysis, which is a conventional technique, is performed from each document, and a syntax tree is constructed by a dependency analyzer. As an example, a simple sentence "When turning on the power, a picture requesting a floppy disk appears." From this sentence, a syntax tree (directed graph) as shown in FIG. 9 is constructed. In this graph, a directed arc indicates a dependency relationship between words and phrases. The square at the right shoulder of the node (each phrase) indicates whether the word is a verb or a noun (N indicates a noun, V indicates a verb).

The grammar rules for creating this syntax tree are 8
There are five, and if the word of a certain node is a verb adjunct, it is as simple as performing dependency on the noun appearing after that node. In this example, the weights of the arcs are all equal to one. In the directed graph, the number of branches (the number of arcs) passed from a certain node to a certain node is defined as a distance. For example, since “power source” and “request” can be reached via two arcs, distance = 2. When there are a plurality of routes, the calculation is performed using the shortest route. As knowledge to be extracted, only those whose distance is within 3 are considered here. By shortening the distance to some extent in this way, it is possible to eliminate dependency that is assumed to have no relevance between words. When the dependency of the verb-noun is obtained from the above parse tree, a pair of words that exist in a short distance such as “exit” — “picture”, “request” — “floppy disk”, “insert” — “power”, etc. Can be taken out.

Further, when a pair is requested for the verb-verb dependency, "request"-"insert", "exit"-"insert" can be obtained. Verb sought-verb, verb-
From each pair of nouns, V1-V2, V1-N1, V2-N
In the case of the two dependency relations, a set consisting of four words, "power", "turn on", "floppy disk", "request" and "floppy disk", "request", "picture", and "exit" Can be extracted. Also, a set consisting of six words, “power”, “turn on”, “floppy disk”, “request”, “picture”, and “exit” can be extracted. By summing up the set of four words and the set of six words extracted in this way, it is possible to sum up the documents that use the same word in the same dependency structure from among a large number of documents.

A set (that is, knowledge) composed of four words having a structure of "noun 2"-"verb 2", "noun 1"-"verb 1", "verb 1"-"verb 2" is transferred to an actual call center. Let's extract it from the call record document. Knowledge consisting of four words, "extension H / W"-"remove" and "BIOS"-"return" could be extracted. The text from which this knowledge was extracted is as follows. `` Removing the additional H / W and restoring the BIOS, re-partitioning with FDISK and returning to the factory with the recovery CD '', `` Even if removing the additional H / W and restoring the BIOS and returning to the factory with the recovery CD The ISDN card cannot be used "," Resume function cannot be restored even if all extension H / W is removed and BIOS is returned to the factory setting with F5, BIOS, H / W
Considered to be a problem with W and determined that an investigation was required at the service center. "

In addition, knowledge consisting of four words "file"-"not found" and "message"-"go out" could be extracted. The text from which this knowledge was extracted is as follows. "A message appears when the program file error file cannot be found."
"The message" or required file not found "is now displayed. I want to delete the message."
"Even if I enter \ INSTALL with X, I get an error message saying" File not found "and cannot install."

In addition, "PC"-"display", "OS"
-The knowledge consisting of the six words "return", "method"-"don't know" could also be extracted. The text from which this knowledge was extracted is as follows. `` PC model A, white characters are displayed on a black screen, OS from XX mode
`` I don't know how to return to '', `` After selecting PC model A, game, command prompt is displayed and I do not know how to return to OS '', `` PC model A, Japanese DOS game icon, black screen “C in OS” is displayed in white letters and the OS
I don't know how to return to ".

In addition, knowledge consisting of six words, "power"-"turn on", "floppy disk"-"request", "picture"-"go out" could be extracted. The text from which this knowledge was extracted is as follows. "When turning on the power, a picture requesting a floppy disk appears", "I tried to check the network settings, but when I turned on the power, a picture requesting the floppy disk appeared and I could not start the OS", "When I turned on the floppy disk Comes out, and the BIOS recognizes the hard disk. "

In addition, "Internet"-"Connect", "Dialing tone"-"Inaudible", "Message"-
The knowledge consisting of the six words "go out" was also extracted. The sentence from which this knowledge was extracted is "When I try to connect to the Internet of model A, I get a message saying" I can't hear the dial tone "and it doesn't connect", "When I try to connect to the Internet," I can't hear the dial tone "Cannot connect because of the message", "" When trying to connect to the provider on the Internet of model A, a message saying "I can not hear the dial tone" appears. "

The advantages of the knowledge extraction (frequent pattern discovery) method according to the present invention are as follows: (1) Patterns that could not be obtained by the conventional data mining of co-occurrence or order relation using only keywords. Can be extracted. Further, in the related art, a pattern that is erroneously found is not found. (2) The extracted knowledge (frequent patterns) is easy for humans to understand and has excellent visibility. (3) The processing can be speeded up by using a linear list together. Etc.

[0039]

According to the present invention, knowledge that cannot be found by the conventional data mining method or that has been found by mistake can be extracted more appropriately without error. In addition, the extracted knowledge has excellent visibility and is easy for humans to understand. For example, a company call center or the like finds documents of almost the same content that appears in a large number of documents, and examines the content with a large number of appearances to create an FAQ for content that is frequently inquired by customers, By posting the information on a company homepage, it is possible to reduce the number of inquiries, or to notify the operator of the details, thereby facilitating the reduction of the time required for answers.

[Brief description of the drawings]

FIG. 1 is a diagram showing a process of creating a syntax tree from a natural language.

FIG. 2 is a diagram showing a pattern.

FIG. 3 is a diagram showing an overall configuration of the present invention.

FIG. 4 is a flowchart of a process of the present invention.

FIG. 5 is a diagram showing details of a language analyzer.

FIG. 6 is a diagram showing a pattern extraction device.

FIG. 7 is a set (pattern) of four extracted words.
FIG.

FIG. 8 is a set (pattern) of six extracted words.
FIG.

FIG. 9 is a diagram illustrating an example of a pattern.

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06F 15/401 330Z (72) Inventor Hiroshi Matsuzawa 1623-14 Shimotsuruma, Yamato-shi, Kanagawa, Japan M Co., Ltd.Tokyo Basic Research Laboratories (72) Inventor Takeshi Fukuda 1623-14 Shimotsuruma, Yamato-shi, Kanagawa Prefecture Japan IBM Co., Ltd. Tokyo Research Laboratories (72) Inventor Tetsuya Nasukawa Yamato-shi, Kanagawa Prefecture 1623 Shimotsuruma 14 Tokyo Basic Research Laboratory, IBM Japan, Ltd. (72) Inventor Tohru Nagano 1623-14 Shimotsuruma, Yamato City, Kanagawa Prefecture IBM Japan, Ltd.Tokyo Basic Research Laboratory ( 72) Inventor Masayuki Morohashi 4-1-1, Seigaoka, Tama-shi, Tokyo F-term in the Faculty of Business and Information Sciences, Tama University 5B075 ND0 3 NK31 NK32 NK43 PP24 PR04 UU40 5B091 AA15 CA02 CA05 CC01 CC02 CC05

Claims (9)

[Claims] In a method for extracting knowledge from a large amount of document data, a step of extracting words from one document by a morphological analysis technique, estimating a dependency relationship between words, and constructing a syntax tree from the dependency relationship. And a step of finding a frequent pattern included in many parse trees in the constructed parse tree based on pattern constraints, and a step of searching for a document matching the assignment to the found frequent pattern. A knowledge extraction method characterized by including: 2. The knowledge extraction method according to claim 1, wherein in the step of constructing the syntax tree, a linear list is constructed, and a frequent pattern is found using the constructed linear list. 3. In the step of finding a frequent pattern described above, extracting knowledge by searching a pattern in which a combination of a search range, a word, and a label is described using a regular expression using the linear list. Characterized by
The knowledge extraction method according to claim 2. 4. An apparatus for extracting knowledge from a large amount of document data, a language feature analyzer for registering a vocabulary not included in a basic dictionary in a field-dependent dictionary, a language analyzer for performing natural language analysis, and a restriction on patterns. A pattern extraction device that finds data that matches a specific pattern based on the data, and a frequent pattern display device that displays the extracted frequent patterns, a basic dictionary for general fields from document data, and a generation for phrase generation processing Perform knowledge extraction with reference to rules, generation rules for syntax tree generation, and field-dependent dictionaries,
A knowledge extraction device characterized by the following. 5. The linguistic feature analysis device divides an input document into a word string with a part of speech using a morphological analysis dictionary, deletes words already registered from the word string using a field-dependent dictionary, and 5. The knowledge extracting apparatus according to claim 4, further comprising means for calculating the appearance frequency of the words, rearranging the words in the order of the frequency, and additionally registering the words in a field-dependent dictionary. 6. A morphological analyzer, wherein the language analyzer is a morphological analyzer,
Includes a phrase generation device, a dictionary application device, and a dependency analysis device, and parses in the form of a linear list and a syntax tree in consideration of distance, dependency, and label according to a phrase generation rule and a syntax tree generation rule. Means for generating data, wherein the morphological analysis device divides the input document into words using morphological analysis, adds labels including parts of speech or attributes, and unifies the expression using a synonym dictionary. Including,
The knowledge extraction device according to claim 4, wherein: 7. The pattern extraction device includes a frequent pattern extraction device and a specific pattern matching document extraction device, and the frequent pattern extraction device uses a syntax analysis data to generate a word, a positional relationship between words, and a label. Based on the combination of, including a means for examining co-occurrence relations and extracting frequent patterns, the specific pattern matching document extraction device determines whether or not the syntax analysis data includes a word constructing a specific pattern, an attribute, 5. The knowledge extraction apparatus according to claim 4, further comprising means for extracting a document that matches the frequent pattern by checking whether or not there is a dependency relationship between the phrases, and outputting the extracted document. apparatus. 8. The knowledge according to claim 4, wherein the frequent pattern display device includes display means for displaying a document having a frequent pattern found by the pattern extraction device and a syntax tree matching the frequent pattern. Extraction device. 9. A program for extracting knowledge from a large amount of document data, extracting words from one document by a morphological analysis technique, estimating a dependency relationship between words, and constructing a syntax tree from the dependency relationship. And a step of finding a frequent pattern included in many parse trees in the constructed parse tree based on pattern constraints, and a step of searching for a document matching the assignment to the found frequent pattern. , A computer-readable medium storing a program for causing a computer to execute the program.