JP2001344256A - Word class automatic determination device, example sentence retrieval device, medium, and information aggregate - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem that it is impossible to always select only a meaning code proper for a condition of sentence input and improper word classes are frequently generated to degrade the example retrieval performance in the case that a synonym dictionary like a thesaurus dictionary having equivocality is used to generate word classes and such a problem that development requires much labor and time in the case where a word class proper for the condition of sentence input is manually set, and also such a problem that it is impossible to give a proper class to a word which doesn't exist in an example sentence in the case of automatic construction from an example sentence corpus, in a conventional example retrieval means. SOLUTION: Meaning codes in a synonym dictionary 4 and inter-word relations analyzed from an example sentence corpus 1 are used together to automatically construct word classes, and thus word classes properly corresponding to conditions of sentence input are automatically constructed while utilizing merits of the synonym dictionary which can give classes to words which don't exist in example sentences.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an automatic word classifier for performing word clustering used for natural language processing such as sentence structure analysis and intention analysis, and speech language processing such as speech recognition and speech synthesis, and an example sentence retrieval apparatus. , Media, and information aggregates.

[0002]

2. Description of the Related Art A similar example sentence retrieval apparatus to which the prior art is applied will be described below as an example. A similar example search device is a device that searches for an example sentence similar to an input sentence. Recently, translation and interpreting technology, document editing, summarization technology,
It is widely used as one element of speech language and natural language processing technology.

[0003] For example, in interpreting software, it is difficult to analyze a flank expression that deviates from the grammatical rules, such as is often found in spoken words, by linguistic analysis in accordance with grammatical rules handled in written language translation. Therefore, in many cases, a technique of learning an example sentence similar to an actually input sentence and performing language analysis by searching for a similar example is used.

In such a similar example sentence search, even if a word that does not exist in the learning example sentence is input, similar words are classified into classes in advance so that other words included in the same class can be used as example sentences. If it is included in, it treats it in the same way as the word, and introduces a system that can search for examples with high performance.

In the prior art, even when sentence input is performed in a very limited situation, even if it is necessary to construct a word class according to the situation, it is very difficult to construct a word class. There is a problem that development costs are required. In order to reduce such development costs, the word class includes a general thesaurus dictionary built over many years based on knowledge of the word concept accompanying the common sense of the builder in advance. There is often used a "vocabulary classification table in the form of a classified vocabulary table" issued by the National Language Institute, etc.). (For example, Furuse, Sumida, Iida: Transactions of Information Processing Society of Japan Vol 35, no3, 1994-3).

FIG. 4 shows a conventional example retrieval apparatus 14 using word classes. Before performing the search, in order to construct an example sentence DB from the example sentence corpus 13 in advance, a dependency structure analysis of each example sentence is performed, and the result is stored for each sentence.

At the time of example sentence retrieval, a morphological analysis of an input sentence is performed by a morphological analysis 10, and a content word is extracted by a content word extraction 11. Then, in the example sentence selection 12, the content word included in the input sentence is compared with the content word included in the example sentence corpus 13, and an example sentence including a content word similar to the input sentence is determined as a similar example sentence. I do. At this time, if the content words included in the input sentence are not in the example sentence corpus 13, the content words in the example sentence and the input sentence are converted into the meaning codes according to the thesaurus using the thesaurus dictionary 4, and the words having the same meaning code are converted. Is a word class, and if it is included in the same word class, it is regarded as the same word and an example search is performed.

[0010] The example search device 14 enables structural analysis of an input sentence of a spoken language sentence that cannot be handled by the conventional grammatical rules for written words, and also enables similar example sentences to be searched. Further, even if a word that is not in the example sentence is input, a similar example search can be performed without using a development cost for constructing a word class by using a thesaurus dictionary.

[0009]

However, in the conventional example search apparatus, since a word class is created using a semantic code in a thesaurus dictionary, there is a problem of polysemy occurring in the thesaurus dictionary. . That is, since the thesaurus corresponds to all various situations, many words have a plurality of meaning codes. Therefore, it is not always possible to select only the appropriate semantic code for the input sentence,
There is a problem that forming an inappropriate word class frequently lowers the example search performance.

[0010] In addition to thesaurus dictionaries,
The same can be said when a word class is created using a thesaurus that words are classified based on semantic similarity.

In view of the above problems, the present invention automatically constructs a word class using both the meaning codes of a thesaurus such as a thesaurus and the relationship between words analyzed from an example sentence corpus. A word class automatic determination device, an example sentence search device, a medium, and an information set that can automatically construct a word class appropriately corresponding to a situation in which a sentence is input while utilizing the advantages of a thesaurus such as a thesaurus that can classify nonexistent words. It is intended to provide the body.

[0012]

In order to solve the above-mentioned problems, a first aspect of the present invention (corresponding to claim 1) is that a word-dependent relationship between words extracted from a learning example corpus, An automatic word class determining apparatus comprising: a word class automatic determining unit that performs word clustering that regards similar words as being of the same class using a thesaurus that is classified based on a similarity. .

Further, a second aspect of the present invention (corresponding to claim 2)
Comprises dependency structure analysis means for extracting the dependency relationship by performing dependency structure analysis between words in the learning example corpus, wherein the word class automatic determination means matches words having the same semantic code in the thesaurus to the same. An initial class creating means for creating an initial class as a class,
Based on the extracted dependencies, words that depend on words that are different from any other words belonging to the same initial class among the created initial classes are excluded from the same initial class, and another word is removed. An automatic word class determining apparatus according to the first aspect of the present invention, further comprising a clustering unit that regards the class as a class.

A third aspect of the present invention (corresponding to claim 3)
Comprises dependency structure analysis means for extracting the dependency relationship by performing dependency structure analysis between words in the learning example corpus, wherein the word class automatic determination means sets the extracted words as one word and one class. An initial class creating means for creating a class, and a condition that a plurality of words belonging to each of the created initial classes depending on the same word have the same semantic code in the thesaurus, based on the extracted dependency. Only when the condition is satisfied, words belonging to the thesaurus that have the semantic code are set to the same class, and a word that does not satisfy the condition is provided with a clustering unit that does not belong to the same class. Is a word class automatic determination device.

A fourth aspect of the present invention (corresponding to claim 4)
Wherein the clustering means removes, from the same class, words that are dependent on a word different from any other words belonging to the same class among words included in the same class. 3 is an automatic word class determining apparatus according to the present invention.

A fifth aspect of the present invention (corresponding to claim 5)
A morphological analysis unit for performing morphological analysis on an input sentence and extracting a content word included in the input sentence, and an example sentence including a content word the same as or similar to the extracted content word. From the example sentence corpus, and the same or similar content word is the same content word as the extracted content word, or the same content word as the extracted content word. If not included in the example sentence corpus, it is a content word belonging to the same class as the class to which the extracted content word belongs, and the class includes the word class automatic class according to any of the first to fourth aspects of the present invention. An example sentence search device is characterized in that the class determined by the determination device is used.

The sixth invention (corresponding to claim 6)
Is a medium carrying a program and / or data for causing a computer to execute all or some of the functions of all or some of the means for automatically determining a word class according to any of the first to fourth aspects of the present invention. And a medium that can be processed by a computer.

The seventh invention (corresponding to claim 7)
Is a program and / or data for causing a computer to execute all or a part of the functions of all or part of the automatic word class determining device according to any one of the first to fourth aspects of the present invention. An information aggregate that is a feature.

[0019]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) First, a first embodiment will be described.

FIG. 1 is a block diagram of the automatic word class determining device 7 according to the first embodiment of the present invention, and FIG. 2 is a flowchart of the automatic word class determining device 7. FIG.
Shows a specific example when the word class is determined.

As shown in FIG. 6, as an example sentence for learning,
There are "Please choose wine", "I want to drink wine", "I want to drink soda", "Please give me juice", "Please give me iced coffee", and "Is alcohol included?"

First, before determining a word class, the dependency structure analysis unit 2 analyzes the dependency structure of each sentence of the learning example stored in the example sentence corpus 1. For example, the dependency structure analysis uses a semantic structure analysis using a case frame. From the analyzed dependency structure, a content word pair having a dependency / non-dependency relationship is recorded in the content word dependency pair table 5.

In the example of FIG. 6, as a result of the dependency structure analysis of the example sentence corpus 1 shown in (a), a content word dependency pair table 5 shown in (b) is obtained.

Here, in the flowchart of FIG. 2, a dependent word is a word that is clustered to determine a word class, and a dependent word is a word on which the dependent word depends.
Specifically, in the learning sentence “I want to drink wine,” if “Wine” is classified, “Wine” is a dependent word, and the dependent word on which “Wine” depends is “Drinking. ". Also, in the example sentence for learning "Does alcohol exist?", If "alcohol" is classified, "alcohol" is a dependent word, and the dependent word on which "alcohol" depends is "enter". is there.

Next, a state in which the word class determination unit 3 classifies all content words according to the meaning codes of the thesaurus is set as an initial class (S1). That is, FIG.
In the example, class 3 in the thesaurus in (c)
0, wine, soda, juice, iced coffee,
Alcohol and milk make up the initial class.

Next, words that do not depend on the same word as any of the words belonging to the same class are removed from the word class from the content word dependency pair table 5 having the dependency analyzed earlier, and the word alone is removed. To be one class. With respect to all the content word pairs, the result of searching for a word to be excluded from the class under the above conditions is determined as a word class (S2, S3, S4, S
5, S6). Then, the determined word class is stored in the word class table 6.

In the example of FIG. 6, the only word that depends on "enter" is "alcohol", and no other words belonging to the initial class depend on "enter". Therefore, “alcohol” is removed from the initial class, and “alcohol” alone is classified into one class. Therefore, the word class table 6 contains
As shown in FIG. 6C, the class A31 and the class B3
2 is stored.

In the above description, all content words are classified into classes according to the thesaurus semantic codes. However, the thesaurus semantic codes are applied only to the dependent words of the content word pairs. Alternatively, the initial state may be a situation in which the dependent word is a single class of the word alone.

As described above, in the initial state, words having the same meaning code in the thesaurus are in the same word class, and words used in a special meaning depending on the input state of the sentence are excluded from the example sentence from the initial class. By treating words alone as one class, it is possible to classify words that do not exist in the example sentence, while distinguishing words that have special meaning depending on the learning example corpus Therefore, even if the example sentence corpus is small, a word class appropriate for a situation where a sentence is input can be determined.

(Second Embodiment) Next, a second embodiment will be described.

FIG. 1 is a block diagram of an automatic word class determining apparatus of the present embodiment, which is the same as that of the first embodiment. FIG. 3 is a flowchart of the automatic word class determination device 7. FIG. 7 shows a specific example of determining a word class.

Also in this embodiment, as in the first embodiment, the dependency structure analysis of the example sentence is performed in advance to extract the dependent content word pairs. In FIG. 7, as a result of the dependency structure analysis of the example sentence corpus 1 shown in (a), a content word dependency pair table 5 shown in (b) is obtained.

When the word class is determined by the word class determination unit 3, first, a single word alone class is set to an initial state for all words (S7). In the example of FIG. 7, as shown in the initial state 35 of FIG.

Next, as a result of examining the previously analyzed content word pairs, if words dependent on the same word have the same meaning code in the thesaurus (S8, S9), these words belong to the same word. It is assumed that all the words belonging to the meaning codes on the thesaurus belong to the same word class (S11).
In the example of FIG. 7, “juice” and “soda” both depend on “request”. It is also assumed that “juice” and “soda” have the same meaning code in the thesaurus. Then, the words belonging to the meaning codes on the thesaurus to which "juice" and "soda" belong are set to the same word class as shown in the class 30 of the thesaurus in (d).

All words that do not satisfy the above conditions are 1
One word class (S8, S9, S10).

As described above, by classifying words that do not exist in the example sentence by setting the initial state as a word-only class and collecting only semantic codes on a thesaurus whose dependency relation is clear as a word class. However, since words with special meanings can be distinguished and handled depending on the learning example corpus, even if the example sentence corpus is small, a word class appropriate for the situation where a sentence is input Can be determined.

Further, after the word class is determined in this way, the following two conditions are checked for a word regarded as a word class, and a word to be separated from the word class is selected. (Condition 1) If a word that depends on the same word as the one on which it depends does not belong to the same class, the corresponding word is excluded from the word class, and is set as one word per class. (Condition 2) A word that depends on the same word as the word on which it depends depends on a word that is different from any word belonging to the same class while the word that depends on the same class exists in this class. , But it is also treated as one class per word. Under these two conditions, a word that satisfies the above condition is determined as another class from a word class that follows the meaning code of the thesaurus.

The determined class is stored in the word class table 6 as in the first embodiment.

In the example of FIG. 7, “wine” is replaced by “soda”
It relies on "choice", which does not depend on any other words shown in class 30 in the thesaurus, while relying on the same independent word "drink". Therefore, "wine" satisfies the above condition 2, and is included in the word class 30 in the thesaurus and is also treated as one word and one class.

Also, "alcohol" depends on "enter", but none of the other words in class 30 in the thesaurus depend on "enter". Therefore, since "alcohol" satisfies condition 1,
Removed from word class 30 in the thesaurus, another 1
Treat as word 1 class.

In this way, three classes are obtained, such as class A31, class B32, and class C33 in (e).

In this way, the word state is set by setting the initial state as a word-only class and grouping only semantic codes on the thesaurus whose dependency relationship is clear as a word class. Dependent on the learning example corpus while allowing words that do not exist in the example sentence to be classified into words that do not exist in the example sentence Thus, even if the example sentence corpus is small, it is possible to determine a word class appropriate for a situation where a sentence is input, since words having a special meaning can be reliably distinguished and handled.

The word class determined in the above embodiment can be used for natural language processing such as sentence structure analysis and intention analysis, and speech language processing such as speech recognition and speech synthesis.

As an example, FIG. 5 shows an example sentence search device 24 using the word class determined in the above embodiment as the class dictionary 23.

The example sentence retrieval device 24 first performs a morphological analysis on the sentence input by the morphological analyzer 20. Next, a content word is extracted by the content word extracting means 21 using the result of the morphological analysis. Further, an example sentence including the same or similar content word as the content word extracted by the example sentence selection unit 22 is selected from the example sentence corpus 1. Here, the same or similar content word is the same content word as the extracted content word, or if the same content word as the extracted content word is not included in the example sentence corpus 1, the extracted content word This is a content word of the example sentence corpus 1 belonging to the same class as the class to which the word belongs.

As described above, by using the word class determined in the above embodiment, a high-performance example sentence search device 24 can be provided.

It should be noted that the word class determining section 3 of the present embodiment
Is an example of a word class automatic determination means of the present invention, an example sentence corpus 1 of the present embodiment is an example of a learning example corpus of the present invention, and a thesaurus dictionary 4 of the present embodiment is an example of a thesaurus of the present invention. The word class determining unit 3 of the present embodiment is an example of the initial class determining unit of the present invention, and the word class determining unit 3 of the present embodiment also serves as an example of the clustering unit of the present invention. The dependency structure analysis unit 2 according to the embodiment is an example of a dependency structure analysis unit according to the present invention. Also, the morphological analysis unit 20 and the content word extraction unit 21 of the present embodiment are examples of the morphological analysis unit of the present invention, and the example sentence selection unit 22 of the present embodiment is an example of the example sentence search unit of the present invention. Yes, the class dictionary 23 of the present embodiment is an example of a class of the present invention, and the thesaurus dictionary 4 of the present embodiment is an example of a synonym dictionary of the present invention.

As described in detail above, the automatic word class determining apparatus according to the present embodiment automatically constructs a word class using both the meaning code of the thesaurus and the relationship between words analyzed from the example sentence corpus. With the advantage of a thesaurus that can classify words that do not exist in example sentences,
It is possible to automatically construct a word class appropriately corresponding to a situation where a sentence is input.

Also, the automatic word class determining apparatus according to the present embodiment sets the one having the same meaning code in the thesaurus as an initial class, analyzes the dependency structure between words in the learning example corpus, and finds that it belongs to the same initial class. By removing words that depend on words that are different from any other words in the initial class and treating them as another word class,
By utilizing the advantages of a thesaurus that can classify words that do not exist in example sentences, it is possible to automatically construct an appropriate word class that reflects the usage of special words depending on the situation in which the sentence is input.

Also, the automatic word class determining apparatus of the present embodiment performs a dependency structure analysis between words in the learning example corpus with one word and one class as an initial state, and as a result, a plurality of words dependent on the same word are found. Only when the thesaurus has the same meaning code in the thesaurus, words with this meaning code are in the same class, and words that do not meet the above conditions are not classified, reflecting the usage of words depending on the situation where the sentence is input However, an appropriate word class can be assigned to a word that does not exist in the corpus.

Further, the word class automatic determining apparatus of the present embodiment has the same dependency relationship as one of the words included in the class determined as described above with other words included in the same class. If the word does not have it, by removing it from the class, an appropriate word class can be given to a word that does not exist in the corpus while reflecting the usage of the word depending on the situation in which the sentence is input.

The thesaurus of the present invention is not limited to the thesaurus dictionary 4 in the present embodiment, in which words are classified into a tree structure. For example, words are grouped into groups 40 to 43 as shown in FIG.
Words belonging to each group may have a meaning code added to the group to indicate the meaning. In short, the synonym dictionary of the present invention only needs to classify words based on semantic similarity.

Further, a medium carrying a program and / or data for causing a computer to execute all or a part of the functions of all or a part of the automatic word class determining apparatus of the present invention, wherein the medium is processed by the computer A medium characterized by the possibility also belongs to the present invention.

Further, there is also provided an information aggregate which is a program and / or data for causing a computer to execute all or a part of the functions of all or a part of the automatic word class determining apparatus of the present invention. It belongs to the present invention.

Further, the data of the present invention includes a data structure, a data format, a data type, and the like. The medium of the present invention includes a recording medium such as a ROM, a transmission medium such as the Internet, and a transmission medium such as light, radio waves, and sound waves. The medium carried by the present invention includes, for example, a recording medium on which a program and / or data is recorded, a transmission medium for transmitting the program and / or data, and the like.
In addition, the term “processable by the computer” of the present invention means that it can be read by a computer if it is a recording medium such as a ROM, and if it is a transmission medium, it can be a program and / or a program to be transmitted. Or that the data can be handled by a computer as a result of the transmission. The information aggregate of the present invention includes, for example, software such as a program and / or data.

Further, a program recording medium for recording a program and / or data for causing a computer to execute all or a part of the functions of all or a part of the automatic word class determining apparatus of the above-described embodiment is a computer. And the read program and / or data may be a program recording medium that executes the function in cooperation with the computer.

[0058]

As is apparent from the above description, the present invention makes use of the advantages of a thesaurus that can classify words that do not exist in example sentences, and provides a word class that appropriately responds to situations where sentences are input. Can automatically provide a word class automatic determination device, an example sentence search device, a medium, and an information aggregate.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an automatic word class determination device according to first and second embodiments of the present invention.

FIG. 2 is a flowchart showing the operation of the automatic word class determination device according to the first embodiment of the present invention.

FIG. 3 is a flowchart showing the operation of the automatic word class determining apparatus according to the second embodiment of the present invention.

FIG. 4 is a block diagram showing a configuration of a conventional example search device.

FIG. 5 is a block diagram showing an example of an example sentence search device using a word class determined by the word class automatic determination device according to the first embodiment or the second embodiment of the present invention;

FIG. 6 is a diagram showing an example of class determination according to the first embodiment of the present invention.

FIG. 7 is a diagram showing an example of class determination according to the second embodiment of the present invention.

FIG. 8 is a diagram showing an example of a synonym dictionary according to the first and second embodiments of the present invention.

[Explanation of symbols]

 1: Example sentence corpus 2: Dependency structure analysis unit 3: Word class determination unit 4: Thesaurus dictionary 5: Content word dependence pair table 6: Word class table

Claims (7)

[Claims] 1. A word that regards similar words as the same class using a dependency relationship between words extracted from the learning example corpus and a thesaurus that the words are classified based on the semantic similarity. An automatic word class determining apparatus comprising an automatic word class determining means for performing clustering. 2. A method according to claim 1, further comprising a dependency structure analysis unit for extracting the dependency relationship by performing a dependency structure analysis between words in the learning example corpus, wherein the word class automatic determination unit has the same meaning code in the thesaurus. Initial class creating means for creating an initial class with the word as the same class; and, based on the extracted dependencies, depending on a word different from any other word belonging to the same initial class among the created initial classes. 2. The automatic word class determining apparatus according to claim 1, further comprising: a clustering unit that removes the word in question from the same initial class and regards the word as another word class. 3. Dependency structure analysis means for extracting the dependency relationship by performing dependency structure analysis between words in the learning example corpus, wherein the word class automatic determination means converts the extracted words into one word Initial class creating means for creating an initial class as a class, based on the extracted dependencies, a plurality of words belonging to each of the created initial classes depending on the same word have the same meaning code in the thesaurus. 2. The method according to claim 1, further comprising: a clustering unit that sets words belonging to the thesaurus to have the same meaning code in the same class only when the condition is satisfied, and does not set the words that do not satisfy the condition to the same class. Word class automatic determination device. 4. The method according to claim 1, wherein the clustering unit removes, from the same class, words that are dependent on words different from any other words belonging to the same class among words included in the same class. 4. The automatic word class determination device according to claim 3, wherein 5. A morphological analysis unit for performing morphological analysis on an input sentence and extracting a content word included in the input sentence, wherein a morphological analysis unit includes a content word that is the same as or similar to the extracted content word. An example sentence search means for selecting an example sentence to be used from an example sentence corpus, wherein the same or similar content word is the same content word as the extracted content word, or the same as the extracted content word. If the content word is not included in the example sentence corpus,
It is a content word belonging to the same class as the class to which the extracted content word belongs, and a class determined by the word class automatic determination device according to any one of claims 1 to 4 is used as the class. An example sentence retrieval device, characterized in that: 6. A program and / or data for causing a computer to execute all or a part of the functions of all or a part of the automatic word class determining apparatus according to claim 1. A medium that can be processed by a computer. 7. A program and / or data for causing a computer to execute all or a part of functions of all or a part of the automatic word class determining device according to claim 1. An information aggregate characterized by