CN111295661A - Word sense disambiguation method and apparatus, word sense expansion method, device and apparatus, computer readable storage medium - Google Patents
Word sense disambiguation method and apparatus, word sense expansion method, device and apparatus, computer readable storage medium Download PDFInfo
- Publication number
- CN111295661A CN111295661A CN201880071178.1A CN201880071178A CN111295661A CN 111295661 A CN111295661 A CN 111295661A CN 201880071178 A CN201880071178 A CN 201880071178A CN 111295661 A CN111295661 A CN 111295661A
- Authority
- CN
- China
- Prior art keywords
- word
- training
- target
- determining
- disambiguation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/237—Lexical tools
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Machine Translation (AREA)
Abstract
A sense disambiguation method and apparatus based on hypernyms, a sense expansion method and apparatus using the sense disambiguation method, and a computer-readable storage medium. The word sense disambiguation method comprises the following steps: receiving an input sentence (S101); determining a disambiguation target word in the input sentence based on a predetermined ambiguous word bank (S102); determining related words of the target word based on a syntactic analysis and a context information analysis of the input sentence (S103); determining one or more hypernyms of the related word (S104); and determining the word meaning of the target word in the input sentence based on the word shapes, the parts of speech and the syntactic relation with the target word of the related word and the one or more hypernyms (S105).
Description
The present application claims priority to chinese patent application No. 201711048364.7, filed 2017, month 10, and day 31, the disclosure of which is incorporated herein by reference in its entirety.
The present invention relates to the field of artificial intelligence, and more particularly, to a word sense disambiguation method and apparatus, a word sense expansion method and apparatus using the word sense disambiguation method, and a computer-readable storage medium.
Word Sense Disambiguation (WSD) refers to determining the word sense of a multi-sense word in a particular context of natural language. Word sense disambiguation is a fundamental problem in the field of natural language processing. When an ambiguous word exists in a sentence to be processed in natural language, if the correct sense of the ambiguous word in the context of the sentence cannot be correctly determined, word ambiguity occurs, thereby seriously affecting the correct understanding and processing of the natural language by the machine. In natural language based applications such as language recognition, machine translation, information retrieval, text classification, automatic summarization, etc., there is a need to solve the problem of word sense disambiguation for ambiguous words.
Currently, word sense disambiguation schemes based on corpora mainly include supervised and unsupervised approaches. The unsupervised method does not need to train a corpus, but the disambiguation precision of the unsupervised method cannot meet practical requirements. The existing supervision method needs a large-scale high-quality corpus to train the disambiguation model, and once words which are not covered by the corpus appear in the actual sentence to be disambiguated, the situation that the ambiguous words cannot be determined is likely to occur.
Disclosure of Invention
In view of the above problems, the present invention provides a word sense disambiguation method and apparatus, a word sense expansion method and apparatus using the word sense disambiguation method, and a computer readable storage medium.
According to an embodiment of the present invention, there is provided a word sense disambiguation method including: receiving an input sentence; determining a disambiguation target word in the input sentence based on a predetermined ambiguity word bank; determining related words of the target words based on the syntactic analysis and the context information analysis of the input sentence; determining one or more hypernyms of the related word; and determining the word meaning of the target word in the input sentence based on the related word and the one or more hypernyms.
Further, according to a word sense disambiguation method of an embodiment of the present invention, the determining related words of the target word based on the syntactic analysis and the context information analysis of the input sentence includes: determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; and determining related words of the target words according to a preset rule based on the part of speech, the result of the syntactic analysis, the result of the context analysis of the target words and the like.
Furthermore, the word sense disambiguation method according to an embodiment of the present invention further comprises a word sense disambiguation module pre-trained to perform the word sense disambiguation method, wherein training the word sense disambiguation module comprises: marking training data for training; performing data processing on the training data and obtaining the predetermined ambiguous word bank; for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank; determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence; determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; and training the word sense disambiguation module using the training features.
According to another embodiment of the present invention, there is provided a word sense expansion method including: receiving an input sentence; determining a disambiguation target word and a non-ambiguous word in the input sentence based on a predetermined ambiguous word bank; determining a word sense of the disambiguation target word in the input sentence using a word sense disambiguation module; determining synonyms and hypernyms corresponding to word senses of the non-ambiguous word and the disambiguation target word respectively based on a predetermined synonym library; and expanding the input sentence by using the synonym and the hypernym, wherein the determining the word sense of the disambiguation target word in the input sentence by using the word sense disambiguation module comprises: determining related words of the target words based on the syntactic analysis and the context information analysis of the input sentence; determining one or more hypernyms of the related word; and determining the word meaning of the target word in the input sentence based on the related word and the one or more hypernyms.
Further, according to a word sense expansion method of another embodiment of the present invention, the determining related words of the target word based on the syntactic analysis and the context information analysis of the input sentence includes: determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; and determining related words of the target words according to a preset rule based on the part of speech, the result of the syntactic analysis, the result of the context analysis of the target words and the like.
Furthermore, a word sense expansion method according to another embodiment of the present invention further includes a word sense disambiguation module that is pre-trained to perform the word sense disambiguation method, wherein training the word sense disambiguation module includes: marking training data for training; performing data processing on the training data and obtaining the predetermined ambiguous word bank; for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank; determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence; determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; and training the word sense disambiguation module using the training features.
According to still another embodiment of the present invention, there is provided a word sense disambiguation apparatus including: a receiving unit configured to receive an input sentence; a target word determination unit configured to determine a disambiguation target word in the input sentence based on a predetermined ambiguous word bank; a related word determination unit configured to determine a related word of the target word based on a syntactic analysis and a context information analysis of the input sentence; a hypernym determination unit configured to determine one or more hypernyms of the related words; and a word sense disambiguation unit configured to determine a word sense of the target word in the input sentence based on the related word and the one or more hypernyms.
Furthermore, a word sense disambiguation apparatus according to still another embodiment of the present invention, wherein the related word determining unit is further configured to: determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; and determining related words of the target words according to a predetermined rule based on the parts of speech, the result of the syntactic analysis, the result of the context analysis of the target words, and the like.
Furthermore, the word sense disambiguation apparatus according to still another embodiment of the present invention further includes a training unit configured to: marking training data for training; performing data processing on the training data and obtaining the predetermined ambiguous word bank; for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank; determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence; determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; and training the word sense disambiguation unit using the training features.
According to still another embodiment of the present invention, there is provided a word sense expansion apparatus including: a receiving module configured to receive an input sentence; a target word determination module configured to determine a disambiguation target word and a non-ambiguous word in the input sentence based on a predetermined ambiguous word bank; a word sense disambiguation module configured to determine a word sense of the disambiguation target word in the input sentence; a word sense expansion module configured to determine synonyms and hypernyms corresponding to word senses of the non-ambiguous word and the disambiguation target word, respectively, based on a predetermined synonym library; and expanding the input sentence using the synonyms and hypernyms, wherein the word sense disambiguation module is further configured to include: a related word determination unit configured to determine a related word of the target word based on a syntactic analysis and a context information analysis of the input sentence; a hypernym determination unit configured to determine one or more hypernyms of the related words; and a word sense disambiguation unit configured to determine a word sense of the target word in the input sentence based on the related word and the one or more hypernyms.
Further, according to a further embodiment of the present invention, the word sense expanding device, wherein the related word determining unit is further configured to: determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; and determining related words of the target words according to a predetermined rule based on the parts of speech, the result of the syntactic analysis, the result of the context analysis of the target words, and the like.
Furthermore, the word sense expansion apparatus according to still another embodiment of the present invention further includes a training module configured to: marking training data for training; performing data processing on the training data and obtaining the predetermined ambiguous word bank; for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank; determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence; determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; and training the word sense disambiguation unit using the training features.
According to still another embodiment of the present invention, there is provided a word sense disambiguation apparatus including: a processor; and a memory configured to store computer program instructions; wherein the computer program instructions, when executed by the processor, cause the processor to perform a word sense disambiguation method.
According to still another embodiment of the present invention, there is provided a word sense expanding apparatus including: a processor; and a memory configured to store computer program instructions; wherein the computer program instructions, when executed by the processor, cause the processor to perform a word sense expansion method.
According to yet another embodiment of the invention, a computer-readable storage medium is provided, having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, cause the processor to perform a word sense disambiguation method.
According to yet another embodiment of the present invention, there is provided a computer-readable storage medium having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, cause the processor to execute a word sense extension method.
According to the word sense disambiguation method and device, the related words of the disambiguation target words are determined through syntactic analysis, the related words are expanded to the hypernyms of the words, the word sense of the disambiguation target words is determined by considering the related words and the hypernyms of the related words, and dependence on the size of a training corpus is greatly reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and are intended to provide further explanation of the claimed technology.
The above and other objects, features and advantages of the present invention will become more apparent by describing in more detail embodiments of the present invention with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings, like reference numbers generally represent like parts or steps.
FIG. 1 is a flow diagram illustrating a method of word sense disambiguation according to an embodiment of the invention;
FIG. 2 is a flow diagram further illustrating a method of word sense disambiguation according to an embodiment of the invention;
FIG. 3 is a flow diagram illustrating a method of training of a word sense disambiguation module according to an embodiment of the invention;
FIG. 4 is a block diagram illustrating a word sense disambiguation apparatus according to an embodiment of the invention;
FIG. 5 is a flow diagram illustrating a word sense expansion method according to an embodiment of the invention;
FIG. 6 is a block diagram illustrating a word sense expansion apparatus according to an embodiment of the present invention;
FIG. 7 is a diagram illustrating a word sense expansion process according to an embodiment of the invention;
FIG. 8 is a hardware block diagram illustrating a word sense disambiguation apparatus according to an embodiment of the invention;
FIG. 9 is a hardware block diagram illustrating a word sense expansion apparatus according to an embodiment of the present invention; and
fig. 10 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present invention.
In order to make the objects, technical solutions and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. It is to be understood that the described embodiments are merely a subset of embodiments of the invention and not all embodiments of the invention, with the understanding that the invention is not limited to the example embodiments described herein. All other embodiments, which can be derived by a person skilled in the art from the embodiments described in the present application without inventive step, shall fall within the scope of protection of the present application.
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. A word sense disambiguation method, a training party of a word sense disambiguation module implementing the word sense disambiguation method, and a word sense disambiguation apparatus using the word sense disambiguation method according to embodiments of the present invention will first be described with reference to fig. 1 to 4.
FIG. 1 is a flow diagram illustrating a method of word sense disambiguation according to an embodiment of the invention. As shown in fig. 1, a word sense disambiguation method according to an embodiment of the invention includes the following steps.
In step S101, an input sentence is received. In an embodiment of the invention, for example, the sentence "his martial arts are high" is received. Thereafter, the process proceeds to step S102.
In step S102, disambiguation target words in the input sentence are determined based on a predetermined ambiguous word bank. In an embodiment of the invention, the ambiguous lexicon is generated for a training corpus in a training phase to be described later. And for the input sentence, determining the ambiguous words in the ambiguous word bank as disambiguation target words by searching the ambiguous word bank. For example, for the sentence "his martial arts are high" received in step S101, "high" is determined as the disambiguation target word. The disambiguation target word "high" has different word senses of "super, super strong" and "high pick". Thereafter, the process proceeds to step S103.
In step S103, a related word of the target word is determined based on the syntactic analysis and the context information analysis of the input sentence. Hereinafter, how to determine a related word of a target word based on a syntactic analysis of an input sentence and a context information analysis will be described in detail with reference to fig. 2. For example, for the sentence "his martial arts are high" received in step S101, after "high" is determined as the disambiguation target word in step S102, it is determined as the related word of the target word "high" in step S103. Thereafter, the process proceeds to step S104.
In step S104, one or more hypernyms of the related word are determined. For example, in the case where "high" is used as the disambiguation target word and "martial arts" is used as the related word of the target word "high", the hypernym "talent, talent" of the related word "martial arts" is determined. Thereafter, the process proceeds to step S105.
In step S105, the word sense of the target word in the input sentence is determined based on the related word and the one or more hypernyms. For example, based on the related word "martial arts" and the hypernym "talent, talent art", it is easy to determine that the meaning of "high" and "talent, talent art" is "high, super strong" rather than "high picking".
By the word sense disambiguation method according to the embodiment of the invention described with reference to fig. 1, through syntactic analysis and context information analysis of the input sentence, related words of the target word are determined, and the related words are expanded to the superior words thereof, so that the determination of the disambiguation target word sense is realized by considering the related words and the superior words thereof, and the dependence on the size of the training corpus is greatly reduced. For example, even if the related word "martial art" does not appear in the training corpus with a small size, the correct meaning of the target word "high" in the sentence can be correctly determined only by the hypernym "of the related word" martial art ". If the related words are not expanded to the hypernyms, it is likely that the word senses of the target words cannot be correctly determined because the related words do not appear in the training corpus of a limited scale.
FIG. 2 is a flow diagram further illustrating a word sense disambiguation method according to an embodiment of the invention. As shown in fig. 2, a word sense disambiguation method according to an embodiment of the invention includes the following steps.
In step S200, the word sense disambiguation module is trained. In an embodiment of the present invention, a Support Vector Machine (SVM) classifier may be used as the word sense disambiguation module, and thus training needs to be performed on the word sense disambiguation module using a training corpus before performing the word sense disambiguation method. Hereinafter, a training method of the word sense disambiguation module according to an embodiment of the present invention will be described in detail with reference to fig. 3. After the trained word sense disambiguation module is obtained, the process proceeds to step S201.
Steps S201 and S202 in fig. 2 are the same as steps S101 and S102 described above with reference to fig. 1, respectively, and a repetitive description thereof will be omitted. Thereafter, the process proceeds to step S203. Steps S203 and S204 are specific steps of the related word process of determining the target word in step S103 described with reference to fig. 1.
In step S203, the parts of speech of each word in the input sentence is determined based on the part of speech analysis tag for the input sentence. In the embodiment of the invention, the part of speech of the input sentence is obtained by utilizing part of speech tagging (POS) processing. Thereafter, the process proceeds to step S204.
In step S204, related words of the target word are determined according to a predetermined rule based on the results of the part-of-speech and syntactic analyses and the results of the context analysis of the target word, and the like.
In an embodiment of the present invention, the syntactic relationship type according to syntactic analysis may be represented, for example, by table 1 below:
| type of relationship | Marking | Examples of the invention |
| Relationship between major and minor | SBV | I send her a bunch of flowers (I ← flower) |
| Moving guest relationship | VOB | I send her bunch of flowers (send → flower) |
| Inter-guest relationships | IOB | I send her bunch of flowers (send → her) |
| Preposition object | FOB | He reads what (book ← read) |
| Concurrent language | DBL | He asks me to eat (please → me) |
| Centering relationships | ATT | Red apple (Red ← apple) |
| Relation of Zhuang Zhong | ADV | Very beautiful (very special ← beautiful) |
| Dynamic complement relationship | CMP | Done the job (do → done) |
| In a parallel relationship | COO | Mountain and sea (mountain → sea) |
| Intermediary relation | POB | In the trade area (in → in) |
| Left additive relationship | LAD | Mountain and sea (He ← sea) |
| Right additive relationship | RAD | Kids (Children → people) |
| Independent structure | IS | The two separate sentences being structurally independent of each other |
| Core relationships | HED | Core of the entire sentence |
| Punctuation | WP | Punctuation |
TABLE 1
After the parts of speech of each word are determined in step S203 and the syntactic relationship type is determined in step S204, the related words of the target word may be determined according to a predetermined rule. For example, for an input sentence "his martial arts are high", the part of speech of the target word "high" is an adjective, "he" is a pronoun, "martial arts" is a noun, "very" is an adverb, and further syntactic analysis shows that there is a medium relationship between "martial arts" and "high", thereby determining that "martial arts" is a related word of the target word "high". After determining the related word, the process proceeds to step S205.
Step S205 and step S206 are the same as steps S104 and S105 described with reference to fig. 1, respectively, and are processes of determining the word sense of the target word in the input sentence based on the related word and the one or more hypernyms, and a repeated description thereof will be omitted here.
FIG. 3 is a flow diagram illustrating a training method of a word sense disambiguation module according to an embodiment of the invention. As shown in fig. 3, the training method of the word sense disambiguation module according to an embodiment of the present invention includes the following steps.
In step S301, training data for training is labeled. Thereafter, the process proceeds to step S302.
In step S302, data processing is performed on the training data, and a predetermined ambiguous word bank is obtained. In an embodiment of the invention, useful data is filtered and extracted by data processing and an ambiguous word bank comprising a predetermined number of ambiguous words is obtained. Thereafter, the process proceeds to step S303.
In step S303, for each training sentence in the training data, a disambiguated training target word in each training sentence is determined based on a predetermined ambiguous word bank. The method of determining the disambiguation training target word for each training sentence in the training data in step S303 may be performed by searching for an ambiguous word bank, as in step S102 described above with reference to fig. 1 and step S202 described with reference to fig. 2. Thereafter, the process proceeds to step S304.
In step S304, a training related word of the training target word is determined based on the syntactic analysis and the context information analysis for each training sentence. The method of determining the training related words of the training target words for disambiguating the training target words in each training sentence in the training data in step S304 is the same as in step S103 described above with reference to fig. 1 and steps S203 and S204 described above with reference to fig. 2, and can be implemented by obtaining the part of speech of the training sentence using part of speech tagging (POS) processing, determining the type of syntactic relationship using syntactic analysis, and determining the related words of the target words according to predetermined rules. Thereafter, the process proceeds to step S305.
In step S305, the shapes of the training target words, the training related words, the training target words, and the hypernyms of the training related words, the parts of speech, and the syntactic relationship with the target words are determined as training features. In the embodiment of the present invention, the training target words, the training related words, the training target words, and the hypernyms of the training related words, and the parts of speech, and the like of these words are extracted as features for training, and conversion (for example, feature hash implantation) is performed on the features to obtain features suitable for machine learning. Thereafter, the process proceeds to step S306.
In step S306, the word sense disambiguation module is trained using the training features. In the embodiment of the invention, a training feature SVM classifier is utilized, and a trained model is saved as a word sense disambiguation module.
FIG. 4 is a block diagram illustrating a word sense disambiguation apparatus according to an embodiment of the present invention. As shown in fig. 4, the word sense disambiguation apparatus 400 according to the embodiment of the present invention includes a receiving unit 401, a target word determining unit 402, a related word determining unit 403, an hypernym determining unit 404, and a word sense disambiguation unit 405.
Specifically, the receiving unit 401 is configured to receive an input sentence. The target word determination unit 402 is configured to determine a disambiguating target word in the input sentence on the basis of a predetermined ambiguous word bank. The related word determination unit 403 is configured to determine related words of the target word based on a syntactic analysis and a context information analysis of the input sentence. The hypernym determination unit 404 is configured to determine one or more hypernyms of the related word. The word sense disambiguation unit 405 is configured to determine a word sense of the target word in the input sentence based on the related word and the one or more hypernyms. The related word determination unit 403 is further configured to determine parts of speech of each word in the input sentence based on the part of speech analysis tag of the input sentence; and determining related words of the target words according to a predetermined rule based on the parts of speech, the result of the syntactic analysis, the result of the context analysis of the target words, and the like. The respective units of the word sense disambiguation apparatus 400 as described above perform the word sense disambiguation method according to the embodiment of the present invention described with reference to fig. 1 and 2.
Furthermore, the word sense disambiguation apparatus 400 according to an embodiment of the present invention may further include a training unit (not shown). The training unit is configured to: marking training data for training; performing data processing on the training data and obtaining the predetermined ambiguous word bank; for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank; determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence; determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; and training the word sense disambiguation unit using the training features.
The word sense disambiguation method and the word sense disambiguation apparatus according to the embodiment of the invention are described above with reference to fig. 1 to 4; hereinafter, a word sense expansion method and a word sense expansion apparatus using the word sense disambiguation method according to an embodiment of the present invention will be described with further reference to fig. 5 to 7.
Fig. 5 is a flowchart illustrating a word sense expansion method according to an embodiment of the present invention. As shown in fig. 5, the word sense expansion method according to an embodiment of the present invention includes the following steps.
In step S501, an input sentence is received. In an embodiment of the present invention, a word sense expansion method according to an embodiment of the present invention is used for performing word sense expansion on a word in a received input sentence. Thereafter, the process proceeds to step S502.
In step S502, disambiguation target words and non-ambiguous words in the input sentence are determined based on a predetermined ambiguous word library. In an embodiment of the invention, the predetermined ambiguous lexicon may be determined in the training phase as described above with reference to fig. 3. Thereafter, the process proceeds to step S503.
In step S503, the word sense of the disambiguation target word in the input sentence is determined using the word sense disambiguation module. In an embodiment of the present invention, the word sense disambiguation module performs the word sense disambiguation method described with reference to fig. 1 and 2, i.e., determines related words of the target word through syntactic analysis and context information analysis of the input sentence, and extends the related words to hypernyms thereof, thereby achieving determination of the disambiguation target word sense by considering the related words and hypernyms thereof. Thereafter, the process proceeds to step S504.
In step S504, synonyms and hypernyms corresponding to word senses of the non-ambiguous word and the disambiguation target word, respectively, are determined based on a predetermined synonym library. In embodiments of the present invention, the predetermined synonym library may be a forest of existing synonyms. Thereafter, the process proceeds to step S05.
In step S505, the input sentence is expanded using the synonym and the hypernym.
Fig. 6 is a block diagram illustrating a word sense expansion apparatus according to an embodiment of the present invention. As shown in fig. 6, the word sense expansion apparatus 600 according to the embodiment of the present invention includes a receiving module 601, a target word determining module 602, a word sense disambiguation module 603, and a word sense expansion module 604.
In particular, the receiving module 601 is configured to receive an input sentence. The target word determination module 602 is configured to determine a disambiguated target word and a non-ambiguous word in the input sentence based on a predetermined ambiguous word bank. The word sense disambiguation module 603 is configured to determine the word sense of the disambiguation target word in the input sentence. The word sense expansion module 604 is configured to determine synonyms and hypernyms corresponding to word senses of the non-ambiguous word and the disambiguation target word, respectively, based on a predetermined synonym library; and expanding the input sentence by using the synonym and the hypernym.
More specifically, the word sense disambiguation module 603 is further configured to include: a related word determination unit 6031 configured to determine a related word of the target word based on the syntactic analysis and the context information analysis of the input sentence; a hypernym determination unit 6032 configured to determine one or more hypernyms of the related word; and a word sense disambiguation unit 6033 configured to determine a word sense of the target word in the input sentence based on the related word and the one or more hypernyms. The related word determination unit 6031 is further configured to: determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; and determining related words of the target words according to a predetermined rule based on the parts of speech, the result of the syntactic analysis, the result of the context analysis of the target words, and the like.
Furthermore, the word sense expanding apparatus 600 according to an embodiment of the present invention may further include a training module (not shown). The training module is configured to: marking training data for training; performing data processing on the training data and obtaining the predetermined ambiguous word bank; for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank; determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence; determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; and training the word sense disambiguation unit using the training features.
FIG. 7 is a schematic diagram illustrating a word sense expansion process according to an embodiment of the invention. In particular, fig. 7 illustrates an example in which the word sense expansion apparatus 600 according to the embodiment of the present invention described with reference to fig. 6 performs the word sense expansion method according to the embodiment of the present invention described with reference to fig. 5.
As shown in fig. 7, the receiving module 601 receives an input sentence "how many game items are respectively in olympic".
The input sentence enters the target word determination module 602, and based on a predetermined ambiguous word bank, disambiguated target words and non-ambiguous words in the input sentence are determined. In this example, the target word determination module 602 determines that "respectively" in the input sentence "how many game items respectively are in olympic" are disambiguated target words, and the other words are non-ambiguous words.
The target word determination module 602 provides the determined disambiguation target words "individually" to the word sense disambiguation module 603. The word sense disambiguation module 603 performs the word sense disambiguation method according to an embodiment of the present invention on the disambiguation target word "respectively" and determines the word sense of the disambiguation target word "respectively".
The word sense "respectively" determined in the word sense disambiguation module 603 and the word determined to be a non-ambiguous word in the target word determination module 602 enter the word sense expansion module 604. The word sense expansion module 604 is used for expanding the input sentence "how many match items are in the input sentence" olympic ", into an expanded sentence" [ olympic | olympic chance<Competitive competition>][Respectively divided into separate heads Are respectively standing together][ possessions | possessions possession all]How much of a match][ item | Category type Category]？”。
FIG. 8 is a hardware block diagram illustrating a word sense disambiguation apparatus according to an embodiment of the present invention. As shown in fig. 8, a word sense disambiguation apparatus 800 according to an embodiment of the invention includes a processor 801 and a memory 802. The memory 802 is configured to store computer program instructions which, when executed by the processor 801, perform the word sense disambiguation method as described above with reference to the above figures.
Fig. 9 is a hardware block diagram illustrating a word sense expanding apparatus according to an embodiment of the present invention. As shown in fig. 9, the word sense expanding device 900 according to the embodiment of the present invention includes a processor 901 and a memory 902. The memory 902 is configured to store computer program instructions which, when executed by the processor 901, perform the word sense disambiguation method as described above with reference to the above figures.
Fig. 10 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present invention. As shown in fig. 10, a computer-readable storage medium 1000 according to an embodiment of the present invention has stored thereon computer program instructions 1001. The computer program instructions 1001, when executed by a processor, perform the word sense disambiguation method and the word sense expansion method according to embodiments of the invention described with reference to the above figures.
The word sense disambiguation method and apparatus, and the word sense expansion method and apparatus using the word sense disambiguation method according to the embodiments of the present invention are described above with reference to the accompanying drawings. Related words of the disambiguation target words are determined through syntactic analysis, and the related words are expanded to hypernyms of the related words, so that the word meaning of the disambiguation target words is determined by considering the related words and the hypernyms of the related words, and the dependence on the size of the training corpus is greatly reduced.
The basic principles of the present invention have been described above with reference to specific embodiments, but it should be noted that the advantages, effects, etc. mentioned in the present invention are only examples and are not limiting, and the advantages, effects, etc. must not be considered to be possessed by various embodiments of the present invention. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the invention is not limited to the specific details described above.
The block diagrams of devices, apparatuses, systems involved in the present invention are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
The flow charts of steps in the present invention and the above description of the method are only given as illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order given, some steps may be performed in parallel, independently of each other or in other suitable orders. Additionally, words such as "thereafter," "then," "next," etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods.
Also, as used herein, "or" as used in a list of items beginning with "at least one" indicates a separate list, such that, for example, a list of "A, B or at least one of C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the described example is preferred or better than other examples.
It should also be noted that the components or steps may be broken down and/or re-combined in the apparatus and method of the present invention. These decompositions and/or recombinations are to be regarded as equivalents of the present invention.
It will be understood by those of ordinary skill in the art that all or any portion of the methods and apparatus of the present invention may be implemented in hardware, firmware, software, or any combination thereof, in any computing device (including processors, storage media, etc.) or network of computing devices. The hardware may be implemented with a general purpose processor, a Digital Signal Processor (DSP), an ASIC, a field programmable gate array signal (FPGA) or other Programmable Logic Device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The software may reside in any form of computer readable tangible storage medium. By way of example, and not limitation, such computer-readable tangible storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk, as used herein, includes Compact Disk (CD), laser disk, optical disk, Digital Versatile Disk (DVD), floppy disk, and Blu-ray disk.
The intelligent control techniques disclosed herein may also be implemented by running a program or a set of programs on any computing device. The computing device may be a general purpose device as is well known. The disclosed intelligent techniques may also be implemented by simply providing a program product containing program code for implementing the methods or apparatus, or by any storage medium having such a program product stored thereon.
Various changes, substitutions and alterations to the techniques described herein may be made without departing from the techniques of the teachings as defined by the appended claims. Moreover, the scope of the present claims is not intended to be limited to the particular aspects of the process, machine, manufacture, composition of matter, means, methods and acts described above. Processes, machines, manufacture, compositions of matter, means, methods, or acts, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or acts.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the invention to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (16)
- A word sense disambiguation method comprising:receiving an input sentence;determining a disambiguation target word in the input sentence based on a predetermined ambiguity word bank;determining related words of the target words based on the syntactic analysis and the context information analysis of the input sentence;determining one or more hypernyms of the related word; andand determining the word meaning of the target word in the input sentence based on the word shapes, the word parts and the syntactic relations with the target word of the related word and the one or more hypernyms.
- The word sense disambiguation method of claim 1, wherein said determining related words of the target word based on a syntactic analysis and a contextual information analysis of the input sentence comprises:determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; andand determining related words of the target words according to a preset rule based on the part of speech, the result of the syntactic analysis, the result of the context analysis of the target words and the like.
- The word sense disambiguation method of claim 1 or 2, further comprising pre-training a word sense disambiguation module that performs the word sense disambiguation method, wherein training the word sense disambiguation module comprises:marking training data for training;performing data processing on the training data and obtaining the predetermined ambiguous word bank;for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank;determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence;determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; andtraining the word sense disambiguation module using the training features.
- A word sense expansion method, comprising:receiving an input sentence;determining a disambiguation target word and a non-ambiguous word in the input sentence based on a predetermined ambiguous word bank;determining a word sense of the disambiguation target word in the input sentence using a word sense disambiguation module;determining synonyms and hypernyms corresponding to word senses of the non-ambiguous word and the disambiguation target word respectively based on a predetermined synonym library; andexpanding the input sentence by using the synonym and the hypernym,wherein the determining, by the word sense disambiguation module, the word sense of the disambiguation target word in the input sentence comprises:determining related words of the target words based on the syntactic analysis and the context information analysis of the input sentence;determining one or more hypernyms of the related word; anddetermining a word sense of the target word in the input sentence based on the related word and the one or more hypernyms.
- The word sense expansion method of claim 4, wherein the determining related words of the target word based on the syntactic analysis and the context information analysis of the input sentence comprises:determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; andand determining related words of the target words according to a preset rule based on the part of speech, the result of the syntactic analysis, the result of the context analysis of the target words and the like.
- The word sense expansion method of claim 4 or 5, further comprising pre-training a word sense disambiguation module that performs the word sense disambiguation method, wherein training the word sense disambiguation module comprises:marking training data for training;performing data processing on the training data and obtaining the predetermined ambiguous word bank;for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank;determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence;determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; andtraining the word sense disambiguation module using the training features.
- A word sense disambiguating apparatus comprising:a receiving unit configured to receive an input sentence;a target word determination unit configured to determine a disambiguation target word in the input sentence based on a predetermined ambiguous word bank;a related word determination unit configured to determine a related word of the target word based on a syntactic analysis and a context information analysis of the input sentence;a hypernym determination unit configured to determine one or more hypernyms of the related words; anda word sense disambiguation unit configured to determine a word sense of the target word in the input sentence based on the related word and the one or more hypernyms.
- The word sense disambiguation apparatus of claim 7, wherein the related word determining unit is further configured to:determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; and determining related words of the target words according to a predetermined rule based on the parts of speech, the result of the syntactic analysis, the result of the context analysis of the target words, and the like.
- The word sense disambiguation device of claim 7 or 8, further comprising a training unit configured to:marking training data for training;performing data processing on the training data and obtaining the predetermined ambiguous word bank;for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank;determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence;determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; andtraining the word sense disambiguation unit using the training features.
- A word sense expansion apparatus comprising:a receiving module configured to receive an input sentence;a target word determination module configured to determine a disambiguation target word and a non-ambiguous word in the input sentence based on a predetermined ambiguous word bank;a word sense disambiguation module configured to determine a word sense of the disambiguation target word in the input sentence;a word sense expansion module configured to determine synonyms and hypernyms corresponding to word senses of the non-ambiguous word and the disambiguation target word, respectively, based on a predetermined synonym library; and expanding the input sentence using the synonyms and hypernyms,wherein the word sense disambiguation module is further configured to comprise:a related word determination unit configured to determine a related word of the target word based on a syntactic analysis and a context information analysis of the input sentence;a hypernym determination unit configured to determine one or more hypernyms of the related words; anda word sense disambiguation unit configured to determine a word sense of the target word in the input sentence based on the related word and the one or more hypernyms.
- The word sense expansion apparatus of claim 10, wherein the related word determining unit is further configured to:determining the part of speech of each word in the input sentence based on the part of speech analysis label of the input sentence; and determining related words of the target words according to a predetermined rule based on the parts of speech, the result of the syntactic analysis, the result of the context analysis of the target words, and the like.
- The word sense expansion apparatus of claim 10 or 11, further comprising a training module configured to:marking training data for training;performing data processing on the training data and obtaining the predetermined ambiguous word bank;for each training sentence in the training data, determining a disambiguation training target word in each training sentence based on the predetermined ambiguity word bank;determining training related words of the training target words based on the syntactic analysis and the context information analysis of each training sentence;determining the word shapes, the part of speech and the syntactic relation with the target word of the training target word, the training related word, the training target word and the hypernym of the training related word as training characteristics; andtraining the word sense disambiguation unit using the training features.
- A word sense disambiguating apparatus comprising:a processor; anda memory configured to store computer program instructions;wherein the computer program instructions, when executed by the processor, cause the processor to perform the word sense disambiguation method of claim 1 or 2.
- A word sense expansion apparatus comprising:a processor; anda memory configured to store computer program instructions;wherein the computer program instructions, when executed by the processor, cause the processor to perform the word sense expansion method of claim 4 or 5.
- A computer readable storage medium storing computer program instructions, wherein the computer program instructions, when executed by a processor, cause the processor to perform the word sense disambiguation method of claim 1 or 2.
- A computer readable storage medium storing computer program instructions, wherein the computer program instructions, when executed by a processor, cause the processor to perform the word sense expansion method of claim 4 or 5.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2017110483647 | 2017-10-31 | ||
| CN201711048364.7A CN109726385A (en) | 2017-10-31 | 2017-10-31 | Word sense disambiguation method and equipment, meaning of a word extended method and device |
| PCT/CN2018/104334 WO2019085640A1 (en) | 2017-10-31 | 2018-09-06 | Word meaning disambiguation method and device, word meaning expansion method, apparatus and device, and computer-readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111295661A true CN111295661A (en) | 2020-06-16 |
Family
ID=66293105
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711048364.7A Pending CN109726385A (en) | 2017-10-31 | 2017-10-31 | Word sense disambiguation method and equipment, meaning of a word extended method and device |
| CN201880071178.1A Pending CN111295661A (en) | 2017-10-31 | 2018-09-06 | Word sense disambiguation method and apparatus, word sense expansion method, device and apparatus, computer readable storage medium |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711048364.7A Pending CN109726385A (en) | 2017-10-31 | 2017-10-31 | Word sense disambiguation method and equipment, meaning of a word extended method and device |
Country Status (3)
| Country | Link |
|---|---|
| JP (1) | JP2021501420A (en) |
| CN (2) | CN109726385A (en) |
| WO (1) | WO2019085640A1 (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134955A (en) * | 2019-05-14 | 2019-08-16 | 中电协通科技(张家口)有限公司 | A kind of semantic processes method |
| CN110309318B (en) * | 2019-05-29 | 2022-11-29 | 西安电子科技大学 | Intention representation system and method of information communication network, and information data processing terminal |
| CN110321434A (en) * | 2019-06-27 | 2019-10-11 | 厦门美域中央信息科技有限公司 | A kind of file classification method based on word sense disambiguation convolutional neural networks |
| CN111199149B (en) * | 2019-12-17 | 2023-10-20 | 航天信息股份有限公司 | Sentence intelligent clarification method and system for dialogue system |
| CN110991196B (en) | 2019-12-18 | 2021-10-26 | 北京百度网讯科技有限公司 | Translation method and device for polysemous words, electronic equipment and medium |
| CN111310481B (en) * | 2020-01-19 | 2021-05-18 | 百度在线网络技术（北京）有限公司 | Speech translation method, device, computer equipment and storage medium |
| CN111310475B (en) * | 2020-02-04 | 2023-03-10 | 支付宝(杭州)信息技术有限公司 | Training method and device of word sense disambiguation model |
| CN111414523A (en) * | 2020-03-11 | 2020-07-14 | 中国建设银行股份有限公司 | Data acquisition method and device |
| CN111737972A (en) * | 2020-05-20 | 2020-10-02 | 华为技术有限公司 | Method and device for realizing natural language understanding in human-computer interaction system |
| CN112052678A (en) * | 2020-09-24 | 2020-12-08 | 大众问问(北京)信息科技有限公司 | Model training and corpus processing method and device and computer equipment |
| CN112580335B (en) * | 2020-12-28 | 2023-03-24 | 建信金融科技有限责任公司 | Method and device for disambiguating polyphone |
| CN113204962B (en) * | 2021-05-31 | 2024-08-16 | 平安科技（深圳）有限公司 | Word sense disambiguation method, device, equipment and medium based on graph expansion structure |
| CN113704416B (en) * | 2021-10-26 | 2022-03-04 | 深圳市北科瑞声科技股份有限公司 | Word sense disambiguation method and device, electronic equipment and computer-readable storage medium |
| CN113901836A (en) * | 2021-11-16 | 2022-01-07 | 东莞理工学院 | Word sense disambiguation method and device based on context semantics and related equipment |
| CN115204182B (en) * | 2022-09-09 | 2022-11-25 | 山东天成书业有限公司 | Method and system for identifying e-book data to be corrected |
| CN118468889A (en) * | 2024-07-12 | 2024-08-09 | 西安银信博锐信息科技有限公司 | NLP-based customer intention analysis method and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8504355B2 (en) * | 2009-11-20 | 2013-08-06 | Clausal Computing Oy | Joint disambiguation of syntactic and semantic ambiguity |
| CN102306144B (en) * | 2011-07-18 | 2013-05-08 | 南京邮电大学 | Terms disambiguation method based on semantic dictionary |
| CN105718442A (en) * | 2016-01-19 | 2016-06-29 | 齐鲁工业大学 | Word sense disambiguation method based on syntactic analysis |
| CN106202036B (en) * | 2016-06-29 | 2019-05-21 | 齐鲁工业大学 | A kind of verb Word sense disambiguation method and device based on interdependent constraint and knowledge |
| CN106598947A (en) * | 2016-12-15 | 2017-04-26 | 山西大学 | Bayesian word sense disambiguation method based on synonym expansion |
-
2017
- 2017-10-31 CN CN201711048364.7A patent/CN109726385A/en active Pending
-
2018
- 2018-09-06 JP JP2020524159A patent/JP2021501420A/en active Pending
- 2018-09-06 WO PCT/CN2018/104334 patent/WO2019085640A1/en active Application Filing
- 2018-09-06 CN CN201880071178.1A patent/CN111295661A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019085640A1 (en) | 2019-05-09 |
| CN109726385A (en) | 2019-05-07 |
| JP2021501420A (en) | 2021-01-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111295661A (en) | Word sense disambiguation method and apparatus, word sense expansion method, device and apparatus, computer readable storage medium | |
| WO2018049960A1 (en) | Method and apparatus for matching resource for text information | |
| Antony et al. | SVM based part of speech tagger for Malayalam | |
| CN102479191B (en) | Method and device for providing multi-granularity word segmentation result | |
| US10943064B2 (en) | Tabular data compilation | |
| US10496756B2 (en) | Sentence creation system | |
| CN109918657A (en) | A method of extracting target keyword from text | |
| Ojha et al. | Training & evaluation of POS taggers in Indo-Aryan languages: a case of Hindi, Odia and Bhojpuri | |
| CN109408802A (en) | A kind of method, system and storage medium promoting sentence vector semanteme | |
| JP2019082931A (en) | Retrieval device, similarity calculation method, and program | |
| Sefid et al. | Automatic slide generation for scientific papers | |
| Torres-Moreno | Artex is another text summarizer | |
| Zhang et al. | Ecnu: A combination method and multiple features for aspect extraction and sentiment polarity classification | |
| Yuwana et al. | On part of speech tagger for Indonesian language | |
| Rani et al. | MHE: Code-mixed corpora for similar language identification | |
| KR20170131893A (en) | Apparatus and method of generation and classification for text classifier based on open directory project | |
| Spiccia et al. | A word prediction methodology for automatic sentence completion | |
| Visser et al. | Sentiment and intent classification of in-text citations using bert | |
| US20110106849A1 (en) | New case generation device, new case generation method, and new case generation program | |
| Mandal et al. | Adaptive Voting in Multiple Classifier Systems for Word Level Language Identification. | |
| Pailai et al. | A comparative study on different techniques for thai part-of-speech tagging | |
| WO2021063060A1 (en) | Text information extraction method and apparatus, storage medium and device | |
| Noraset et al. | WebSAIL wikifier at ERD 2014 | |
| Shiang et al. | Spoken Term Detection and Spoken Content Retrieval: Evaluations on NTCIR 11 SpokenQuery&Doc Task. | |
| Chaimongkol et al. | Utilizing LDA clustering for technical term extraction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200616 |