KR101195341B1 - Method and apparatus for determining category of an unknown word - Google Patents

Abstract

Embodiments of the present invention disclose a method and apparatus for determining a category of unregistered words. The method includes selecting synonyms of an unregistered word from a dictionary based on a word formation rule, generating a context of an unregistered word from a corpus, and an unregistered word based on the context and synonyms of an unregistered word And determining a category to which the belongs. The method and apparatus according to the invention can more effectively and more accurately determine the category of unregistered words.

Description

METHOD AND APPARATUS FOR DETERMINING CATEGORY OF AN UNKNOWN WORD}

TECHNICAL FIELD The present invention generally relates to the field of information processing, and more particularly, to a method and apparatus for determining the category of an unknown word.

With the spread of the Internet and the increasing trend of social information, there is more and more text information, and accordingly, the social demand for text information processing is increasing. More and more people want to communicate with computers using natural language, and they want automatic processing of large amounts of textual information. In order to process textual information in a better way, it is necessary to collect a large amount of linguistic data resources such as dictionaries, for example. However, as one of the important means for text processing, dictionaries are always edited manually, which is time consuming and inefficient. In addition, in word segmentation techniques, segmentation errors for unregistered words have a significant effect on the recall rate of the overall word segmentation, and also affect the accuracy of subsequent grammar and word semantic interpretations, and thus information. It causes some trouble with the process. For example, in other information processing techniques, such as information extraction, if the attributes of an unregistered word are not clear enough, the results of the information extraction will not only have ambiguity but also errors due to unregistered words and defects of that information. will be. Therefore, it is urgent to determine the category of unregistered words.

Chinese Patent Application Publication No. CN1717679 discloses a word class tagging method. The method mainly uses a pre-recorded keyword-word class repository for tagging words segments collectively. If the segment of words contains a special keyword, the segment of these words is tagged as a word class corresponding to the keyword.

US Patent Application Publication No. US20060100856A1 discloses a method of estimating word meaning. The basic idea of the method is to extract usage examples of each new word through web search, and to extract semantic class candidates based on the examples according to the existing usage dictionary, and when there are two or more candidates, In the corpus, the semantic class with the new word and the highest co-occurrence frequency is selected.

Chinese Patent Application Publication No. CN1369877 discloses a method of estimating a category of a new word. This method first determines the separation probability for each letter in the new word and then combines the probabilities of each letter based on the word category to find the overall separation probability of each category. Based on the comparison of the overall probabilities against the threshold, each word category with a probability above the threshold is added as a possible category for a multi-character word.

In 'Hybrid Models for Semantic Classification of Chinese Unknown Words' on pages 188-195 of NAACL HLT 2007, Xiaofei Lu was established based on man-made rules and statistical methods and contexts. A method of estimating word class of mixed types is disclosed, and this rule and statistical method provide semantic class candidates for context-based methods.

Sense-tagging Chinese Corpus on pages 7-14 of the In Proceedings of the 2nd Chinese Language Processing Workshop, page 7-14 of the 2nd Chinese Processing Workshop Newsletter in 2000 Chen, H.-H. and C.-C. Lin discloses a semantic class tagging method through inter-translation in a Chinese-English dictionary. The basic process of this method consists of four steps: 1) selecting a new word and retrieving all possible English translations for that word based on a given Chinese-English dictionary; 2) for all translations in WordNet. Searching for the corresponding semantic item, 3) searching the mapping table and matching the semantic item obtained in step 2 with the Cilin semantic tag, and 4) one of the semantic tags obtained in step 3 through the semantic neutralization method. Selecting as the final result.

However, no technology currently solves the problem of efficiently determining the category of unregistered words in order to implement automatic tagging. Existing techniques generally use pre-edited dictionaries to perform word class analysis on new words, so the rationality of tagging results of this method depends on the construction of the corresponding dictionary or knowledge base, and their The performance is quite low.

Therefore, a technical solution for determining the category of unregistered words with efficient and good performance is required.

In connection with the above problems present in the prior art, it is an object of the present invention to provide a method and apparatus for determining the category of unregistered words.

According to one aspect of the present invention, a method of determining a category of unregistered words is provided. The method includes selecting synonyms of an unregistered word from a dictionary based on a word formation rule, generating a context of an unregistered word from a corpus, and an unregistered word based on the context and synonyms of an unregistered word And determining a category to which the belongs.

According to a second aspect of the invention, there is provided an apparatus for determining a category of unregistered words. The apparatus includes a synonym selector configured to select synonyms of an unregistered word from a dictionary based on word forming rules, a context generator configured to generate a context of an unregistered word from a corpus, and a context of an unregistered word; And a category determiner configured to determine a category to which the unregistered word belongs based on the synonyms.

Other features and advantages of the present invention will become apparent from the following description of the preferred embodiments of the present invention with reference to the accompanying drawings.

The method and apparatus according to the invention can more effectively and more accurately determine the category of unregistered words.

Other objects and effects of the present invention will become much clearer and easier to understand through the following description with reference to the accompanying drawings and by a clearer understanding of the present invention.
1 is a block diagram of an apparatus for determining a category of an unregistered word according to an embodiment of the present invention.
2 is a flowchart of a method of determining a category of an unregistered word, according to an embodiment of the present invention.
3 is a flowchart of a method of determining a category of an unregistered word according to another embodiment of the present invention.
4 is a flowchart of a method of determining a category of an unregistered word, according to another embodiment of the present invention.
5 is a flowchart of a method of determining an unregistered word according to another embodiment of the present invention.
In all of the accompanying drawings, like reference numerals refer to the same, similar or corresponding features or functions.

In the following, the invention will be explained and illustrated in more detail with reference to the accompanying drawings. It should be noted that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and not intended to limit the protection scope of the present invention.

For clarity, the terms used in the present invention are first described.

1. Dictionary

(ci-lin, 단어 숲), HowNet, WordNet 등과 같이 일반적으로 50,000 이상의 수록어를 갖는, 처리되어야 할 언어의 핵심 어휘를 기록하고 있는 것을 말한다. 사전은 하나 이상의 단어들을 포함할 수 있다. 각각의 단어에 대하여, 품사, 카테고리, 단어 의미 및 예문과 같은 정보가 태깅될 수 있다. 표 1은 사전의 데이터 구조의 예를 제공하며, "北京"(Beijing, 북경), "保健品"(bao-jian-pin, 의료용품), 및 "愉快"(yu-kuai, 행복)의 세 단어가 제시되어 있고, 각각의 단어는 각자의 품사와 카테고리를 갖는다.With a dictionary, for example,

(ci-lin, word forest), howNet, WordNet, etc., which generally contains more than 50,000 entries, records the core vocabulary of the language to be processed. The dictionary may include one or more words. For each word, information such as part of speech, category, word meaning and example sentence can be tagged. Table 1 provides an example of the dictionary's data structure, with three categories: "北京" (Beijing, Beijing), "保健 品" (bao-jian-pin, medical supplies), and "愉快" (yu-kuai, happiness). The words are presented, and each word has its own parts of speech and category.

number word Parts of speech category One Beijing noun city 2 保健 品 noun matter 3 愉快 adjective feeling ... ... ... ...

2. Corpus

Corpus is a collection of free text. The free text can be sentences, segments, articles, or any combination thereof.

3. Letters, direct constituents and words

Character is the minimum text unit. For example, in Chinese, "天" (tian, heaven), "我" (wo, me), and "好" (hao, good) are letters respectively.

(ke-xue-ji-shu-bu, 과학기술부)의 직접 요소들은

(ke-xue, 과학),

(ji-shu, 기술) 및 "部"(bu, 부서)이며,

(bing-jing, 얼음 결정체)의 직접 요소들은

(얼음)과 "晶"(결정체)이다.Direct element: A small unit that forms a large unit is called a constituent of a large unit, so a small unit that directly constitutes a large unit is called a direct element of a large unit. The direct element of a word may be a phoneme or a word smaller than that word. For example,

(ke-xue-ji-shu-bu, Ministry of Science and Technology)

(ke-xue, science),

(ji-shu, technology) and "部" (bu, department),

Direct elements of (bing-jing, ice crystals)

(Ice) and "晶" (crystals).

(wo-men, 우리)는 두 문자를 포함하는 단어이고,

(ji-suan-ji, 컴퓨터)는 세 문자를 포함하는 단어이다.A word is a string of specific meaning, formed of one or more letters. E.g,

(wo-men, we) is a word that contains two letters,

(ji-suan-ji, computer) is a word that contains three letters.

4. Unregistered Word

Unregistered words are words that are not currently included in the dictionary.

5. Category

Categories may include semantic classes and supersenses with broader coverage than semantic classes.

(shen-ti-bu-wei, 신체부위)"와 "人物(ren-wu, 인물)"의 2개의 의미 클래스를 가질 수 있다.The semantic class may be, for example, "城市" (cheng-shi, city), "心情" (xin-qing). The semantic class may have a number of words, for example, the words "北京 (Beijing)" and "上海 (Shanghai)" may belong to the semantic class "城市" (cheng-shi, city). A word can have multiple semantic classes, for example, the word "bi" (bi-bang)

(shen-ti-bu-wei) and "ren-wu" (person).

Supersense is a broader category than the semantic class, for example "location" and "substance", and "location" has a wider range than the semantic class "city".

The present invention relates to a method for determining a category of unregistered words. The method includes selecting synonyms of an unregistered word from a dictionary based on a word formation rule, generating a context of an unregistered word from a corpus, and an unregistered word based on the context and synonyms of an unregistered word And determining a category to which the belongs.

According to one embodiment of the invention, words sharing one or more word forming elements with an unregistered word may be selected from the dictionary as synonyms of the unregistered word, thereby selecting synonyms of the unregistered word based on the word formation rule. Implement a process that According to a further embodiment of the present invention, the process of selecting in the dictionary the synonyms of the unregistered word based on the word formation rule may be implemented through the following steps: determining the part of speech of the unregistered word, the unregistered word and one. Selecting words in the dictionary that share the above word forming elements, and selecting words having the same parts of speech as the unregistered word among all selected words as synonyms of the unregistered word.

According to one embodiment of the invention, the process of creating a context of an unregistered word from a corpus may be implemented through the following steps: retrieving an unregistered word from a corpus, applying a window to the unregistered word Importing adjacent characters, performing word segmentation on the characters adjacent to the imported unregistered word, and using the respective words obtained after the word segmentation and the weight of the words as the context of the unregistered word, Determining weights of respective words. According to another embodiment of the invention, the process of creating a context of an unregistered word from a corpus can be implemented through the following steps: retrieving an unregistered word from a corpus, and using dependencies as the context of an unregistered word. Analyzing the dependencies of unregistered words in dependent tree mode.

According to one embodiment of the invention, the process of determining the category to which the unregistered word belongs based on the synonyms and context of the unregistered word may include the following steps: generating statistics about the categories to which the synonym belongs. Generating from the corpus the context of all the words contained in each category as the context of their category, calculating the similarity between the context of the unregistered words and the contexts of the respective categories, and the category with the maximum similarity Determining as a category to which the unregistered word belongs. According to another embodiment of the present invention, the process of determining a category to which an unregistered word belongs based on synonyms and context of an unregistered word may include the following steps: generating a context of the synonyms from a corpus, unregistered Calculating a similarity between the context of the word and the contexts of the synonyms, extracting a set from the synonyms based on the similarity obtained through the calculation, summing the similarities corresponding to the synonyms in the extracted set and belonging to the same category And determining a category to which the unregistered word belongs based on the sum of the similarities. According to another embodiment of the present invention, the process of determining a category to which an unregistered word belongs based on synonyms and context of an unregistered word may include the following steps: generating a context of the synonyms from a corpus, unregistered Calculating similarities between the context of the word and the contexts of the synonyms, generating statistics about the categories to which the synonyms belong, receiving the predetermined weighting factors associated with the synonyms, the received predetermined factors Weighting similarities corresponding to the associated synonyms based on; extracting a set from the synonyms based on the weighted similarities; summing the weighted similarities corresponding to the synonyms in the extracted set and belonging to the same category And the unregistered word based on the sum of the similarities Determining the category.

In the following, respective embodiments of the present invention will be described in detail.

1 shows a block diagram of an apparatus 100 for determining a category of unregistered words, according to one embodiment of the invention.

The apparatus 100 for determining a category of an unregistered word according to the present invention may include a synonym selector 110, a context generator 120, and a category determiner 130. The synonym selector 110 may select synonyms of an unregistered word from a dictionary based on a word formation rule. The context generator 120 may generate a context of an unregistered word from the corpus. The category determiner 130 may determine a category to which the unregistered word belongs based on the context and the synonyms of the unregistered word.

According to one embodiment of the invention, the synonym selector 110 may include means for selecting in the dictionary words that share one or more word forming elements with the unregistered word as synonyms of the unregistered word. According to one embodiment of the invention, the synonym selector 110 includes means for determining the part of speech of an unregistered word, means for selecting words from a dictionary that share one or more word forming elements with an unregistered word, and all selected words. Means for selecting words having the same part-of-speech among the unregistered words as synonyms of the unregistered word.

According to one embodiment of the invention, the context generator 120 provides means for retrieving unregistered words in the corpus, means for fetching words proximate to the unregistered words by means of a window, imported Means for performing word segmentation on words proximate to an unregistered word, and determining the weights of the respective words obtained after word segmentation, in order to use the respective words obtained after the word segmentation and the weights of those words as context of the unregistered word. And means for doing so.

According to one embodiment of the invention, the context generator 120 uses the means for retrieving the unregistered words in the corpus, and the dependency of the unregistered words in the dependent tree mode to use the dependency as the context of the unregistered words. It may include means for analyzing the.

According to one embodiment of the invention, the context generator 120 may further comprise means for generating the context of the synonyms from the corpus.

According to one embodiment of the invention, the category determiner 130 is a means for generating statistics about the categories to which the synonyms belong, to generate the context of all words contained in each category from the corpus as the context of the respective category. Means for calculating a similarity between the context of an unregistered word and the contexts of respective categories, and means for determining a category corresponding to the maximum similarity as a category to which the unregistered word belongs.

According to an embodiment of the present invention, the category determiner 130 may include: means for calculating similarities of contexts of unregistered words and contexts of synonyms, means for extracting a set from synonyms based on the similarities, within an extracted set Means for summing similarities corresponding to synonyms and belonging to the same category, and means for determining a category to which an unregistered word belongs based on the summed similarities. In one embodiment, the means for determining the category to which the unregistered word belongs based on the summed similarities included in the category determiner 130 may perform a K-nearest neighbor algorithm. .

According to one embodiment of the invention, the category determiner 130 is associated with means for calculating similarities of contexts of unregistered words and contexts of the synonyms, means for generating statistics about the categories to which the synonyms belong, synonyms Means for receiving predetermined weighting factors, means for weighting similarities corresponding to associated synonyms based on the received predetermined weighting factors, means for extracting a set from the synonyms based on the similarities, extraction Means for summing weighted similarities that correspond to synonyms in the aggregated set and belonging to the same category, and means for determining a category to which an unregistered word belongs based on the summed similarities. In one embodiment, the designation of the predetermined weighting factors satisfies the following policies: if the unregistered word and the word in the category share the last letter and the second to last letter, then the predetermined weighting factor associated with the category is λ _1. Set to; Otherwise, if the unregistered word and the word in the category share the first letter and the last letter, set the predetermined weighting factor associated with the category to λ ₂ ; Otherwise, if the unregistered word and the word in the category share only the first letter or the last letter, set the predetermined weighting factor associated with the category to λ ₃ ; Otherwise, the predetermined weighting factor associated with the category is set to λ ₄ (where λ ₁ ≧ λ ₂ ≧ λ ₃ ≧ λ ₄ ). In one embodiment, the means for extracting the set from the synonyms based on the similarities, included in the category determiner 130, means for sorting the similarities in descending order, and a predetermined number of similarities at the top And means for extracting the synonyms corresponding to the set.

2 is a flowchart of a method of determining a category of an unregistered word, according to an embodiment of the present invention.

In step 201, synonyms of an unregistered word are selected from the dictionary based on a word formation rule.

According to one embodiment of the invention, the word forming rule may include word forming elements, element attributes and element relationships. Word forming elements may include letters and / or direct elements forming a word, element attributes may include tagging, length, part-of-speech of a word, and element relationships may be relationships between respective elements of a word. , For example, may include parallelism, modifications, restrictions, and the like.

(ji-chu, 기초), "基本"(ji-ben, 기본),

(dian-ji-zhe, 창립자), "地基"(di-ji, 토대)이고, "民"을 포함하는 단어는 "人民"(ren-min, 인민)과 "民主"(min-zhu, 민주)라고 가정하면, 이 단어들 모두가 미등록 단어 "基民"의 유의어로 여겨진다. 현재, 유의어들의 집합은 {

, "基本",

, "地基", "人民", "民主"}이다. 도 3에 도시된 예는 이 실시예를 설명한다.In one example, words that share one or more letters and / or direct elements with an unregistered word may be selected from the dictionary as synonyms of the unregistered word. For example, suppose the unregistered word is "基 民" (ji-min, fund investor), which contains two letters: "基" (ji, investor) and "民" (min, person). In the dictionary, a word containing the word "基"

(ji-chu, basic), "基本" (ji-ben, basic),

(dian-ji-zhe, founder), "地基" (di-ji, foundation), and the words containing "民" are "人民" (ren-min) and "民主" (min-zhu, democratic) ), All of these words are considered synonyms of the unregistered word "基 民". Currently, the set of synonyms is {

, "基本",

, "地基", "人民", "民主"}. The example shown in FIG. 3 describes this embodiment.

In addition, in another example, a part-of-speech of an unregistered word, such as a noun, adjective, verb, or the like, is first determined, then words that share one or more letters and / or direct elements are selected from a dictionary, and words having the same part-of-speech are found. Synonyms of unregistered words are selected. The embodiments shown in Figs. 4 and 5 describe this embodiment.

In step 202, a context of unregistered words is generated from the corpus.

According to one embodiment of the invention, the context of a word may be generated by applying a window in dependent tree mode or in other ways well known to those skilled in the art.

(wo-men, 우리)이고, 코퍼스가 복수의 문장을 포함하는데 한 문장이

(yi-ding-hao-hao-ba-wo-wo-men-mei-ge-ren-de-ren-sheng-dao-lu, 우리 각자의 인생 경로를 단단히 붙잡아야 한다.)이며, 윈도우의 크기를 6으로 설정하는 것으로 가정해보자.In the following, an example is used to illustrate how to apply a window to obtain the context of a given word from a corpus. Given word

(wo-men, we), and the corpus contains multiple sentences, one sentence

(yi-ding-hao-hao-ba-wo-wo-men-mei-ge-ren-de-ren-sheng-dao-lu, we must firmly grasp each of our life paths.) Let's assume that we set 6 to 6.

가 문장

에 포함되어 있는 것이 검색된다.First, this word is retrieved from the corpus. In this example, the word

Sentence

What is included in is searched.

에 근접한 문자들을 가져온다. 크기 6을 갖는 윈도우는 코퍼스 내의 이 단어를 포함하는 문장이나 단락 안에 이 단어를 포함하도록 정의될 수 있다. 단어를 포함하는 방식은, 예를 들면, 이 단어(즉,

)를 중심에 두고 바로 앞의 3문자

와 바로 뒤의 3 문자

를 가져오거나, 예를 들어, 이 단어에서 시작하여 바로 뒤에 오는 6 문자

를 가져오거나, 예를 들어, 이 단어를 끝에 두고 있는 바로 앞의 6 문자

를 가져오거나, 예를 들어, 바로 앞의 1 또는 2 문자와 바로 뒤의 5 또는 4 문자를 가져오는 것 등을 의미한다.Next, apply a window to the word

Gets characters close to. A window of size 6 may be defined to include this word in a sentence or paragraph that contains this word in the corpus. The way to include a word is, for example, this word (i.e.

3 characters in front of the center

3 characters immediately after and

, Or, for example, 6 characters starting at this word and immediately following

, Or, for example, the six characters immediately preceding the word

, Or, for example, to bring the 1 or 2 characters immediately before and the 5 or 4 characters immediately after.

)를 중심에 두고 바로 앞의 3문자

와 바로 뒤의 3 문자

을 가져온 경우에,

와

의 2개의 문자 그룹이 얻어지며, 이어서 2개의 문자 그룹에 대하여 단어 분할이 수행되며, 예를 들어, 다음과 같은 단어 분할 결과가 얻어질 수 있다:

.After importing the same number of characters as the size of the window, word splitting is performed on the imported words that are close to this word. For example, this word (ie

3 characters in front of the center

And the 3 characters immediately after

If you imported

Wow

Two character groups of are obtained, followed by word division for two character groups, for example, the following word division results can be obtained:

.

Next, the weights of the respective words obtained after word division are determined. The results obtained after word division may have corresponding vectors <v ₁ , v ₂ , ..., v _n >, where n indicates the number of word division results of this word. In the above example, there are a total of four word splitting results, so n = 4, and v _i indicates the weight of the corresponding word (i = 1… n). For example, there are a plurality of weight-computing methods such as word frequency x inverse document frequency (TFIDF), BOOL (whether present), inverse document frequency (IDF), and pointwise mutual information (PMI). Usually, the number of occurrences of a word of context word contributes little to determining the meaning of this word, but whether or not it appears is of critical importance, and therefore, in a preferred embodiment of the present invention, IDF is used to calculate weights. (inverse document frequency) can be used.

Through the above process, respective words obtained after word division and the weights of the words can be obtained, and these words and the obtained weights can be used as the context of a given word.

In addition, by searching for unregistered words in the corpus and analyzing unregistered words in the dependency tree mode, the dependencies obtained by the analysis are used as the context of a given word.

Through the above context generating method, the context of an unregistered word can be obtained.

In step 203, the category to which the unregistered word belongs is determined based on the context and synonyms of the unregistered word.

The process of determining the category to which an unregistered word belongs based on the context and synonyms of the unregistered word can be implemented in a number of ways. In the following detailed description of FIGS. 3 to 5, a plurality of specific embodiments are provided for determining a category to which an unregistered word belongs based on context and synonyms of the unregistered word.

In the embodiment shown in Fig. 3, first, statistics for synonyms of an unregistered word may be created to determine each category to which these synonyms belong, and then a context of respective categories is generated, The context is obtained based on the context of all words contained in the respective categories generated from the corpus, and then known or conventional similarity calculation methods of the prior art between the context of the unregistered word and the contexts of the respective categories. Can be used to calculate the similarities of and finally, the category corresponding to the maximum similarity is determined as the category to which the unregistered word belongs.

In the embodiment shown in FIG. 4, first, contexts of synonyms may be generated from the corpus, which may use the same implementation method as generating the context of an unregistered word in step 202. Similarities between the context and the contexts of the synonyms are calculated, and a set is extracted from the synonyms of the unregistered word based on the calculated similarities, which may include a predetermined number of synonyms, and then, extraction Similarities corresponding to synonyms in the aggregated set and belonging to the same category are summed, and finally, a category to which an unregistered word belongs is determined based on the summed similarities. In the embodiment shown in FIG. 4, for example, a K-nearest neighbor (KNN) algorithm or other method known to those skilled in the art can be used.

In the embodiment shown in FIG. 5, first, contexts of synonyms can be generated from a corpus, and the similarity between the context of an unregistered word and the contexts of the synonyms is calculated, and then weighted by the weighted factors Better similarity results can be obtained, and furthermore, the categories to which unregistered words belong can be determined based on better similarity. Specifically, first, contexts of the synonyms can be generated from the corpus, similarities between the context of the unregistered word and the contexts of the synonyms are calculated, statistics about the categories to which the synonyms belong, and the synonyms associated with the synonyms Receive predetermined weighting factors, weight similarities corresponding to associated synonyms based on the received predetermined weighting factors, and extract a set from the synonyms of an unregistered word based on the weighted similarities- May include a predetermined number of synonyms-summing the weighted similarities of the synonyms in the set belonging to the same category, and determining the category to which the unregistered word belongs based on the summed similarities.

In the following, the embodiments of FIGS. 3 to 5 are described in detail.

3 illustrates a flowchart of a method for determining a category of an unregistered word, according to another embodiment of the present invention.

In step 301, an unregistered word is received.

(bing-jing, 얼음 결정체)이라고 가정하자.In this embodiment, the received unregistered word is

Suppose (bing-jing, ice crystals).

In step 302, words that share one or more word forming elements with an unregistered word are selected from the dictionary as synonyms of the unregistered word.

As mentioned above, the word forming rules may include word forming elements, element attributes and element relationships, etc. The word forming elements may further include letters and / or direct elements forming the word, and the like. Given a word and a dictionary, if the words in the dictionary share one or more word forming elements with the word unregistered, these words are all considered synonyms of the word unregistered and put into the thesaurus. The foregoing may be regarded as a specific implementation manner for selecting synonyms of unregistered words in a dictionary based on word formation rules.

(bing, 얼음)과 "晶"(jing, 결정)의 두 문자를 포함하는

(bing-jing, 얼음 결정체)이라고 가정하자. 사전에서 문자

을 포함하는 단어들이

(bing-dao, 스케이트),

(bing-gui, 얼음 저장고),

(bing-yu, 진눈깨비),

(bing-xue, 빙설)이고, 문자 "晶"을 포함하는 단어들이 "水晶"(shui-jing, 수정), "晶粒"(jing-li, 수정 입자),

(수정체)이며, 이때, 미등록 단어의 유의어 집합은 {

,

,

,

, "水晶", "晶粒",

}이라고 가정하자.In the following, an example of sharing the same letter is shown for illustration. For example, if an unregistered word

(bing, ice) and "晶" (jing, crystal) containing two characters

Suppose (bing-jing, ice crystals). Character from dictionary

Words that contain

(bing-dao, skate),

(bing-gui, ice cellar),

(bing-yu, sleet),

(bing-xue, ice and snow), and the words containing the letter "晶" are "shui-jing", "晶粒" (jing-li, crystal particle),

(Correction), where the set of synonyms for an unregistered word is {

,

,

,

, "水晶", "晶粒",

}

In step 303, a context of unregistered words is generated from the corpus.

The context of an unregistered word may be created by applying a window in dependent tree mode or in other ways known to those skilled in the art, and a specific implementation manner is described in step 202 and will not be described in detail herein.

In step 304, statistics are generated about the categories to which the synonyms belong.

In a step, categories of respective synonyms of an unregistered word are obtained, and then statistics are obtained for each of the obtained categories to determine all categories to which these synonyms belong.

은 카테고리 C1에 속하고,

은 카테고리 C2에 속하며,

은 카테고리 C4에 속하고,

은 카테고리 C4에 속하며, "水晶"은 카테고리 C3에 속하고, "晶粒"은 카테고리 C3에 속하며,

은 카테고리 C3에 속한다. 앞에서 언급한 바와 같이, 사전 내의 각각의 단어는 그의 품사, 카테고리, 단어 의미 및 예문, 그리고 기타 정보로 태깅될 수 있으며, 따라서 각각의 단어가 어느 카테고리에 속하는지가 사전으로부터 직접 도출될 수 있다. 게다가, 단어의 카테고리가 또한 수작업으로 설정될 수 있다.E.g,

Belongs to category C1,

Belongs to category C2,

Belongs to category C4,

Belongs to category C4, "水晶" belongs to category C3, "晶粒" belongs to category C3,

Belongs to category C3. As mentioned above, each word in a dictionary can be tagged with its parts of speech, category, word meaning and example, and other information, so that which category each word belongs to can be derived directly from the dictionary. In addition, the category of words can also be set manually.

를 포함하고, 카테고리 C2에 속하는 단어들은

를 포함하며, 카테고리 C3에 속하는 단어들은 "水晶", "晶粒",

를 포함하고, 카테고리 C4에 속하는 단어들은

와

를 포함한다.In this example, words belonging to category C1

And words belonging to category C2

Words belonging to category C3 include "水晶", "晶粒",

And words belonging to category C4

Wow

.

의 유의어들이 속하는 카테고리들이 각각 C1, C2, C3 및 C4라는 것이 도출될 수 있다.Unregistered words from

It can be derived that the categories to which the synonyms of are belonging to C1, C2, C3 and C4, respectively.

In step 305, the contexts of all words contained in each of the categories as the context of each of the categories are generated from the corpus.

로 기록되어 있는 카테고리 C1은

이외에

와

를 추가로 포함하고,

로 기록되어 있는 카테고리 C2는

이외에

를 추가로 포함하지만,

로 표시되어 있는 카테고리 C3는 단지 "水晶", "晶粒",

만을 포함하고,

로 기록되어 있는 카테고리 C4는 단지

만을 포함한다.In this step, all the words included in each category are first determined. E.g,

Category C1 recorded as

besides

Wow

, &Lt; / RTI &gt;

Category C2 recorded as

besides

Contains additional

Categories C3 marked with "Water", "晶粒",

Including only

Category C4 is listed as

Includes only.

의 문맥,

의 문맥, 및

의 문맥이 모두 C1의 문맥 = {

의 문맥,

의 문맥, 및

의 문맥}으로 기록되어 있는 카테고리 C1의 문맥으로서 역할할 수 있다.Depending on how the context of the word is generated from the corpus as described in step 202, the context of each word included in the four categories C1 to C4 can be generated. The context of all words contained in each category may be considered the context of this category, for example, included in category C1.

Context,

The context of, and

Context of all C1 contexts = {

Context,

The context of, and

Can serve as the context of category C1, recorded in the context of.

In step 306, the similarities between the context of the unregistered word and the context of each of the categories are calculated.

As mentioned above, the context of an unregistered word can be regarded as a vector, and the context of a category can also be regarded as a vector due to combining the context of all words included in the category, thus calculating the similarity between the two vectors. A vector cosine distance can be used for this purpose, and the cosine distance is represented by Equation 1.

Where X and Y are two vectors, n is the length of the X and Y vectors, and x _j and y _j are the jth elements in the X and Y vectors.

As far as the scenario of the present invention is concerned, X can be the context of an unregistered word, Y can be the context of a category, and x _j and y _j represent the weights corresponding to the j th element in the X and Y contexts, respectively. If the number of elements contained in the two contexts is different, all the elements of the two vectors can be extracted to reconstruct the new context vectors X 'and Y'. For X ', if the element in X' does not appear in X, the corresponding weight is set to zero. However, the calculation of the similarity between X and Y is implemented by calculating the similarity between X 'and Y' through equation (1). Through the calculation of the cosine distance, the similarity between the context of the unregistered word and the contexts of the respective categories can be obtained as follows:

은 단어

의 문맥을 나타내고, context(C1)은 카테고리 C1의 문맥을 나타내며, Sim(A, B)는 A와 B 간의 유사도를 나타낸다. 따라서, 미등록 단어

의 문맥과 카테고리들 C1, C2, C3, C4의 각자의 문맥 간의 유사도는 0.71, 0.67, 0.81 및 0.65이다.here,

Silver word

Context (C1) represents the context of category C1, and Sim (A, B) represents the similarity between A and B. Thus, unregistered words

The similarity between the context of and the context of each of the categories C1, C2, C3, C4 is 0.71, 0.67, 0.81 and 0.65.

In addition, the similarity between the two can also be calculated in other ways known to those skilled in the art.

In step 307, the category corresponding to the maximum similarity is determined as the category of unregistered words.

의 문맥과 카테고리 C3의 문맥 간의 유사도가 제일 높은 것을 알 수 있으며, 따라서 미등록 단어

의 카테고리가 카테고리 C3로서 결정된다.Comparing the similarities calculated in step 306, unregistered words

We can see that the similarity between the context of and the context of category C3 is the highest

The category of is determined as category C3.

4 shows a flowchart of a method for determining a category of unregistered words, according to another embodiment of the invention.

In step 401, an unregistered word is received.

(bing-jing, 얼음 결정체)이라고 가정하자.In this embodiment, the same as the embodiment of FIG. 3, the received unregistered word

Suppose (bing-jing, ice crystals).

In step 402, the part of speech of an unregistered word is determined.

의 품사가 명사라고 가정하자.There may be a plurality of ways to determine the part of speech of an unregistered word. For example, the part of speech of an unregistered word can be estimated using various known models or determined using manual tagging. In this example, unregistered words

Suppose that the part of speech is a noun.

In step 403, words sharing the word forming element are selected from the dictionary.

이라고 가정하면, 미등록 단어

과 하나의 문자를 공유하는 집합이

인 것으로 결정될 수 있다.For example, as in step 302, an unregistered word

Assuming unregistered words

And have a set to share one character

Can be determined to be.

Unlike step 302, this time, the set is not immediately considered to be a synonym of an unregistered word, and the part-of-speech filtering process of step 404 is further performed.

In step 404, as synonyms of the unregistered word, words having the same part-of-speech as the unregistered word are selected from the selected words.

As mentioned above, the word forming rule may include word forming elements, element attributes, element relationships, and the like, and element attributes may include, for example, a tag, a length, a part of speech, and the like of a word. . In the embodiment of FIG. 4, parts of speech within the word formation rule are used to implement the synonym selection of an unregistered word.

의 품사가 명사라는 것이 단계(402)로부터 결정될 수 있으며, 집합

내의 각자의 단어들의 품사는 사전에서 구해질 수 있고, 따라서 단계(404)에서, 이 집합 내의 명사들이 미등록 단어

의 유의어로 선택될 수 있다.In this embodiment, unregistered words

It can be determined from step 402 that the part of speech is a noun and the set

The part-of-speech of each word in can be obtained from a dictionary, so that in step 404, nouns in this set are unregistered words.

Can be chosen as a synonym of.

In step 405, a context of unregistered words is generated from the corpus.

The context of an unregistered word may be created by applying a window in dependent tree mode or in other ways known to those skilled in the art, and a specific implementation manner is described in step 202 and will not be described in detail herein.

In step 406, contexts of synonyms are generated from the corpus.

The contexts of the synonyms can be created by applying the window in dependent tree mode or in other ways known to those skilled in the art, and a detailed implementation manner is described in step 202 and will not be described in detail herein.

In step 407, the similarities between the context of the unregistered word and the contexts of the synonyms are calculated.

The context of an unregistered word can be regarded as a vector, and the context of a synonym can also be regarded as a vector, so vector cosine distance equation 1 can be used to calculate the similarity between two vectors.

As far as the scenario of the present invention is concerned, X can be in the context of an unregistered word, Y can be in the context of a synonym of an unregistered word, and x _j and y _j are weighted corresponding to the j th element in the X and Y contexts, respectively. Indicates. Thus, through the calculation of the cosine distance, the similarity between the context of an unregistered word and the context of its synonyms can be obtained as follows:

은 미등록 단어

의 문맥을 나타내고,

는 미등록 단어

의 유의어 의 문맥을 나타내며, Sim(A, B)는 A와 B 간의 유사도를 나타낸다. 따라서, 미등록 단어

의 문맥과 그의 유의어들

의 문맥들 간의 유사도는 각각 0.30, 0.67, 0.81, 0.74, 0.69 및 0.56이다.here,

Is an unregistered word

Indicates the context of

Is an unregistered word

Synonyms of Sim (A, B) represents the similarity between A and B. Thus, unregistered words

Context and its synonyms

The similarities between the contexts of are 0.30, 0.67, 0.81, 0.74, 0.69 and 0.56, respectively.

In step 408, based on the similarities, a set is extracted from the synonyms of the unregistered word.

The number of synonyms in the set to be extracted may be predetermined. In one example, the set may be set to include a predetermined number of synonyms, which may be any number less than or equal to the total number of synonyms of an unregistered word. In this embodiment, the predetermined number is represented by K, and assume that this predetermined number is 5, that is, K = 5.

First, the similarities obtained in step 407 may be sorted in descending order.

이다.In this embodiment, there are a total of six similarities calculated in step 407. Sorting the similarities in descending order yields the following sequence, 0.81, 0.74, 0.69, 0.67, 0.56, and 0.30, with the synonyms corresponding to the similarities in this sequence, respectively.

to be.

Then, the synonyms corresponding to the predetermined number of similarities at the top are extracted into a set.

이 집합의 요소로서 집합에 들어가기 위해 추출된다.In this embodiment, since there are a total of six synonyms for the unregistered word, although the predetermined number K = 5, the first five similarities are selected among the similarities arranged in descending order, that is, 0.81, 0.74, 0.69, 0.67, 0.56 And the synonyms corresponding to these similarities

As an element of this set, it is extracted to enter the set.

In step 409, similarities corresponding to synonyms in this set and belonging to the same category are summed.

를 포함하고, 카테고리 C3에 속하는 단어들은 "水晶", "晶粒",

을 포함하며, 카테고리 C4에 속하는 단어들은

을 포함한다. 따라서, 단계(408)에서 추출된 집합에 들어 있는 유의어들은 각각 C2, C3, C4에 속한다.In this step, the category to which the synonyms of the unregistered words belong is determined first, which can be implemented according to the manner of step 304, thus obtaining the same result as in step 304, ie words belonging to category C2

Words, belonging to category C3 are "水晶", "晶粒",

Words belonging to category C4

. Thus, the synonyms in the set extracted in step 408 belong to C2, C3, and C4, respectively.

Next, the similarities between the contexts of the synonyms belonging to the same category as the context of the unregistered word are summed, thereby obtaining similarities between the unregistered word and the respective categories, for example,

In step 410, the category of unregistered word is determined according to the sum of similarities.

과 카테고리 C3 간의 유사도가 가장 높은 것으로 도출되며, 그에 따라 카테고리 C3가 미등록 단어의 카테고리로서 결정될 수 있다.The similarities between the unregistered word obtained in step 409 and the respective categories are sorted, and accordingly the unregistered word

The similarity between and C3 is derived as the highest, so category C3 can be determined as a category of unregistered words.

In addition, in some embodiments of the present invention, other rules may be used to determine the category to which an unregistered word belongs based on the sum of similarities. For example, the maximum similarity between an unregistered word and its category may not be selected, and among the similarities, the category corresponding to the median value is determined as the category of the unregistered word.

5 is a flowchart of a method of determining a category of an unregistered word, according to another embodiment of the present invention.

In step 501, an unregistered word is received.

(dian-ji-chang, 전기 모터 공장)이라고 가정하자.In this embodiment, the received unregistered word is

Suppose (dian-ji-chang, electric motor factory).

In step 502, words sharing one or more word forming elements with an unregistered word are selected from the dictionary as synonyms of the unregistered word.

(tong-dian, 전원 켬),

(lai-dian, 걸려온 전화),

(da-dian-hua, 전화를 걸다),

(dian-qi-chang, 전기 제품 공장),

(chang-zhang, 공장장),

(chang-zhu, 공장 소유주)이다.Similar to step 302, the synonyms selected for this unregistered word based on the word formation rule of step 502

(tong-dian, power on),

(lai-dian, incoming call),

(da-dian-hua, make a phone call),

(dian-qi-chang, electrical appliance factory),

(chang-zhang, factory manager),

(chang-zhu, factory owner).

In step 503, a context of unregistered words is generated from the corpus.

The context of an unregistered word may be created by applying a window in dependent tree mode or in other ways known to those skilled in the art, and a specific implementation manner is described in step 202 and will not be described in detail herein.

In step 504, contexts of synonyms are generated from the corpus.

The contexts of the synonyms of the unregistered word may be generated by applying the window in dependent tree mode or in other ways known to those skilled in the art, and the specific implementation manner is described in step 202 and will not be described in detail herein.

In step 505, the similarities between the context of the unregistered word and the contexts of the synonyms are calculated.

의 문맥과 그 유의어들의 문맥들 사이의 유사도가 다음과 같이 구해질 수 있다.This step is similar to step 407 and will not be described here in detail. In step 505, unregistered words

The similarity between the context of and the contexts of its synonyms can be obtained as follows.

In step 506, statistics about the categories to which the synonyms belong are created.

를 포함하고, 카테고리 C2에 속하는 단어들은

와

를 포함하며, 카테고리 C3에 속하는 단어들은

를 포함하고, 카테고리 C4에 속하는 단어들은

를 포함한다.This step may be implemented in the manner described in step 304, and the following may be obtained, that is, words belonging to category C1

And words belonging to category C2

Wow

Words belonging to category C3

And words belonging to category C4

.

In step 507, predetermined weighting factors associated with the synonyms are received.

In determining the category, the context of the word is very important, and the structure information of the word is also very important in determining the category. Accordingly, the present invention proposes the concept of mixed similarity, that is, weighting similarities between the context of an unregistered word and the contexts of synonyms with the structure information of the word. In this embodiment, the structure information of the word is, for example, a predetermined weighting factor λ (w, w _i ). Weighting the similarity between the context of an unregistered word and the context of a synonym with a predetermined weighting factor is shown in Equation 2 below.

Here, w is an unregistered word, w _i is a synonym of an unregistered word, λ (w, w _i ) refers to a weighting factor based on the structure information of the unregistered word w and its synonym w _j , and CTS (w, w _i ) Denotes the similarity between the context of the unregistered word w and the context of its synonym w _j .

Multiple ways can be used to specify the weighting factor. In one embodiment, the designation of weighting factors must satisfy the following policies.

와 같이 미리 정해진 가중 인자 λ(w,w_i)를 λ₁으로 설정하고,If the unregistered word w and its synonym w _j share the last character and the second to last character, for example:

Set the predetermined weighting factor λ (w, w _i ) to λ ₁ ,

와 같이 미리 정해진 가중 인자 λ(w,w_i)를 λ₂로 설정하며,Otherwise, if the unregistered word w and its synonym w _j share the first and last characters, for example,

Set the predetermined weighting factor λ (w, w _i ) to λ ₂ ,

Otherwise, if the unregistered word w and its synonym w _j share the first letter or the last letter, for example, the predefined weighting factor λ (w, w _i ), such as λ (基 民, 市民) = λ ₃ Is set to λ ₃ ,

In other situations, the weighting factor λ (w, w _i ) is set to λ ₄ ,

Where λ ₁ ≧ λ ₂ ≧ λ ₃ ≧ λ ₄ and the corresponding numbers can be obtained through experiments.

In step 508, similarities corresponding to the associated synonyms may be weighted using predetermined weighting factors.

은 각각 λ₄ = 0.382로 설정되고,

는 λ₂ = 10로 설정된다.In one example, according to step 507,

Are each set to λ ₄ = 0.382,

Is set to λ ₂ = 10.

Similarities between the above-described weighting factors obtained according to step 507 and the context of the unregistered word and the contexts of the synonyms obtained according to step 505 may be applied to Equation 2, thus weighted similarity as shown below. Save them.

In step 509, based on the similarities, a set is extracted from the synonyms of the unregistered word.

This step is similar to step 408. First, the weighted similarities obtained in step 507 may be sorted in descending order. Then, the synonyms corresponding to the predetermined number of similarities at the top are extracted into a set.

가 집합의 요소로서 집합에 넣기 위해 추출된다.In this embodiment, assuming similarly that the predetermined number K = 5, the first five similarities among the similarities sorted in descending order are selected, that is, 3.0, 0.172, 0.115, 0.103, 0.076 are selected and correspond to these similarities. Synonyms

Is extracted to put into the set as an element of the set.

In step 510, weighted similarities corresponding to synonyms in this set and belonging to the same category are summed.

Step 510 is similar to step 409.

와

가 카테고리 C2에 속하고,

가 카테고리 C3에 속하며,

와

가 카테고리 C4에 속한다는 것을 알 수 있다. 따라서, 단계(509)에서 추출된 집합 내에 들어 있는 유의어들은 각각 C2,C3,C4에 속하며, 이 카테고리들이 바로 미등록 단어의 후보 카테고리들이다.First, from the result of step 506, the extracted set

Wow

Belongs to category C2,

Belongs to category C3,

Wow

It can be seen that belongs to category C4. Therefore, the synonyms included in the set extracted in step 509 belong to C2, C3 and C4, respectively, and these categories are candidate categories of unregistered words.

Next, the similarities between the contexts of the synonyms belonging to the same category as the context of the unregistered word are summed, thereby obtaining similarities between the unregistered word and the respective categories, for example,

In step 511, the category of unregistered word is determined according to the sum of the similarities.

과 카테고리 C3 간의 유사도가 가장 높은 것으로 도출되며, 그에 따라 카테고리 C3가 미등록 단어의 카테고리로서 결정될 수 있다.The similarities between the unregistered word obtained in step 510 and the respective categories are sorted, and accordingly the unregistered word

The similarity between and C3 is derived as the highest, so category C3 can be determined as a category of unregistered words.

According to the present invention, the synonyms of an unregistered word are selected from a dictionary based on a word formation rule, a context of an unregistered word is generated from a corpus, and accordingly determine a category to which the unregistered word belongs based on the context and synonyms of the unregistered word. do. The present invention solves the problem of low performance in the prior art, solves the problem of how to automatically select synonyms from existing dictionaries based on word formation rules to achieve category selection with high coverage, and In order to accurately calculate semantic similarity, we solve the problem of how to merge structure information and context information of words.

The method according to the invention can be implemented in software, hardware or a combination of hardware and software. The hardware portion may be implemented with dedicated logic and the software portion may be stored in memory and executed by a suitable instruction execution system such as, for example, a microprocessor, personal computer (PC) or mainframe.

It should be noted that for the purpose of understanding the present invention, the above description is known to those skilled in the art and omits some more specific technical details that may be essential to the implementation of the present invention.

The purpose of providing a description of the invention is to explain and describe the invention and is not intended to be exhaustive or to limit the invention to the form disclosed. Various modifications and variations are apparent to those skilled in the art.

Accordingly, selecting and describing preferred embodiments better describes the principles and practical application of the present invention, and all modifications and changes are defined by the appended claims without departing from the spirit of the invention. It is well understood by those skilled in the art that they fall within the scope of protection.

Claims (22)

delete As a method of determining the category of unregistered words,
Selecting synonyms of an unregistered word in a dictionary based on a word formation rule;
Generating a context of the unregistered word from a corpus; And
Determining a category of the unregistered word according to the context and the synonyms of the unregistered word
Including,
And said word forming rule comprises word forming elements, element attributes and element relationships. The method of claim 2, wherein the selecting of the synonyms of the unregistered word in the dictionary based on the word formation rule comprises:
Selecting from the dictionary words that share one or more word forming elements with the unregistered word as synonyms of the unregistered word. The method of claim 2, wherein the selecting of the synonyms of the unregistered word in the dictionary based on the word formation rule comprises:
Determining a part-of-speech of the unregistered word;
Selecting words from a dictionary that share one or more word forming elements with the unregistered words; And
Picking out words having the same part-of-speech from the selected words as synonyms of the unregistered word;
&Lt; / RTI &gt; The method of claim 2, wherein generating a context of an unregistered word from the corpus is as follows:
Retrieving the unregistered word from the corpus;
Applying a window to fetch characters near the unregistered word;
Performing word segmentation on the letters adjacent to the imported unregistered word; And
Determining weights of respective words obtained after the word division, in order to use the words obtained after the word division and the weights of the words as the context of the unregistered word.
&Lt; / RTI &gt; The method of claim 2, wherein generating a context of an unregistered word from the corpus is as follows:
Retrieving the unregistered word from the corpus; And
Analyzing the dependency of the unregistered word in dependent tree mode to use the dependency as the context of the unregistered word
&Lt; / RTI &gt; The method of claim 2, wherein the determining of the category of the unregistered word according to the context and the synonyms of the unregistered word comprises:
Creating statistics regarding the categories of synonyms;
Generating, from the corpus, contexts of all words included in respective categories as contexts of respective categories;
Calculating a similarity between the context of the unregistered word and the context of the respective categories; And
Determining a category corresponding to a maximum similarity as a category of the unregistered word
&Lt; / RTI &gt; The method of claim 2, wherein the determining of the category to which the unregistered word belongs depends on the context and the synonyms of the unregistered word.
Generating contexts of the synonyms from the corpus;
Calculating similarities between the context of the unregistered word and the contexts of the synonyms;
Extracting a set from the synonyms based on the similarities;
Summing similarities corresponding to synonyms in the set and belonging to the same category; And
Determining a category of the unregistered word according to the sum of the similarities
&Lt; / RTI &gt; The method of claim 2, wherein the determining of the category of the unregistered word according to the context and the synonyms of the unregistered word comprises:
Generating contexts of the synonyms from the corpus;
Calculating similarities between the context of the unregistered word and the contexts of the synonyms;
Creating statistics regarding the categories of synonyms;
Receiving predetermined weighting factors associated with the synonyms;
Weighting similarities corresponding to the associated synonyms using the predetermined weighting factors;
Extracting a set from the synonyms based on the similarities;
Summing the weighted similarities corresponding to the synonyms in the set and belonging to the same category; And
Determining a category of the unregistered word according to the sum of the similarities
&Lt; / RTI &gt; The method of claim 9, wherein the designation of the predetermined weighting factor is
A policy for setting a predetermined weighting factor associated with the category to λ ₁ if the unregistered word and a word in a category share a last letter and a last second letter,
Otherwise, if the unregistered word and a word in a category share a first letter and a last letter, a policy for setting a predetermined weighting factor associated with the category to λ ₂ ,
Otherwise, if the unregistered word and the word in the category share only the first letter or the last letter, the predetermined weighting factor associated with the category is set to λ ₃ , otherwise, the predetermined weighting factor associated with the category is set. satisfy the policy set to λ ₄ ,
Wherein λ ₁ ≥λ ₂ ≥λ ₃ ≥λ ₄ . The method of claim 8 or 9, wherein extracting a set from the synonyms based on the similarities,
Sorting the similarities in descending order; And
Extracting the synonyms as the set corresponding to a predetermined number of similarities at the highest level;
&Lt; / RTI &gt; delete An apparatus for determining a category of unregistered words,
A synonym selector configured to select synonyms of the unregistered word in a dictionary based on a word formation rule;
A context generator configured to generate a context of the unregistered word from a corpus; And
A category determiner configured to determine a category to which the unregistered word belongs according to context and synonyms of the unregistered word
Including,
And the word forming rule comprises word forming elements, element attributes and element relationships. The synonym selector of claim 13, wherein
Means for selecting words in the dictionary that share one or more word forming elements with the unregistered word as synonyms of the unregistered word. The synonym selector of claim 13, wherein
Means for determining a part of speech of the unregistered word;
Means for selecting words from a dictionary that share one or more word forming elements with the unregistered word; And
Means for selecting words having the same parts of speech from the selected words as the synonyms of the unregistered word;
/ RTI &gt; The method of claim 13, wherein the context generator,
Means for retrieving the unregistered word from the corpus;
Means for applying a window to retrieve characters in proximity to the unregistered word;
Means for performing word segmentation on letters adjacent to the imported unregistered word; And
Means for determining the weights of the words obtained after the word division, in order to use the respective words obtained after the word division and the weights of the words as the context of the unregistered word.
/ RTI &gt; The method of claim 13, wherein the context generator,
Means for retrieving the unregistered word from the corpus; And
Means for analyzing the dependency of the unregistered word in dependency tree mode to use dependency as the context of the unregistered word
/ RTI &gt; The method of claim 13, wherein the category determiner,
Means for producing statistics regarding the categories of synonyms;
Means for generating from said corpus the contexts of all words contained in respective categories as the context of respective categories;
Means for calculating a similarity between the context of the unregistered word and the contexts of the respective categories; And
Means for determining a category corresponding to a maximum similarity as a category to which the unregistered word belongs.
/ RTI &gt; The apparatus of claim 13, wherein the context generator comprises means for generating contexts of the synonyms from the corpus,
The category determiner,
Means for calculating similarities between the context of the unregistered word and the contexts of the synonyms;
Means for extracting a set from the synonyms based on the similarities;
Means for summing similarities corresponding to the synonyms in the set and belonging to the same category; And
Means for determining the category to which the unregistered word belongs according to the sum of similarities
/ RTI &gt; The apparatus of claim 13, wherein the context generator comprises means for generating contexts of the synonyms from the corpus,
The category determiner,
Means for calculating similarities between the context of the unregistered word and the contexts of the synonyms;
Means for producing statistics regarding the categories of synonyms;
Means for receiving predetermined weighting factors associated with the synonyms;
Means for weighting similarities corresponding to the synonyms associated with the received predetermined weighting factors;
Means for extracting a set from the synonyms based on the similarities;
Means for summing the weighted similarities corresponding to synonyms in the set and belonging to the same category; And
Means for determining a category to which the unregistered word belongs according to the similarities summed up
/ RTI &gt; The method of claim 20, wherein the designation of the predetermined weighting factor is
A policy for setting a predetermined weighting factor associated with the category to λ ₁ if the unregistered word and a word in a category share a last letter and a last second letter,
Otherwise, if the unregistered word and a word in a category share a first letter and a last letter, a policy for setting a predetermined weighting factor associated with the category to λ ₂ ,
Otherwise, if the unregistered word and the word in the category share only the first letter or the last letter, the predetermined weighting factor associated with the category is set to λ ₃ , otherwise, the predetermined weighting factor associated with the category is set. satisfy the policy set to λ ₄ ,
Wherein λ ₁ ≥λ ₂ ≥λ ₃ ≥λ ₄ . The apparatus of claim 19 or 20, wherein the means for extracting a set from the synonyms based on the similarities,
Means for sorting the similarities in descending order; And
Means for extracting the synonyms as the set corresponding to a predetermined number of similarities at the top;
/ RTI &gt;

Images