KR100886687B1 - Method and apparatus for auto-detecting of unregistered word in chinese language - Google Patents

Abstract

A method and an apparatus for auto-detecting an unregistered word in Chinese are provided to extract unregistered words from a web-document which is a translation target document by using HTML tag information, statistic information, monosyllable token information, etc. A removing unit(102) removes an HTML tag of an inputted web-document when receiving a web-document which includes Chinese sentences, and a tag classification unit(104) classifies each sentence in the document based on a meta tag and general tag processing manner. An extracting unit(106) using a general tag includes: a monosyllable based extracting module(116) extracts unregistered words on the basis of monosyllable token; and a verb based extracting module(118) extracts unregistered verb words which consist of 4 syllables. An extracting unit(108) using a meta tag extracts an unregistered word by using a word included in meta tag information, and a morpheme analyzing unit(110) analyzes morphemes and outputs the analyzed results. A radix based extracting module(114) extracts an unregistered word based on radixes by using the analyzed results.

Description

Chinese non-registered language automatic extraction method and apparatus {METHOD AND APPARATUS FOR AUTO-DETECTING OF UNREGISTERED WORD IN CHINESE LANGUAGE}

The present invention relates to Chinese translation technology. Unlike the method of extracting statistical proper nouns embedded in the existing machine translation system, the present invention provides information such as html tags, statistical information, single-syllable tokens, and targets a large amount of web documents that are actually translated. The present invention relates to a method and apparatus for automatically extracting Chinese non-registered words suitable for extracting Chinese non-registered words.

The present invention is derived from a study conducted as part of the IT growth engine technology development project of the Ministry of Information and Communication [Task Management Number: 2006-S-037-02, Title: Application-specific development of Korean-China automatic translation technology].

The existing researches on extracting Chinese non-registered words focused on extracting proper nouns. Proper nouns, including names of persons, names of persons, and names of organizations, are relatively likely to be unregistered words in machine translation systems. Therefore, many statistical methods are used to automatically extract proper nouns.

Using the corpus tagged with proper nouns, tagging information about the starting token of proper nouns, the token of the proper nouns, and the last token of the proper nouns is obtained. Create an estimation module and use it to estimate proper nouns. Since the common Chinese words of proper nouns can be used as personal names, the performance of extracting names is lower than that of names or organization names. Since this method requires a large learning corpus with proper noun information, its performance depends on the quantity and quality of the corpus. Therefore, if a low performance proper noun estimation module is applied to a machine translation system, wrong estimation results may be misapplied, leading to a decrease in translation performance.

In the prior art Chinese translation system operating as described above, the unregistered words in the heavy machine translation system for the web document include not only proper nouns, but also new words, abbreviations, terminology, etc., which are continuously made on the web. . Therefore, it is impossible to extract unregistered words such as new words using the existing proper noun estimation method.

The reason is that there is no learning corpus, and even if there is a learning corpus, there is a problem that the estimation is impossible because there is no outstanding characteristic of the constituents that constitute the new word.

Accordingly, the present invention provides a method and apparatus for automatically extracting Chinese non-registered words that can extract Chinese non-registered words using html tag information, statistical information, and single-syllable token information for a large amount of web documents that are actually translated.

According to an embodiment of the present invention, when a web document including a Chinese sentence is received, a process of removing the html tag of the input web document and classifying the meta tag and the general tag processing method for each sentence in the web document Process, outputting the analysis result through morphological analysis, using the analysis result to extract the unregistered words centered on the root, a method to extract the unregistered words based on the single syllable token, and a four syllable At least one of a method of extracting a non-registered verb, a method of extracting a word capable unregistered word by determining whether a single syllable token is a word, and a method of extracting an unregistered word using a word included in the meta tag information Extracting unregistered words using the method.

According to an embodiment of the present invention, when the web document including a Chinese sentence is received, a removal unit for removing the html tag of the input web document, and the meta tag and the general tag processing method for each sentence in the web document A tag classification unit, a morpheme analysis unit which performs morphological analysis, and outputs an analysis result, a root-centered extraction module based on the root-based unregistered word using the analysis result, and a single syllable to extract the unregistered word based on the single-syllable token An extraction method unit using a general tag including a center extraction module, a verb center extraction module that extracts four syllables non-registered words, and a single syllable token to determine whether a word is possible in the single syllable token. Extraction method using the word availability of the word, and extracts the unregistered words using the words contained in the meta tag information It includes parts of extraction method using the tag.

In the present invention, the effects obtained by the representative ones of the disclosed inventions will be briefly described as follows.

According to the present invention, it is possible to construct a dictionary for Chinese analysis easily and quickly through the Chinese unregistered word extraction method. In particular, the dictionary reinforcement for new words and unregistered words has the effect of greatly improving the accuracy in the actual web document translation system. .

Hereinafter, the operating principle of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. Terms to be described later are terms defined in consideration of functions in the present invention, and may be changed according to intentions or customs of users or operators. Therefore, the definition should be made based on the contents throughout the specification.

The present invention is to determine whether the HTML tag information, statistical information through morphological analysis, and whether or not a single-syllable token can be used for words of a large-capacity web document that is actually translated, and extracts Chinese non-registered words using the information.

That is, sentences classified by meta tag through tag classification module are classified by token length, frequency information is extracted and extracted, and sentences classified by general tag are unregistered words extraction method using statistical information. The unregistered words are extracted by using a module that extracts the information based on a single syllable token and a module that extracts the tokens tagged with consecutive verbs. In addition, an unregistered word is extracted using a single syllable token determination module and an unregistered word estimating module using the result as a method for extracting an unregistered word using whether a single syllable token is available.

1 is a block diagram showing the structure of an automatic non-registered word extraction apparatus according to a preferred embodiment of the present invention.

Referring to FIG. 1, the apparatus for automatically extracting unregistered words 100 includes an HTML detagging module 102, a tag classification module 104, an extraction method unit 108 using a general tag, and an extraction method unit using meta tags. The extraction method unit 108 using the general tag includes a statistical extraction method unit 112 and an extraction method unit 122 using the word syllable of a single syllable token.

The process of performing automatic extraction of Chinese non-registered words using the non-registered word automatic extraction device 100 will be described in detail. When an actual web document updated in real time is input to the non-registered word automatic extracting device 100, first, HTML The detagging module 102 removes the html tag from the web document and passes it to the tag classification module 104. The tag classification module 104 classifies the sentences into meta tags, link tags, and general tags for each sentence, and the sentences classified as meta tags are processed by the extraction method unit 108 using the meta tags. The sentences classified as are processed by the extraction method unit 106 using the general tag to extract the unregistered words in the sentences.

Here, the method for extracting unregistered words using meta tag information is to easily and quickly construct unregistered words by using meta information that has already been constructed by the web document creator, and the statistical extraction method using morphological analysis results is based on new words. By using the property that unregistered words are cut into single syllable tokens, we want to increase the recall rate of unregistered words.

First, the processing method in the extraction method unit 108 using the meta tag will be described. In order to summarize the contents of the document when the web developer prepares the web document, the keyword and the description of the meta tag are summarized. ) Attach keywords representing the document content to the attribute. Most of these keywords are nouns, many of which are synonymous or terminology. By taking advantage of the large number of newly constructed web documents quickly and easily, collecting keyword values used in meta tags makes it possible to extract unregistered words with less effort. By classifying keywords by token length and finding the frequency, single words and compound nouns can be extracted.

That is, by classifying keywords by token length, a word consisting of one token calculates a frequency, and if a number of times more than a predetermined frequency is searched for, a word is extracted as an unregistered word. In addition, for words consisting of more than two roots, morphological analysis is first performed (132), and if the result of the morphological analysis is a noun, the compound noun 134 is extracted. As a result, the extracted unregistered word is extracted as an unregistered word estimation result 3 (134).

Meanwhile, referring to the processing method of the extraction method unit 106 using the general tag, the statistical extraction method 112 in the extraction method unit 106 using the half tag is the root center extraction module 114 and the single syllable token center extraction. It consists of three sub-modules 114, 116, 118, such as module 116, verb center extraction module 118. The three sub-modules 114, 116, and 118 each have advantages as independent modules. First, the root center extraction module 114 is a module for extracting unregistered words based on roots, where roots are proper nouns such as names, geography, companies, and schools, such as "山, 節, 街頭, 村, 學校, 公司". It is the following suffix. It is a method of extracting unregistered words by using the characteristic that proper nouns are composed around these suffixes.

Therefore, a high-frequency root dictionary is constructed using the existing dictionary and the corpus, the root is searched using the root dictionary from the results obtained through morphological analysis, and the non-registered words using the part-of-speech information and the word length based on the found root By estimating a range, it extracts as an unregistered word.

"라는 미등록어가 포함된 중국어 입력문장이 들어오면 "深/형용사

/명사" 으로 단음절 토큰으로 분리하여 분석될 가능성이 매우 높다. "深"와 같은 단음절 토큰은 중국어에서 "깊다"라는 형용사로서 자주 쓰이는 일반 단어이기 때문에 단어분석이 잘못되면 번역 성능에 치명적인 저하를 초래한다. 연속된 단음절 토큰으로 분리된 문자열은 원래 하나의 단어일 확률이 크며, 따라서 이러한 단음절 토큰열은 미등록어 추출 후보로 선택하고, 빈도 정보를 구하여 기 설정된 빈도수를 초과하여 기재되어 있는 경우는 미등록어로서 추출한다. The single syllable token-centered extraction module 116 is a module for extracting unregistered words based on the single syllable tokens. The unregistered unregistered words have a characteristic of being separated into single syllable tokens in the morpheme analysis 110. For example "

When you enter a Chinese input sentence containing "unregistered words""深 / adjective

/ Noun "is very likely to be separated into single-syllable tokens. A single-syllable token such as" 深 "is a common word often used as an adjective" deep "in Chinese. A string separated by consecutive single-syllable tokens is likely to be a single word. Therefore, such single-syllable token strings are selected as candidates for extracting unregistered words, and if the frequency information is obtained and listed above the preset frequency, it is not registered. Extract as fish.

The verb center extraction module 118 is a module for extracting unregistered words based on tokens tagged with consecutive verbs, and many Chinese words include four syllables, among which a lot of lion words are included. Although it is not a idiom, there are many words that are often used in practice. Most of these words are unregistered terminology, and to extract them, we extract 4 syllable unregistered term using successive verb string frequency. In this way, the three sub-modules 114, 116 and 118 are extracted as the unregistered word estimation result 1 (120).

/마우스깔개"와 같이 "鼠標/마우스

/깔다" 두음절 + 한음절로 구성된 미등록어에 대해서는 추출할 수 없다. 이 문제를 해결하기 위하여, 정확히 태깅된 학습코퍼스를 이용하여 학습을 진행한다. In the extraction method unit 106 using the general tag, the extraction method unit 122 using the word availability of the single syllable token may be regarded as a complementary device to the extraction method of the non-registered word using the conventional single syllable token string. The single syllable token string method uses only one syllable token string because "

"Mouse / mouse"

It is not possible to extract unregistered words consisting of two syllables + one syllable. To solve this problem, the learning is performed using a correctly tagged learning corpus.

The learning corpus used in the conventional method was limited in both quantity and quality because it was a learning corpus with proper noun information. However, since the learning corpus used in this device is sufficient for word separation and part-of-speech information, a large amount of learning corpus is used. You can get it easily.

Accordingly, the word syllable determination module 124 of the single syllable token obtains statistical information such as a probability that each Chinese character becomes a word alone, a probability before a word, and a probability after the word using information in the learning corpus. In addition, when single syllables are independently available as words, statistical information is supplemented by referring to parts of speech information of left and right words. Through this learned module, it is determined whether single-syllable tokens that are inputted are independently possible in one word.

Subsequently, the single syllable token-based non-registered word estimation module 126 determines that if the probability value does not exceed a preset threshold, it cannot be separated independently, that is, it is incorrectly separated, and uses the part-of-speech information based on the token. Try to expand from side to side. The unregistered word candidates estimated based on the extended heuristics are classified according to length and frequency is obtained to extract the unregistered word estimation result 2 (128).

2 is a flowchart illustrating a procedure for automatically extracting unregistered words of an apparatus for automatically extracting unregistered words according to a preferred embodiment of the present invention.

Referring to FIG. 2, when the web document is input in step 200, the HTML tag is removed in step 202, and tag classification is performed for each sentence of the input web document in step 204. In step 206, if the tag is a meta tag, the process proceeds to step 208 to classify the words for each token length, and in step 210, if the token length is one token, the process proceeds to step 212. If the case is extracted as a high frequency word proceeds to step 228. In addition, if the word is more than two tokens, the process proceeds to step 214 to analyze the morpheme, and if the analyzed morpheme is a noun, it is extracted as a compound noun and proceeds to step 228.

Meanwhile, in step 206, when the tag is a general tag, in step 218, when the tag is a general tag, the morpheme is analyzed in step 218, first, in step 220, an unregistered word is extracted around a root, and in step 222, an unregistered word is centered on a single syllable. Extract In step 224, unregistered words centered on verbs are extracted, and in step 226, unregistered words centered on single-syllable tokens are extracted. In this case, the order of steps 220 to 226 may be changed according to the implementation method. Thereafter, in step 228, the words extracted as the unregistered words are finally outputted.

As described above, the apparatus for extracting Chinese non-registered words using a large-capacity web document analyzes and uses various characteristics to extract Chinese non-registered words, and uses as many and accurate non-registered words as possible for new words, abbreviations, terminology, unregistered verbs, and the like. By extracting, Chinese analysis dictionary construction is more easily and efficiently performed.

As described above, the present invention targets a large-capacity web document, which is actually translated, determines HTML tag information, statistical information through morphological analysis, and whether or not a single syllable token is possible, and uses the information to determine the non-Chinese language. Extract

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the scope of the following claims, but also by those equivalent to the scope of the claims.

1 is a block diagram showing the structure of an automatic non-registered word extraction device according to an embodiment of the present invention;

2 is a flowchart illustrating a procedure for automatically extracting unregistered words of an apparatus for automatically extracting unregistered words according to an embodiment of the present invention;

3 is a block diagram showing the structure of a location information server according to a preferred embodiment of the present invention;

<Explanation of Signs of Major Parts of Drawings>

100: unregistered word automatic extraction device 102: HTML tag removal

104: tag classification 106: extraction method using a general tag

108: Extraction Method Using Meta Tag 110: Morphological Analysis

112: statistical extraction method 114: root center extraction

116: single-syllable token-based extraction 118: verb-centered extraction

122: Extraction method using word availability of single-syllable token

124: Determination of word presence of single syllable token

126: single syllable token-based non-registered word estimation module

130: high frequency word extraction 132: stemming

134: Extract compound nouns

Claims (8)

When receiving a web document containing a Chinese sentence, removing the html tag of the input web document, Classifying the meta tag and the general tag processing method for each sentence in the web document; Process of morphological analysis and outputting the analysis result, A method of extracting a non-registered word centered on a root, a method of extracting an unregistered word centered on a single syllable token, a method of extracting a non-registered word consisting of four syllables, and whether the word of the single-syllable token is possible using the analysis result Extracting a non-registered word using at least one of a method of extracting a word possible unregistered word and a method of extracting an unregistered word using a word included in the meta tag information Chinese non-registered language automatic extraction method comprising a. The method of claim 1, Building a root dictionary using the analysis result, extracting unregistered words centering on the root; Obtaining frequency for single syllables and extracting the non-registered words when the frequency is exceeded, The process of extracting verbs of four syllable unregistered words by obtaining a frequency for verb strings tagged as verbs Chinese non-registered language automatic extraction method comprising a. The method of claim 1, The method, Determining whether a single syllable token can be singular using a learning corpus; A process of extracting unregistered words through expansion using left and right context information on tokens that are impossible as single syllable words Chinese non-registered language automatic extraction method comprising a. The method of claim 1, The method, Classifying words included in the meta tag information by token length; Extracting a word having a token length of 1 and exceeding a preset frequency as an unregistered word; If the token length is 2 or more and the result of the morpheme analysis is a noun, the process of extracting a non-registered word Chinese non-registered language automatic extraction method comprising a. A removal unit for removing an html tag of the input web document when receiving a web document including a Chinese sentence; A tag classification unit classifying the meta tag and the general tag processing method for each sentence in the web document; A morpheme analysis unit for outputting an analysis result by performing a morphological analysis; Root-centered extraction module for extracting unregistered words centered on roots using the analysis results, Mono-syllable centered extraction module for extracting unregistered words based on single-syllable tokens, and Verb-centered extraction module for extracting unregistered verbs with 4 syllables Extraction method using a general tag including a, An extraction method unit using the word availability of the single-syllable token to determine whether the single-syllable token is capable of extracting the word and the non-word registered words; Extraction method using a meta tag to extract the unregistered words using the words included in the meta tag information Chinese unregistered automatic extraction system comprising a. The method of claim 5, Extraction method unit using the general tag, Using the morphological analysis results to build a root dictionary, extract the unregistered words around the root, Obtain frequency for single syllable list, extract it as unregistered word if it exceeds preset frequency, A Chinese non-registered word automatic extraction system, characterized by extracting verbs of four syllable unregistered words by obtaining a frequency for a verb string tagged as a verb. The method of claim 5, Extraction method unit using the word availability of the single-syllable token, The Chinese non-registered word automatic extraction system, characterized in that it is possible to determine whether the single word of the single-syllable token is possible using a learning corpus, and to extract the non-registered word through expansion using left and right context information on the token which is impossible as the single-syllable word. The method of claim 5, The extraction method unit using the meta tag, The words included in the meta tag information are classified by token length, and when the token length is 1, words that exceed a preset frequency are extracted as unregistered words. When the token length is 2 or more and the result of the morpheme analysis is a noun, the Chinese non-registered language automatic extraction system, characterized in that the extraction of the non-registered language.

Images