KR100910275B1 - Method and apparatus for automatic extraction of transliteration pairs in dual language documents - Google Patents

Abstract

The present invention relates to a method and apparatus for automatically extracting a tuning fork notation band pair in a bilingual document, and extracting a bilingual document represented by a bilingual language existing on the Internet and separating the bilingual document into a single language document represented by each monolingual document. A bilingual document extraction module for extracting a phonetic notation candidate word extracting module for extracting a phonetic notation candidate words from a single language document among monolingual documents separated from the bilingual document extraction module, and the phonetic notation candidate After selecting the candidate words to be expressed from the tuning fork candidate words extracted from the word extraction module, the tuning fork corresponding to the selected tuning note candidate words in the other monolingual documents using the dynamic window technique or the tokenizer technique. By including the tuning fork notation band pair extraction module for automatically extracting the notation band pair, Automatically extract a pair of phonetic notation bands using a large number of bilingual documents on the Web to build a large volume of unregistered dictionaries, and apply it to cross-language search or machine language translation service for effective performance improvement. can do.

Phonetic notation, Statistics-based phonetic notation model, Dynamic window technique, Tokenizer technique, Bilingual document, Tunnel notation band pair, English, Chinese

Description

Method and apparatus for automatically extracting tuning fork band pairs from bilingual documents {METHOD AND APPARATUS FOR AUTOMATIC EXTRACTION OF TRANSLITERATION PAIRS IN DUAL LANGUAGE DOCUMENTS}

The present invention relates to a method and apparatus for automatically extracting a tuning fork band pair in a bilingual document, and more particularly, to automatically extract a tuning fork band pair using a large amount of bilingual documents on the Web. The present invention relates to a method and apparatus for automatically extracting a pair of phonetic notation band pairs from a bilingual document that can improve the performance by constructing a non-registered dictionary dictionary of DB and applying it to a cross-language search or machine language translation service.

Nowadays, as the internet develops and cultural exchanges between countries increase, many foreign languages are newly generated. Most of these foreign words are transliteration, which causes big problems in language processing. In the past, various methods have been proposed to solve these tone notations. These methods can be largely divided into the following two.

The first is the method of automatically generating the tone notation {Reference: J.-H. Oh, K.-S. Choi, "An English Korean transliteration model using pronunciation and contextual rules", in: Proceedings of the 19th International Conference on Computational Linguistics (COLING), Taipei, Taiwan, pp. 758-764, (2002)}.

The automatic generation of the phonetic notation refers to automatically generating a phonetic notation corresponding to a given foreign language word. For example, "Cliton" refers to the automatic creation of Korean phonetic notation "Clinton" when there is an English word.

On the other hand, these existing methods do not perform high performance due to the diversity of the phonetic notation and various habits of making the phonetic notation of people. For example, when there is an English word "Scofield", it would be correct to originally label it "Scofield" or "Scofield".

However, people do not translate it like this, but they translate it as "Sukho Phil." In this case, it is almost impossible to generate "Shoho Phil" using the conventional tuning method. Because today's computer technology does not yet have artificial intelligence that can detect human habits.

In particular, in languages such as Chinese, not only the meaning of words is considered but also the meaning of the Chinese characters, so this problem is more serious.

The second method is the automatic extraction of tuning fork band pairs from bilingual documents. {Ref .: Richard Sproat, Tao Tao, ChengXiang Zhai, "Named Entity Tranliteration with Comparable Corpora", in: Proceddings of the 21st International Conference on Computational Linguistics. (2006)}.

The automatic extraction of the tuning fork band pairs generally shows higher performance than the automatic generation of the tuning fork notation. However, we have not yet achieved satisfactory performance.

In other words, the conventional method for automatically extracting the tuning fork band pairs extracts the tuning fork band pairs by first extracting the phonetic notation candidates from each of the two languages and then calculating the voice similarity between the candidates.

Meanwhile, in this conventional method, the tuning fork candidate is greatly influenced by the extraction result. The best language for extracting the phonetic transcription candidates is English because the first letter of the proper noun in English is capitalized and there is a space between words.

However, in the case of Chinese, since there is no spacing and no capital letters, the extraction of the candidates for the tuning note remains a difficult topic and does not have high performance. Therefore, if a candidate is extracted from a language that is difficult to extract the phonetic representation of candidates, the extracted candidate includes a large number of errors. Therefore, it is not possible to expect good performance of the extraction of the tuning fork band pairs by this conventional method.

In order to solve this problem, we proposed a method for extracting the phonetic notation only from the English document and extracting the corresponding phonetic notation from the Chinese document using the statistical-based phonetic notation model based on the candidate. {Reference: C.- J. Lee, J.S. Chang, J.-S.R. Jang, "Extraction of transliteration pairs from parallel corpora using a statistical transliteration model", in: Information Sciences 176, 67-90 (2006)}.

However, this method has greatly improved performance compared to the conventional method described above. However, the longer the Chinese sentence length, the lower the performance. Also, the rule-based post-processing technique has a limitation in that it can only deal with a specific situation.

The present invention has been made to solve the above-described problems, an object of the present invention is to automatically extract the tuning fork notation band pair using a large number of bilingual documents on the Web (Web) to generate a large number of unregistered dictionary dictionary DB The present invention provides a method and apparatus for automatically extracting a tuning fork notation pair pair from a bilingual document that can be improved by applying it to a cross-language search or a machine language translation service.

In order to achieve the above object, a first aspect of the present invention comprises the steps of: (a) extracting a bilingual document represented in a bilingual language existing on the Internet and separating it into monolingual documents represented in each monolingual language; (b) extracting the phonetic transcription candidate words from any one of the separated monolingual documents; And (c) selecting a candidate word to be expressed from the extracted tuning fork candidate words, and then applying the selected tuning fork candidate word to another selected monolingual document using a dynamic window technique based on the selected tuning fork candidate word. The present invention provides a method for automatically extracting a tuning fork band pair in a bilingual document including automatically extracting a corresponding tuning fork band pair.

A second aspect of the present invention includes the steps of: (a ') extracting bilingual documents represented in bilingual languages existing on the Internet and separating them into monolingual documents represented in each monolingual language; (b ') extracting the phonetic transcription candidate words from any one of the separated monolingual documents; And (c ') selecting candidate words to be written from the extracted phonetic note candidate words, and then applying a tokenizer technique using word separation to correspond to the selected tuning note candidate words in the other monolingual documents. The present invention provides a method for automatically extracting a tuning fork band pair in a bilingual document, including automatically extracting a tuning fork notation band pair.

Preferably, the tokenizer technique divides each sentence in the other monolingual document into parts based on characters not used in the phonetic notation, and then scores each segment by using a statistical-based phonetic notation model. It is possible to automatically extract the tuning fork band pair corresponding to the selected tuning fork candidate word by backtracking the string having the highest value.

A third aspect of the present invention provides a method for manufacturing a bilingual document, comprising: (a ") extracting a bilingual document represented in a bilingual language existing on the Internet and separating the bilingual document represented in each monolingual document into a single language; Extracting the phonetic transcription candidate words from any one of the monolingual documents; (c ") selecting candidate words to be written from the extracted phoneme notation candidate words, and dividing each sentence in the other monolingual document into various parts based on characters not used in the phonetic notation; and ( d ") for each segmented portion, using a dynamic window technique to automatically extract a tone pair band pair corresponding to the selected note tone candidate word from the other monolingual document, using the dynamic window technique. It is to provide a method for automatic band pair extraction.

Preferably, the dynamic window technique may include: setting a range of possible phonetic notations of the other monolingual based on the selected length of the tuning fork candidate words; Obtaining maximum phonetic likelihood probability values for the character string including the selected tuning note candidate word and the current window while moving forward through each window within the set window length range; And automatically extracting a tuning fork notation pair by tracing a string corresponding to the largest value among the maximum negative likelihood probability values in each window.

Preferably, when the selected phonetic note candidate word is an English word, the other single language is Chinese, and the length of the English word is L, the window length range may be set between L / 3 and L.

Preferably, the letters not used in the phonetic notation may be made of at least one of punctuation marks, numbers, spaces, or English letters.

Preferably, if the other monolingual document is Chinese, characters not used in the phonetic notation may be made up of surveys.

Preferably, the monolingual document extracting the phonetic transcription candidate words is an English document, and the other monolingual document is a Chinese document.

Preferably, the phonetic transcription candidate words are proper nouns.

Preferably, the bilingual document may include parallel corpus or comparable corpus in all language pairs, including language pairs consisting of Chinese and English.

Preferably, the extracted tuning fork notation band pair may further include registering in a pre-registered non-registered word dictionary DB.

Preferably, the method may further include performing a cross-language search service using the tuning fork notation band pair registered in the non-registered word dictionary DB.

Preferably, the method may further include performing a machine language translation service using the tuning fork notation band pair registered in the non-registered dictionary dictionary DB.

A fourth aspect of the present invention is to provide a computer-readable recording medium having recorded thereon a program which can be executed by the computer for automatically extracting the tuning fork band pairs in the bilingual document.

A fifth aspect of the present invention includes a bilingual document extraction module for extracting bilingual documents expressed in bilingual languages existing on the Internet and separating them into monolingual documents expressed in respective monolingual languages; A phoneme notation candidate word extraction module for extracting a phonetic notation candidate words from any single language document among monolingual documents separated from the bilingual document extraction module; And selecting a candidate word to be expressed in the tuning fork candidate words extracted from the tuning fork candidate word extraction module, and using the dynamic window technique or the tokenizer technique, the selected tuning fork representation candidate in the other monolingual document. The present invention provides an apparatus for automatically extracting a tuning fork band pair in a bilingual document including a tuning fork band band extracting module for automatically extracting a tuning fork band pair corresponding to a word.

A sixth aspect of the present invention includes a bilingual document extraction module for extracting bilingual documents expressed in bilingual languages existing on the Internet and separating them into monolingual documents expressed in respective monolingual languages; A phoneme notation candidate word extraction module for extracting a phonetic notation candidate words from any single language document among monolingual documents separated from the bilingual document extraction module; And selecting candidate words to be written in the tuning fork notation candidate words extracted from the phonetic notation candidate word extraction module, and then dividing each sentence in the other monolingual document into several parts based on characters not used in the phonetic notation. A bilingual document including a tuning fork notation band pair extraction module for automatically extracting a tuning fork notation band pair corresponding to the selected phonetic notation candidate word from the other monolingual documents using the dynamic window technique for each divided portion. To provide an automatic extraction of the tuning fork notation band pair in.

Preferably, the non-registered word dictionary DB for storing the tuning fork notation band pair extracted from the tuning fork notation band pair extraction module may be further included.

Preferably, the non-registered word dictionary DB may be used for a search service for a cross language input through the Internet.

Preferably, the non-registered language dictionary DB may be used to perform a machine language translation service task.

According to the method and apparatus for automatically extracting the tuning fork notation band pairs in the bilingual document of the present invention as described above, the tuning fork notation band pair is automatically extracted using a large amount of bilingual documents on the Web. It is possible to improve performance by constructing a dictionary database of non-registered languages and applying it to cross-language search or machine language translation service.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, embodiments of the present invention illustrated below may be modified in many different forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art.

1 is an overall block diagram illustrating an apparatus for automatically extracting a tuning fork notation band pair in a bilingual document according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus for automatically extracting a tuning fork notation band pair in a bilingual document according to an embodiment of the present invention includes a bilingual document extraction module 100, a tuning fork candidate word extraction module 200, and a tuning fork. A marking band pair extraction module 300 is included.

Here, the bilingual document extraction module 100 extracts a bilingual document expressed in bilingual languages existing on a large amount of webs distributed through the Internet, and expresses a bilingual document represented in each monolingual document. To separate them (eg, English documents, Chinese documents, etc.).

The bilingual document may include, for example, parallel corpus or comparable corpus in all language pairs, including language pairs consisting of Chinese and English.

The phonetic transcription candidate word extraction module 200 may include the phonetic transcription candidate words (eg, proper nouns) in one monolingual document (eg, an English document) among monolingual documents separated from the bilingual document extraction module 100. It performs the function of extracting.

The tuning fork band pair extraction module 300 selects a candidate word to be written from the tuning fork notation candidate words extracted from the tuning fork candidate word extraction module 200 and then uses a dynamic window technique or a tokenizer technique to be described later. And extracting a tuning fork notation pair corresponding to the selected tuning fork candidate word from another monolingual document (eg, a Chinese document).

In addition, the tuning fork notation band pair extraction module 300 selects a candidate word to be written in the tuning fork notation candidate words extracted from the tuning fork candidate word extraction module 200 and then performs the remaining monolingual document (eg, a Chinese document). Each sentence within is divided into parts based on characters not used in the phonetic notation (for example, punctuation, numbers, spacing, English letters, surveys, etc.), and each part is divided using the dynamic window technique. A tuning fork band pair corresponding to the selected tuning fork candidate word may be automatically extracted from another monolingual document.

In addition, a large-capacity non-registered dictionary DB 400 for storing the tuning fork notation band pair extracted from the tuning fork notation band pair extraction module 300 may be further provided.

The non-registered dictionary DB 400 may be used for a search (eg, Chinese query) service for a cross-language input from a user terminal (PC) connected through the Internet.

In addition, the non-registered dictionary DB 400 may be used, for example, to perform a machine language translation service for real-time translation on a web.

In order to achieve the above-mentioned object, a perfect tuning fork notation band pair extraction is required close to about 100%, and the tuning fork notation band pair extraction module 300 of the present invention is one language for which the tuning fork candidate words are well extracted (eg, English) A method of extracting a tuning fork notation word only from a document and extracting a corresponding tuning fork band pair from another language document (for example, a Chinese document) that is difficult to extract the tuning fork candidate word based on the tuning fork candidate word. , By using the dynamic window technique or the tokenizer technique, to increase the performance of less than about 90% to about 99%, fully automating the existing manual non-registered dictionary dictionary construction and constructing a large number of non-registered dictionary dictionaries It is possible to use this, and it is possible to achieve a great performance improvement such as cross-language search or machine language translation service.

The following describes in detail how to extract the tuning fork band pair from a bilingual document using Chinese and English as an example.

In the bilingual (English-Chinese) document of the present invention, the method for automatically extracting the tuning fork notation band pair, first, by applying the tuning fork notation candidate word extraction module 200 to the English sentence of the English-Chinese parallel corpus, After extracting proper nouns, only the English words to be represented in the phonetic tone are selected, and the phonetic notation band pairs are extracted from the corresponding Chinese sentences.

"(클린턴)이란 중국어 단어는 먼저 "KeLinDun"이란 병음으로 변환한 후, 영어 단어인 "Clinton"과 중국어 병음인 "KeLinDun"의 음성적 유사도를 계산하여 비교한다.Subsequently, the Chinese tuning fork band pair extraction is generally compared with English words using the Roman notation of Chinese kanji, that is, Pinyin. For example, "

The Chinese word "(Clinton) is first converted into the Pinyin word" KeLinDun "and then compared by calculating the phonetic similarity between the English word" Clinton "and the Chinese Pinyin" KeLinDun ".

｜Clinton)로 치환되어 P(KeLinDun｜Clinton)확률을 구하는 문제로 전환할 수 있다.In the present invention, it is assumed that E is English, C is Chinese, and TU (Transliteration Unit) is a tuning fork unit. Then the conditional probability P (C | E) is equal to P (

Substituting Clinton, it can be converted to the problem of P (KeLinDun | Clinton) probability.

｜Clinton)는 하기의 수학식 1과 같이 여러 개의 TU로 분할하여 그 근사치를 구할 수 있다.In the present invention, English uses Unigram, Bigram, Trigram, and Chinese can use the first syllable, the last syllable of the Pinyin, or the entire Pinyin as a TU. Then, based on the definition of TU, P (

Clinton can be approximated by dividing into multiple TUs as shown in Equation 1 below.

FIG. 2 is a diagram illustrating a tuning index notation alignment model applied to an embodiment of the present invention and shows how English words and Chinese Pinyin are aligned.

Referring to FIG. 2, the present invention further adds information of a match type (M) to Equation 1 above. The match type (M) may be defined by the size of the English TU and the size of the Chinese TU.

For example, in the above Equation 1, the match type of P (ke | C) is "2-1" because the size of the English TU "C" is 1 and the size of the Chinese TU "ke" is 2. Since the match type (M) complements a new parameter that has not been learned in the parameter estimation process, it has better performance when applying a "statistic-based phonetic notation model" without a pronunciation dictionary as in the conventional technique described above. You can expect

In an embodiment of the present invention, a method of automatically estimating a parameter is applied by applying a conventional statistical-tone notation model without a pronunciation dictionary, and adds match type (M) information to the statistical-based tone model.

That is, if the match type (M) information is added to Equation 1, Equation 2 and Equation 3 may be expressed.

Here, u and v mean English TU and Chinese TU, respectively, and m means match type of u and v, respectively.

"(KeLinDun)을 찾아주는 일 예이다.3 is a diagram illustrating a process of extracting a tuning fork notation band pair from a sentence by using a statistical-based tuning fork notation model applied to an embodiment of the present invention, the correct tuning for a Chinese sentence corresponding to the English word "Clinton" Notation "

This is an example of finding "(KeLinDun).

Referring to FIG. 3, when extracting a phonetic notation band pair by applying a conventional statistical-based phonetic notation model, an error frequently occurs when there are a plurality of Chinese character strings similar to English words given in a sentence. In an embodiment of the present invention, a dynamic window technique and a tokenizer technique are used to solve such an error.

4 is a view for explaining the theoretical basis of the dynamic window technique applied to an embodiment of the present invention, Figure 5 illustrates the process of extracting the correct note notation using the dynamic window technique applied to an embodiment of the present invention It is a figure for following.

4 and 5, the dynamic window technique applied to an embodiment of the present invention does not find a path optimized for a Chinese sentence at once, but based on a given English word, a window length range of a possible Chinese tuning fork word size. It is a technique to find the tuning fork band pair by moving the window within the window length range.

"(KeLinDun), 정확한 음차표기에 한 글자가 더 들어간 "

"(KeLinYiDun), 정확한 음차표기에 한 글자가 빠진 "

"(LinDun)과 각각 정렬해 본 결과 정확한 음차표기와 정렬했을 때 점수가 가장 높았다.If you know the actual length of the Chinese phonetic notation and set it to the window size to find the phonetic notation, you can get very high performance. For example, as shown in FIG. 4, the English word "Clinton" and the correct pitch notation ""

"(KeLinDun), with one more letter in the correct note"

"(KeLinYiDun), one letter missing from the correct note"

Sorting with "(LinDun), the score was highest when aligned with the correct note.

This is because the more accurate the phonetic notation, the better the alignment between the English TU and the Chinese TU. These characteristics are common to not only Chinese but also other languages.

However, since it is difficult to predict the exact size of the Chinese phonetic notation, an embodiment of the present invention predicts a possible range of the phonetic notation word size by analyzing the distribution between the English word length and the Chinese word length in the learning data.

That is, the process of applying the dynamic window is as follows. First, after predicting a window length range based on a given English word, using the above Equation 3 for a Chinese string included in a given English word and the current window while moving forward the windows within the predicted window length range, respectively Obtain the negative likelihood probability value.

In this case, when the English word length is L, the window length range is preferably set between L / 3 to L.

In this way, the Chinese character string with the highest probability value is found while gradually increasing the window size, and the trace pairs are extracted by backtracking the string. And, as shown in Figure 5, when comparing the score between each window as the window size increases as the overall score is lowered, the normalized to the window size and compared. By applying the dynamic window technique, it is possible to effectively solve most of the errors that occur when the statistical-based phonetic notation model is applied.

FIG. 6 is a diagram for explaining a process of extracting an accurate tuning index using a tokenizer technique applied to an embodiment of the present invention.

Referring to FIG. 6, in the tokenizer technique using word separation applied to an embodiment of the present invention, a Chinese sentence is first divided into several parts based on characters not used at all in the Chinese phonetic notation, and statistics-based tuning for each part is performed. It is a technique to extract the tuning fork band pair by applying the notation model.

Specifically, in Chinese, for example, the phonetic notation is often used for phonetic notations such as "施 (shi), 德 (de), 勒 (le), 赫 (he), ..." while the pronunciation is similar. For example, there is a character set that is not used at all in the phonetic notation, such as "是 (shi)," (de), 了 (le), 和 (he), ... ".

"(DaWei)로 음차표기한다.These letters are usually searched around proper nouns, so they often make errors by combining letters such as these with the correct phonetic notation. For example, the notation of "David" can be omitted by omitting the last "d"

"(DaWei)"

"(DaWeiDe)로 잘못 인식될 수 있다. 전술한 종래의 기술에서는 규칙에 기반한 후처리 과정을 거쳐 추출한 음차표기의 양끝에 자주 사용하지 않는 문자가 있으면 제거해주는 방식으로 어느 정도 이런 문제를 해결하였다.Here, if such a noun is followed by a search such as "的" (De), which produces a pronunciation similar to the abbreviated letter "d", "

It can be misrecognized as "(DaWeiDe). The above-mentioned conventional technology solves this problem to some extent by removing an infrequently used character at both ends of the tuning fork notation extracted through rule-based post-processing.

However, this post-processing technique still cannot solve the error shown in FIG. 6. This is because the search "是" (Shi) is combined with other letters, and "者 是" (ZheShi), which is phonetically similar to the English word "Jacey", was recognized instead of the correct phonetic notation "杰西" (JieXi).

At this time, even if "是" is removed through the post-processing technique, the remaining portion "者" is not an accurate note notation. In the tokenizer technique of the present invention, characters such as "是" (Shi) belong to a character set which is not used at all in the phonetic notation, and thus these characters are removed in advance.

Then, the phonetic similarity of the remaining characters "者" (Zhe) and "Jacey" is much lower than the phonetic similarity of "杰西" (JieXi) and "Jacey", so that an accurate tone-matching band pair can be extracted. In addition, the application of the Tokenizer technique breaks down the entire sentence into parts, which greatly reduces the time complexity.

On the other hand, the letters that are not used at all in the phonetic notation means Korean, for example, punctuation, numbers, spacing, English alphabet, etc., for example Chinese, means an investigation.

As described above, the dynamic window technique and the tokenizer technique solve different problems, and therefore, the combination of the two methods can not only provide higher performance but also greatly reduce the time complexity.

That is, the method of applying the above-described dynamic window technique and the tokenizer technique together, the tuning fork extracted from the tuning fork notation candidate word extraction module 200 (see FIG. 1) through the tuning fork band pair extraction module 300 (see FIG. 1). After selecting candidate words to be phonated among the candidate candidates, each sentence in the other monolingual document (eg, Chinese document) is divided into several parts based on characters (eg, survey) not used in the phonetic notation. For example, the partitioned band pairs corresponding to the selected phonetic note candidate words may be automatically extracted from the other monolingual documents using the dynamic window technique for each divided portion.

Meanwhile, in one embodiment of the present invention, although Chinese and English are applied as an example, the present invention is not limited thereto and may be applied to all language pairs.

Hereinafter, through the experiments, the efficiency of the dynamic window technique and the torqueizer technique applied to the embodiment of the present invention was verified.

First of all, for the experiment, 300 sentences including various pairs of tuning fork notation such as place names, names of people, and product names were selected from the English-Chinese parallel corpus. For the training data, we used 860 English-to-sonic word pairs.

As shown in Table 1 below, as a result of applying the dynamic window and the torqueizer technique, the performance could be improved by about 12% compared to the conventional method, which is about 99% higher performance.

Way Word accuracy Character accuracy Character recall Statistically-based phonetic notation model (STM) 75.33% 86.65% 91.11% Statistic-based tuning fork model (STM) + dynamic window (DW) + tokenizer (TOK) 99.00% 99.78% 99.72% Statistically-based phonetic notation model (STM) + traditional method 87.99% 90.17% 91.11%

As described above, by using the method and apparatus for automatically extracting the tuning fork notation band pair in a bilingual document according to an embodiment of the present invention, a large number of non-registered dictionary dictionaries DB can be automatically constructed, which is a cross-language search and a machine. It will play a big role in improving the performance of the language translation system.

For example, Google currently offers a cross-language search service between Korean and English. When you search Google for "Google" in Korean, the word "Google" is translated into the English word "nine writing."

In this case, the user's original search result will be a totally different search result. If the non-registered dictionary dictionary DB constructed by the method and apparatus of the present invention is used, "Google" can be accurately translated to "Google", thereby obtaining accurate search results.

Another example is using Google's machine language translation service to translate a Korean sentence "I like Google" into English, resulting in a meaningless translation of "I like nine writings."

If the present invention is applied here, the above sentence can be accurately translated to "I like Google". In the past, such a non-registered word dictionary DB was built by investing a large amount of manpower and time manually, but the present invention can build a large amount of non-registered word dictionary DB having a practical value by fully automating such a manual method. This will play a huge role in areas such as cross-language search and machine language translation.

On the other hand, the method for automatically extracting the tuning fork band pairs in a bilingual document according to an embodiment of the present invention can also be implemented as computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored.

For example, the computer-readable recording medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a hard disk, a floppy disk, a removable storage device, a nonvolatile memory (Flash memory). Optical data storage, and the like, and also implemented in the form of a carrier wave (eg, transmission over the Internet).

The computer readable recording medium can also be distributed over computer systems connected over a computer network so that the computer readable code is stored and executed in a distributed fashion.

Although a preferred embodiment of the method and apparatus for automatically extracting a tuning fork band pair in a bilingual document according to the present invention has been described above, the present invention is not limited thereto, and the claims and the detailed description of the invention and the accompanying drawings. It is possible to carry out various modifications within the scope of this also belongs to the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is an overall block diagram illustrating a device for automatically extracting a tuning fork band pair in a bilingual document according to an embodiment of the present invention.

2 is a view for explaining a tuning fork notation alignment model applied to an embodiment of the present invention.

3 is a diagram illustrating a process of extracting a tuning fork notation pair from a sentence using a statistical-based tuning fork notation model applied to an embodiment of the present invention.

4 is a view for explaining the theoretical basis of the dynamic window technique applied to an embodiment of the present invention.

FIG. 5 is a diagram for explaining a process of extracting an accurate phonetic notation using a dynamic window technique applied to an embodiment of the present invention. FIG.

FIG. 6 is a view for explaining a process of extracting an accurate tuning fork using a tokenizer technique applied to an embodiment of the present invention. FIG.

Claims (34)

delete delete delete delete delete Extracting bilingual documents represented in bilingual languages existing on the Internet and separating them into monolingual documents represented in respective monolingual languages; Extracting the phonetic transcription candidate words from one monolingual document of the separated monolingual documents; selecting a candidate word to be expressed from the extracted phonetic transcription candidate words, and then each sentence in the other monolingual documents Dividing the data into several parts based on characters that are not used in the tuning fork; And Automatically extracting a tuning fork band pair corresponding to the selected tuning fork candidate word from the other monolingual document using a dynamic window technique for each divided portion, The dynamic window technique, Setting a range of window lengths that can be expressed in another monolingual based on the length of the selected tuning note candidate words; Obtaining maximum phonetic likelihood probability values for the character string including the selected tuning note candidate word and the current window while moving forward through each window within the set window length range; And And automatically extracting a tuning fork notation pair by tracing a string corresponding to the largest value among the maximum negative likelihood probability values in each window. In the dual-language document, if the selected phonetic note candidate word is an English word, the other single language is Chinese, and the length of the English word is L, the window length range is set between L / 3 and L. Automatic Extraction of Tunnel Representation Band Pairs. The method of claim 6, The phonetic similarity probability pair extraction method of a bilingual document, characterized in that obtained by the following equation (4).

Here, C is a string containing the current window, E is a selected tuning note candidate word, u, v are a tuning note unit (TU) of the selected tuning note candidate word and the string including the current window, respectively. m is the match type of u and v. The method of claim 6, Characters not used in the phonetic notation is a method for extracting the tuning fork notation band pairs in a bilingual document, characterized in that consisting of at least one of punctuation, numbers, spacing or English alphabet. The method of claim 6, And if the other monolingual document is Chinese, characters not used in the phonetic notation are made up of surveys. The method of claim 6, The monolingual document extracting the phonetic transcription candidate words is an English document, the other monolingual document is a Chinese document, characterized in that the tuning fork band band pairs in a bilingual document. The method of claim 6, And the phonetic notation candidate words are proper nouns. The method of claim 6, The bilingual document automatically extracts a tuning fork band pair in a bilingual document, characterized in that it comprises a parallel corpus or comparable corpus in all language pairs, including language pairs made of Chinese and English. The method of claim 6, And extracting the tuning fork notation band pairs in the bilingual document further comprising registering the pre-registered non-registered word dictionary DB. The method of claim 13, And performing a cross-language search service using the tuning fork notation band pairs registered in the non-registered dictionary DB. The method of claim 13, And performing a machine language translation service using the tuning fork notation band pair registered in the non-registered dictionary dictionary DB. A computer-readable recording medium having recorded thereon a program capable of executing the method of any one of claims 6 to 15. delete delete delete A bilingual document extraction module for extracting bilingual documents expressed in bilingual languages existing on the Internet and separating them into monolingual documents expressed in respective monolingual languages; A phoneme notation candidate word extraction module for extracting a phonetic notation candidate words from any single language document among monolingual documents separated from the bilingual document extraction module; And After selecting candidate words to be written in the tuning fork notation candidate words extracted from the phonetic notation candidate word extracting module, each sentence in the other monolingual document is divided into several parts based on characters not used in the phonetic notation, And a tuning fork notation band pair extraction module for automatically extracting a tuning fork notation band pair corresponding to the selected tuning note notation word from the other monolingual documents using the dynamic window technique for each divided portion, The dynamic window technique, Setting a range of window lengths that can be represented by another single language based on the length of the selected note title candidate words and moving forward each window within the set window length range, wherein the selected note title candidate words and the current window are included; After obtaining the maximum negative likelihood probability values for the character strings, the phonetic notation band pairs are automatically extracted by tracing the string corresponding to the largest value among the maximum negative similarity probability values in each window. In the dual-language document, if the selected phonetic note candidate word is an English word, the other single language is Chinese, and the length of the English word is L, the window length range is set between L / 3 and L. Automatic extraction of the tuning fork band pair. The method of claim 20, The phonetic similarity band pair automatic extraction apparatus of a bilingual document, characterized in that the negative likelihood probability value is obtained by the following Equation 5.

Here, C is a string containing the current window, E is a selected tuning note candidate word, u, v are a tuning note unit (TU) of the selected tuning note candidate word and the string including the current window, respectively. m is the match type of u and v. delete delete delete The method of claim 20, The monolingual document extracting the phonetic note candidate words is an English document, and the other monolingual document is a Chinese language document for extracting the tuning fork band pairs in a bilingual document. The method of claim 20, The tuning fork notation band pair automatic extracting device of a bilingual document, wherein the tuning fork candidate words are proper nouns. The method of claim 20, The bilingual document automatically extracts the tuning fork band pairs in the bilingual document, characterized in that it comprises a parallel corpus or comparable corpus in all language pairs, including language pairs made of Chinese and English. The method of claim 20, The apparatus for automatically extracting a tuning fork band pair in a bilingual document, further comprising a non-registered word dictionary DB for storing the tuning fork band band extracted from the tuning fork band band extraction module. The method of claim 28, And the non-registered word dictionary DB is used for a search service for a cross-language input through the Internet. The method of claim 28, The non-registered word dictionary DB is a device for automatically extracting a tuning fork notation pair in a bilingual document, characterized in that it is used to perform a machine language translation service. Extracting bilingual documents represented in bilingual languages existing on the Internet and separating them into monolingual documents represented in respective monolingual languages; Extracting the phonetic transcription candidate words from any single language document of the separated monolingual documents; And After selecting a candidate word to be expressed among the extracted phoneme notation candidate words, a phoneme notation corresponding to the selected phoneme notation candidate word in the other monolingual document using a dynamic window technique based on the selected phoneme notation candidate word. Automatically extracting the band pairs, The dynamic window technique, Setting a range of window lengths that can be expressed in another monolingual based on the length of the selected tuning note candidate words; Obtaining maximum phonetic likelihood probability values for the character string including the selected tuning note candidate word and the current window while moving forward through each window within the set window length range; And And automatically extracting a tuning fork notation pair by tracing a string corresponding to the largest value among the maximum negative likelihood probability values in each window. In the dual-language document, if the selected phonetic note candidate word is an English word, the other single language is Chinese, and the length of the English word is L, the window length range is set between L / 3 and L. Automatic Extraction of Tunnel Representation Band Pairs. The method of claim 31, wherein The negative likelihood probability value is calculated by Equation 6 below.

Here, C is a string containing the current window, E is a selected tuning note candidate word, u, v are a tuning note unit (TU) of the selected tuning note candidate word and the string including the current window, respectively. m is the match type of u and v. A bilingual document extraction module for extracting bilingual documents expressed in bilingual languages existing on the Internet and separating them into monolingual documents expressed in respective monolingual languages; A phoneme notation candidate word extraction module for extracting a phonetic notation candidate words from any single language document among monolingual documents separated from the bilingual document extraction module; And After selecting a candidate word to be expressed in the tuning fork candidate words extracted from the tuning fork candidate word extraction module, the selected tuning note candidate words in the other monolingual documents using a dynamic window technique or a tokenizer technique. Including a tuning fork notation band pair extraction module for automatically extracting the tuning fork notation band pair corresponding to, The dynamic window technique, Setting a range of window lengths that can be represented by another single language based on the length of the selected note title candidate words and moving forward each window within the set window length range, wherein the selected note title candidate words and the current window are included; After obtaining the maximum negative likelihood probability values for the character strings, the phonetic notation band pairs are automatically extracted by tracing the string corresponding to the largest value among the maximum negative similarity probability values in each window. In the dual-language document, if the selected phonetic note candidate word is an English word, the other single language is Chinese, and the length of the English word is L, the window length range is set between L / 3 and L. Automatic extraction of the tuning fork band pair. The method of claim 33, wherein And the phonetic similarity probability value is obtained by the following Equation (7).

Here, C is a string containing the current window, E is a selected tuning note candidate word, u, v are a tuning note unit (TU) of the selected tuning note candidate word and the string including the current window, respectively. m is the match type of u and v.

Images