KR20040051351A - Method for machine translation using word-level statistical information and apparatus thereof - Google Patents

Abstract

PURPOSE: A method and a device for generating a translated sentence using a statistical method of a word level are provided to generate the translated sentence of a high quality in a high speed by forming/using an order information database extracted from a large target language corpus through the statistical method. CONSTITUTION: A training module(110) statistically stores the order information from the target language corpus. A morpheme analyzer(121) analyzes a morpheme by receiving a original language sentence. A parameterizer(123) parameterizes the words corresponding to the first speech part forming the original language sentence divided into each morpheme and forms the sentence tagged by the morpheme after hiding the words corresponding to the second speech part. A word arranger(125) replaces each morpheme with the target language word from a translation dictionary database(130) by receiving the tagged sentence. A recovery part(127) recovers/inserts the original word of the parameterized speech part into the replaced target language and recovers the hidden words. A post-processor(129) outputs the original language sentence and the translated sentence based on the generation information after removing a tag.

Description

Method for machine translation using word-level statistical information and apparatus

The present invention relates to a method for generating a translation using a statistical method of a word level and a device thereof, and more particularly, to a translation generating method capable of generating a high quality translation by statistical information extracted from a training procedure using a corpus of a large target language, and To the device.

Until now, the method of generating a translation from a source language to a target language has largely used a generation grammar and an intermediate language. The electronic grammar generation method uses the programming language to implement grammar, the core knowledge of the translation generator, because it is implemented by people with linguistic knowledge to program many linguistic information. The disadvantage is that the syntax is difficult to extend. In addition, since various linguistic phenomena are difficult to display using simple grammar, there are limitations in handling numerous exceptions. In particular, the way of translating by the generated grammar has a lot of difficulties in the dialogue, because of the characteristics of the dialogue which is more grammatically free than the written language.

Another method used for generating translations, using intermediate languages, is applicable in limited domains, but it is limited to use universally. In order to express various expression forms represented by human languages using intermediate languages, This is a demand and the burden of constantly adding new forms of intermediate languages. In addition, it is difficult to express the intermediate language in a complicated situation, and in order to convert the intermediate language back to the target language, there is a difficulty in redefining the complicated conversion rules. That is, there is a problem that the burden for gradual improvement of the translation apparatus is greatly enhanced.

SUMMARY OF THE INVENTION The present invention has been made in an effort to provide a translation generation method and apparatus for generating a high quality translation using a statistical method.

1 is a block diagram of an apparatus for generating a translation using a statistical method of word level according to the present invention.

2 is a flowchart illustrating a flow of a method for generating a translation using a statistical method of word level according to the present invention.

3 is a diagram illustrating an embodiment of a band dictionary applied to the present invention.

In order to achieve the above technical problem, a method of generating a translation using a word level statistical method according to the present invention includes generating and storing order information statistically from a target language corpus; Providing a band dictionary database; Receiving a source language sentence and analyzing the morpheme; Variableizing words corresponding to predetermined first parts of speech constituting the primitive language sentences classified by the morphemes, and concealing words corresponding to predetermined second parts of speech to form sentences tagged with morphemes; Receiving the tagged sentence and replacing the target language word with the corresponding morpheme from the band dictionary database, and arranging the word of the sentence replaced with the target language based on the order information; Recovering the original word of the variable part of speech and inserting it into the replaced target language and restoring the hidden words; And removing the tagging and outputting a translation based on the source information and the generated information according to the characteristics of the target language.

In order to achieve the above technical problem, the apparatus for generating a translation using a word level statistical method according to the present invention includes a band dictionary database; A training module in which order information is statistically stored from a target language corpus; A morpheme analysis unit for receiving a source language sentence and analyzing a morpheme; A variable unit configured to variable words corresponding to predetermined first parts of speech constituting the primitive language sentences classified by the morphemes, and to conceal words corresponding to the predetermined second parts of speech and to form sentences tagged with morphemes; A word array unit configured to receive the tagged sentences and replace the target language words with the corresponding morphemes from the band dictionary database, and arrange the words of the sentences replaced with the target language based on the order information; A recovery unit for recovering the original word of the variable part-of-speech, inserting it into the replaced target language, and restoring the hidden words; And a post-processing unit for removing the tagging and outputting a translation based on the source information and the generated information according to the characteristics of the target language.

In order to achieve the above technical problem, a computer-readable recording medium recording a program for executing a translation method using a word-level statistical method according to the present invention on a computer generates statistical information from a target language corpus and stores it. Making; Providing a band dictionary database; Receiving a source language sentence and analyzing the morpheme; Variableizing words corresponding to predetermined first parts of speech constituting the primitive language sentences classified by the morphemes, and concealing words corresponding to predetermined second parts of speech to form sentences tagged with morphemes; Receiving the tagged sentence and replacing the target language word with the corresponding morpheme from the band dictionary database, and arranging the word of the sentence replaced with the target language based on the order information; Recovering the original word of the variable part of speech and inserting it into the replaced target language and restoring the hidden words; And removing the tagging and outputting a translation based on the source information and the generated information according to the characteristics of the target language.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. 1 is a block diagram of a translation generating apparatus using a word level statistical method according to the present invention, Figure 2 is a flow chart showing the flow of a translation generating method using a statistical method of the word level according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS One embodiment of an apparatus and a method is described together in order to make the description clear and the configuration of the present invention. First, the configuration of a preferred embodiment of the present invention will be mentioned in general. The present invention largely consists of two things. First, the training module 110, which is to create a sequence information database, the morpheme extracting unit 111 is subjected to morphological analysis (step 202) by inputting the target language corpus (step 201). Subsequently, the variable unit 113 converts a word corresponding to a certain part-of-speech (eg, noun and rhetoric) and then removes, conceals, and converts each word by morpheme (step 203). The sequence information extracting unit 115 extracts word sequence information using a trigram (step 204), and stores the sequence information generated as a result of the step 204 into a database 205.

Second, look at the translation module 120. The morpheme analysis unit 121 receives the source language sentence as an input (step 206) and analyzes the morpheme as in the training module 110 (step 207). In this case, stemmers corresponding to native languages are used. The variableizer 123 analyzes the morpheme and attaches the morpheme tag to the original language sentence. The word corresponding to a certain part-of-speech is parametrized in the primitive language sentence, and the word is removed, concealed, and parametrized by the corresponding morpheme, and then output (step 208). The word arranging unit 125 receives the data from the band dictionary database 130 and replaces the word with the target language (step 209). The word arranging unit enters the word arranging having the highest probability with reference to the database 205 storing the sequence information extracted and stored by the training module 110 (step 210). In this case, the word and word array having the highest probability are selected in consideration of the possibility of word replacement for each permutation. If the corresponding trigram information does not exist in the order information database 117 or 205, a probability value is obtained using a bigram and a monogram using a backoff method. After the word array having the highest probability is selected, the recovery unit 127 replaces the variable with the original word and saves the hidden word (step 211). The post-processing unit 129 transforms the sentence generated as a result of the recovery unit 127 to match the sentence order or the ending of the sentence to the form of the sentence, such as a plain sentence, a question. In the case of Korean-English translations, the appropriate subject is inserted for the sentence where the subject is omitted. In addition, the first letter of the first word capitalizes, punctuation marks (step 212). As a result, a translation of the target language is generated (step 213).

Now, preferred embodiments of the present invention will be described in detail with examples. Since the function of the translation generation apparatus corresponds to the translation generation method, the following description will mainly be given of the translation generation method. In the training module 110, sentences of the target language corpus are input in units of sentences 201 and go through each step to be described below. For example, the target language is English and the native language is Korean. That is, the stemming tagging operation is performed by the English morpheme analyzer in the stemming step 202 of the English sentence in the English corpus. As a result, the following stemming tagging result can be obtained.

Original: I'd like a hot coffee with milk.

Tagging Result: i / PRON 'd / AUX like / VERB a / DET hot / ADJ coffee / NOUN with / PREP milk / NOUN

Next, the tagging result is subjected to a variable process of the word 203, and the conjunctions, modifiers, qualifiers, adverbs, etc. are concealed, and the nouns and rhetoric are variable. Punctuation marks are deleted. The resulting sentence looks like this:

i / PRON like / VERB hot / ADJ NOUN with / PREP NOUN

With this result, it proceeds to the next step, word order information extraction (204). To enter sentence boundary information, add <s> and </ s> at the end of the resulting sentence to mark the beginning and end of the sentence. The resulting sentence looks like this:

<s> i / PRON like / VERB hot / ADJ NOUN with / PREP NOUN </ s>

This sentence is input and the order statistics are extracted. Record all trigrams in this sentence, as well as the bigrams and monograms. All the sentences in the English corpus go through these steps to calculate the statistics of all trigrams, bigrams, and monograms that can come from the training corpus data and store them in one table. That is, the frequency of occurrence is calculated for each trigram and monogram, and the probability value is calculated and recorded. The trigram probability calculation is $ prob = $ count_tri / $ count_bi. That is, the probability of one trigram is the frequency of occurrence of the trigram divided by the frequency of occurrence of the bigram. The bigram probability calculation is $ binom_w = $ v_count + $ binom {$ item_bi}; Use $ prob = ($ count_bi + 1) / $ binom_w; Where $ v_count is the total vocabulary and $ binom {$ item_bi} is the cumulative frequency at which the particular item $ item_bi appears. $ count_bi is the frequency of occurrence of a particular biggram. The frequency of the monogram is calculated as 1 / $ v_count. The probability value of this monogram is used instead of 0 for smoothing when there is no order information in the translation module. The result of the calculation is stored in the sequence information database 205. Other information recorded in the sequence information database 205 is a list of non-appearing bigrams for pruning. In order to extract this list, each sentence of the training data is sorted into every possible arrangement, and all kinds of bigrams can be extracted. This is a list of candidate bigrams for pruning.

Now, the Korean sentence is input 206 in the translation module 120 which actually translates the native language sentence and enters the morphological analysis step 207. The original Korean text is tagged with the Korean stemmer.

Input sentence: I will leave for Seoul tomorrow.

Tagging result: low / npp + / jxt tomorrow / mag seoul / nq + / jca departure / ncpa + ha / xsv + d / etm one / nbn + / jp + up / ef

In the tagging result statement, the morpheme is concealed, removed, and parametrized through the variable process (208).

Low / npp nq / jca to / ncpa + ha / xsv

This sentence is replaced with an English word through a word replacement 209 process for the target language based on the band dictionary 130 as shown in FIG. 3, and the morpheme is also replaced with the corresponding English morpheme. A comparison table of the stems of the source language and the stems of the target language is maintained on the program as a table. The resulting sentence looks like this: Here, since there are two band words in the band word dictionary of the verb start, two candidate sentences are generated.

I / PRON NOUN for / PRON start / VERB

I / PRON NOUN for / PRON depart / VERB

For all the cases in which the words for the above two cases are arranged, the highest probability value is obtained by referring to the order information database 205. The probability value is determined by multiplying the probability value of each trigram appearing in the arranged sentences in all cases. This process is the word arrangement 210 step. The pruning list used here is a list of non-occurring bigrams produced during training. By allocating words in all cases and precluding those that are pruning lists in advance, performance can be improved by reducing the scope of the calculation. As a result of calculating the probability value as the result of this step, the following middle sentence is output as an array having the highest score.

I / PRON depart / VERB for / PRON NOUN

The next step is to recover the far end word of the variable (211). The variable NOUN of this sentence is replaced with the far end word Seoul and inserted into the hidden adverb tomorrow after the sentence.

I / PRON depart / VERB for / PRON Seoul tomorrow.

In the sentence post-processing step 212, which is the last step of the translation module, tagging is deleted, a sign is added, and an auxiliary verb will indicating the future is added by a sentence rule corresponding to each sentence. As a result, the following translation is generated:

I will depart for Seoul tomorrow.

In the sentence post-processing step, in addition to generating information such as subject addition, number, person, and tense, translated sentences of the form closest to the target language are finally output (213).

The translation generation method using the word-level statistical method according to the present invention can also be embodied as computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, hard disk, floppy disk, flash memory, optical data storage device, and also carrier wave (e.g. transmission over the Internet). It is also included to be implemented in the form of. The computer readable recording medium can also be distributed over computer systems connected over a computer network so that the computer readable code is stored and executed in a distributed fashion. Also, the font ROM data structure according to the present invention can be read by a computer on a recording medium such as a computer readable ROM, RAM, CD-ROM, magnetic tape, hard disk, floppy disk, flash memory, optical data storage device, and the like. It can be implemented as code.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible in the art without departing from the technical spirit of the present invention. It will be apparent to those of ordinary skill in the art.

As described above, according to the method and apparatus for generating a translation sentence according to the present invention, in the process of translating from a source language to a target language in a conversational area, by using a large-scale target language corpus, a sequence information database extracted in a statistical manner can be used and constructed. And it is possible to generate a high-speed translation, it can be variously applied as an efficient translation generation device of the existing machine translation device.

Claims (11)

(a) statistically generating and storing order information from the target language corpus; (b) having a band dictionary database; (c) receiving a source language sentence and analyzing the morpheme; (d) variable-word the words corresponding to the predetermined first parts of speech constituting the source language sentence classified by the morphemes, and the words corresponding to the predetermined second parts of speech are concealed, and then composed of the sentence tagged with the morpheme step; (e) receiving the tagged sentence and replacing the target language word with the corresponding morpheme from the band dictionary database and arranging the word of the sentence replaced with the target language based on the order information; (f) recovering the original word of the variable part-of-speech, inserting it into the replaced target language, and restoring the hidden words; And and (g) outputting a translation based on the generation information according to the characteristics of the source language and the target language after removing the tagging. The method of claim 1, wherein step (a) (a1) analyzing a morpheme by receiving a sentence from the target language corpus; (a2) after tagging by the morphemes, converting words corresponding to predetermined first parts of speech and hiding words corresponding to predetermined second parts of speech and forming a sentence tagged with morphemes; and (a3) extracting and storing order information of words constituting the input sentence that is variable and morphologically tagged, and storing the translated sentence. The method of claim 2, wherein step (a3) And storing a probability value based on the frequency of the trigram, the bigram, and the monogram between the words. The method of claim 1, wherein step (e) (e1) If the target language word is one or more, the translation sentence generation method comprising arranging for each sentence in which the word is inserted and selecting one of them based on the stored probability value in step (a). The method of claim 4, wherein step (e1) Reflecting the meaning of the word replaced for the sentence arranged for each word, extract the trigram, bigram, and monogram value from the stored order information to obtain the value of each array, and then select the maximum value How to create a translation. Band dictionary database; A training module in which order information is statistically stored from a target language corpus; A morpheme analysis unit for receiving a source language sentence and analyzing a morpheme; A variable unit configured to variable words corresponding to predetermined first parts of speech constituting the primitive language sentences classified by the morphemes, and to conceal words corresponding to the predetermined second parts of speech and to form sentences tagged with morphemes; A word array unit configured to receive the tagged sentences and replace the target language words with the corresponding morphemes from the band dictionary database, and arrange the words of the sentences replaced with the target language based on the order information; A recovery unit for recovering the original word of the variable part-of-speech, inserting it into the replaced target language, and restoring the hidden words; And And a post-processing unit for outputting a translation based on the generated information according to the characteristics of the source language and the target language after removing the tagging. The method of claim 6, wherein the training module A morpheme extraction unit configured to receive the target language corpus in sentence units and analyze morphemes; A variable unit for tagging the morphemes and then converting them to predetermined parts of speech; An order information extracting unit for extracting order information of words constituting the input sentence that is parameterized and tagged according to morphemes; And a storage unit storing the sequence information. The method of claim 6, wherein the storage unit And a probability value based on the frequency of the trigram, the bigram, and the monogram between the words. The method of claim 6, wherein the word array unit And when the target language word is one or more, the translation sentence generation device according to claim 1, wherein the sentence is arranged based on the probability value recorded in the storage unit of the training module. 10. The method of claim 9, wherein the word array unit And generating a bigram list that does not appear in the process of generating the order information and excluding a bigram corresponding to the list. A computer-readable recording medium having recorded thereon a program for executing the steps of claim 1 on a computer.