JP5341375B2 - Parallel translation expression processing apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a translated expression processing device which presumes correspondence between long translated expressions while presuming correspondence among a plurality of words. <P>SOLUTION: The translated expression processing device includes: a translated document set group data storage part which stores a plurality of translated document set group data which are sets of translated documents by a plurality of languages; and a translated document set group data analysis processing part which counts the frequency that a word sequence which appears in a single translated document set data concurrently appears in a plurality of languages based on the translated document set data read from the translated document set data storage part, and extracts and outputs the set of the word sequence by the plurality of languages as a translated phrase set candidate so that a total value of the co-occurrence frequencies in all the translated document set data becomes more than predetermined frequency threshold. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to natural language processing. In particular, the present invention relates to a bilingual expression processing apparatus and program for performing processing such as automatically estimating the correspondence of bilingual expressions between documents in a plurality of languages corresponding to each other.

The prior art for automatically estimating the correspondence between expressions contained in bilingual bilingual sentences is as follows.
[Word Unit Method] Non-Patent Document 1 describes a method of estimating a translation relationship in word units. In addition, the author of this document has released a computer program “GIZA ++” that estimates the translation relationship in word units.
[Method Based on Co-occurrence Probability] Non-Patent Document 2 divides a document in each language into a plurality of expression columns, and based on the co-occurrence probability of a pair of translation expression candidates, a pair of expressions having the highest probability Is described as a bilingual expression.
Franz Josef Och, Hermann Ney, “A Systematic Comparison of Various Statistical Alignment Models”, Computational Linguistics, 2003, volume 29, number 1, pp. 19-51, March 2003. Daniel Marcu, 1 other, “A Phrase-Based, Joint Probability Model for Statistical Machine Translation”, Proceedings of the ACL-02 conference on Empirical methods in natural language processing, 2002, Volume 10, p. 133-139.

The prior art described above has the following problems.
In the method described in Non-Patent Document 1, there is a problem in that a parallel translation relationship between expressions composed of a plurality of words cannot be estimated because a parallel translation relationship between one word and a plurality of words is estimated.
In addition, the technique described in Non-Patent Document 2 has a problem that a long parallel expression cannot be acquired.

  The present invention has been performed based on the above problem recognition, and is a process for estimating expressions (hereinafter referred to as alignment) between parallel translations in a bilingual document pair. It is an object of the present invention to provide a bilingual expression processing apparatus and program that can align a plurality of words so that correspondence between bilingual expressions can be taken, and enables alignment of long bilingual expressions.

[1] In order to solve the above-described problem, a bilingual expression processing apparatus according to an aspect of the present invention includes a bilingual document set group data storage unit that stores a plurality of bilingual document set data that is a set of bilingual documents in a plurality of languages, Based on the bilingual document set data read from the bilingual document set data storage unit, the co-occurrence frequency between the plurality of languages of the word series appearing in the single bilingual document set data is counted, and all the bilingual Bilingual document set group data analysis processing unit for extracting and outputting the set of word sequences in the plurality of languages as parallel phrase set candidates such that the total value of the co-occurrence frequencies in the document set data is equal to or greater than a predetermined frequency threshold It is characterized by comprising.
Here, the plurality of languages is typically a case of two languages, but includes a case of three or more languages.
According to this configuration, it is possible to calculate the co-occurrence frequency of word sequences between languages in a document and extract those whose co-occurrence frequency is equal to or higher than the frequency threshold (ζ). If the co-occurrence frequency is equal to or higher than the frequency threshold, there is a high possibility that the co-occurrence frequency is a parallel translation phrase, and it is appropriate to make this a candidate for a pair of translation phrases.

[2] Further, according to one aspect of the present invention, in the bilingual expression processing apparatus, the bilingual document set group data analysis processing unit searches a word sequence in data of a first language of the plurality of languages. By counting the appearance frequency of the word sequence in the first language in all the parallel document set data, for each word sequence in the first language such that the appearance frequency is equal to or higher than the frequency threshold, The co-occurrence frequency of the word sequence of the first language and the word sequence of the other language is counted by searching the word sequence in the data of another language of the plurality of languages, It is characterized by co-occurrence frequency between languages.
It is possible to actually count the co-occurrence frequency by a specific procedure with this configuration.

[3] Further, according to an aspect of the present invention, in the above-described bilingual expression processing apparatus, the bilingual document set group data analysis processing unit performs a search next to the current word series when sequentially searching for the word series. The new word sequence obtained by adding one word to the current word sequence is preferentially searched before the new word sequence obtained by replacing one word in the current word sequence. It is a target.
That is, the bilingual document set group data analysis processing unit in this aspect performs a depth-first search. When performing breadth-first search, it is necessary to store a large amount of information associated with the spread in the width direction in the memory. The search process can be performed with a realistic apparatus configuration.

  [4] Further, according to one aspect of the present invention, in the bilingual expression processing apparatus, the bilingual document set group data analysis processing unit is a statistic representing a probability that each of the extracted bilingual phrase set candidates is a bilingual phrase set. An amount is calculated, and the calculated statistic is output together with the bilingual phrase set candidate.

[5] Further, according to one aspect of the present invention, in the bilingual expression processing apparatus, the bilingual document set group data analysis processing unit is configured to search the first series during the search for the word series in the data of the other language. A statistic indicating the certainty that the pair of the word sequence of the other language and the word sequence of the other language is a bilingual phrase set, and the statistic calculated for the current word sequence of the other language The probability that the probability is lower than the probability represented by the statistic already calculated for the word sequence obtained by removing one word from the current word sequence of the other language, and the predetermined statistic threshold (τ) represents If it is lower than the likelihood, the search for a new word sequence obtained by adding one word to the current word sequence of the other language is suppressed.
With this configuration, it is possible to perform a search while pruning based on the statistic, thereby reducing the total calculation amount.

  [6] A computer program according to an aspect of the present invention provides a computer program including a parallel document group data storage unit that stores a plurality of parallel document group data that is a set of parallel documents in a plurality of languages. Based on the bilingual document set data read from the storage unit, the co-occurrence frequency between the plurality of languages of the word series appearing in the single bilingual document set data is counted, and all the bilingual document set data The bilingual document group data analysis process for extracting and outputting the word sequence pairs in the plurality of languages as bilingual phrase group candidates such that the total value of the co-occurrence frequencies is equal to or greater than a predetermined frequency threshold is executed. Let

According to the present invention, the expression (word sequence) in the bilingual document set group data is searched, the co-occurrence frequency (number of sentences) of the expression between languages is counted, and the bilingual phrase candidate is extracted based on the result. Therefore, expressions of a plurality of words can be associated as parallel translation phrase candidates.
Also, this method does not restrict the length of the expression to be aligned.
In the above description, the parallel document group data analysis processing unit performs a depth-first search, so that the amount of memory to be stored in the search space is small. Although a huge search space is searched, priority is given to depth, so that a necessary search can be actually performed with a realistic amount of memory.
In addition, since the search space is pruned using the statistics, the total amount of calculation is small, and processing can be performed in a short time.
In addition, in order to align inclusively, multiple length alignment results can be obtained from long to short representations.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the present embodiment, alignment is performed by the following two-stage processing. First, in the first stage, bilingual phrase candidates having a plurality of statistics equal to or greater than a threshold value are extracted from the bilingual document pair group data (parallel data). In the next second stage, the alignment is established by selecting candidates using a plurality of feature quantities from among the parallel translation phrase candidates that are consistent with other alignments. As the feature amount, the statistic amount at the word level and the phrase level and the presence / absence of registration of the parallel dictionary are used. Below, the process for each of these stages will be described.

<1. Extraction of parallel phrase candidates>
FIG. 1 is a block diagram illustrating a functional configuration of the parallel expression processing apparatus 100 according to the present embodiment. The bilingual expression processing apparatus 100 extracts bilingual phrase candidates (translated phrase group candidates) and calculates the statistic of these candidates based on bilingual document pair group data that stores a large number of bilingual document pairs. .

As shown in the figure, the parallel translation expression processing apparatus 100 includes a parallel document pair group data storage unit 1 (parallel translation document group data storage unit) and a parallel document pair group data analysis processing unit 2 (parallel translation document group data analysis processing unit). And a parallel translation phrase candidate data storage unit 3.
The bilingual document pair data storage unit 1 stores a large number of bilingual document pair (group) data which are bilingual document pairs (groups).
The bilingual document pair group data analysis processing unit 2 is based on the bilingual document pair (set) data read out from the bilingual document pair data storage unit 1 and a plurality of word sequences appearing in the single bilingual document pair (set) data. Counts the co-occurrence frequency between languages, and translates pairs of word sequences (sets) in multiple languages so that the total value of co-occurrence frequencies in all parallel document pairs (sets) data is greater than or equal to a predetermined frequency threshold Extracted as a phrase pair (group) candidate and written in the parallel phrase candidate data storage unit 3.
The parallel phrase candidate data storage unit 3 stores the extracted parallel phrase candidate data.
The bilingual document pair data analysis processing unit 2 is configured by an electronic circuit or the like. The bilingual document pair group data storage unit 1 and the bilingual phrase candidate data storage unit 3 are realized using a semiconductor memory, a hard disk device, or the like, and can read and write data via electronic means. .

  FIG. 2 is a schematic diagram showing a configuration of parallel translation document pair data stored in the parallel translation document pair data storage unit. As shown in the drawing, the parallel translation document pair data is expressed as, for example, tabular data, and data items (columns) of sentence numbers, sentences (language J, first language), and sentences (language E, other languages). ). That is, sentences in a plurality of languages are associated. Each line of this data corresponds to a bilingual document pair. The sentence number is a serial number that can uniquely identify the bilingual expression document pair in the bilingual expression document pair group data. The sentence (language J) column stores sentences in the first language (language J) in the parallel translation document pair of each line. The sentence (language E) column stores a sentence in the second language (language E) in the bilingual document pair of each line. Here, the language J and the language E are, for example, languages such as Japanese and English, but are not limited to these two types of languages, and may be arbitrary languages.

  FIG. 3 is a schematic diagram illustrating an example of a bilingual expression document pair stored in the bilingual expression document pair group data storage unit. In this example of data, the language J is Japanese and the language E is English. And the stored data of sentence number “1” is the language J sentence “Typhoon is expected to change to temperate cyclone in the southeast of Kyushu at noon tomorrow” and the language E sentence “The typhoon is "expected to downgrade over the sea southeast of Kyushu tomorrow afternoon." Here, bilingual translations in both languages do not necessarily have to be exact verbatim translations. Note that in the case of a sentence in a language that is not divided when normally used, such as Japanese, a blank character is inserted between morphemes by performing morpheme analysis processing in advance. In addition, Japanese punctuation marks “.”, Punctuation marks “,”, and English periods and commas are also treated as independent morphemes for convenience.

FIG. 4 is a schematic diagram illustrating a configuration and data example of parallel phrase candidate data stored in the parallel phrase candidate data storage unit 3. The bilingual phrase candidate data is, for example, data represented in a tabular format, and includes a phrase (language J) p ^j , a phrase (language E) p ^e, and four types of statistics items (columns). There are four types of statistics used in the present embodiment: significant probability (−log (p-value × 2)), Dice coefficient, phrase average generation probability, and phrase generation probability. Details of these statistics will be described later. As an example, the sixth line of the table data in the figure is a candidate for a pair in which the phrase “meteorological agency” in language J and the phrase “The meteorological agency” in language E are parallel translation phrases. Indicates that the significance probability is 2430.25, the Dice coefficient is 0.743797, the phrase average generation probability is 0.286288, and the phrase generation probability is 0.240436.

Next, specific processing contents of the bilingual expression processing apparatus 100 will be described.
Here, a technique for efficiently extracting phrase pairs having a plurality of statistics equal to or greater than a threshold value using a phrase pair enumeration algorithm and a pruning technique will be described. Here, the phrase is an arbitrary length expression composed of one or more continuous words. If frequent expressions are extracted for each of the two languages (here, Japanese and English), the number of expressions that are acquired increases, so the number of combinations of these two languages is enormous, and the amount of data Calculation of many combinations is difficult. However, when searching for an expression pair in which the number of co-occurring sentences is greater than or equal to the threshold, expressions with the number of appearing sentences greater than or equal to the threshold are extracted for each language, and pairs of expressions obtained here (here Then, when a sentence in which a combination of Japanese expression and English expression) co-occurs, many pairs that fall below the threshold are included. Therefore, in the method of the present embodiment, an algorithm that directly searches for phrase pairs in which the number of co-occurring sentences is equal to or greater than a threshold is used. The basic idea of the algorithm of this method is that depth-first search is performed in two stages and pruned using statistics other than frequency.
In the following, a method for formulating a depth-first search for a phrase having a large number of appearing sentences in a single language and then performing a two-step depth-first search will be described.

<1.1 Depth-first search method for phrases in a single language>
A search in a single language of a phrase having a large number of appearing sentences using a depth-first search is formulated as follows.
Let W = {w ₁ , w ₂ ,..., W _n } be a set of words. A sentence is s, a sentence number is d, and a set of sentence number and sentence pairs (d, s) is a corpus S. Let the phrase be p. A word sequence (word sequence) is defined as f = f ₁ f ₂ ... F ₁ (where f _i εW, iε {1, 2,..., L}. S and p are sequences. Let c (p) be the number of sentences in the phrase p in the corpus S.

  Searching for a phrase whose number of sentences appearing in a single language is greater than or equal to a threshold ζ (frequency threshold) is to enumerate all phrases p that satisfy c (p) ≧ ζ for an arbitrary natural number ζ. Such enumeration can be performed by an algorithm based on a depth-first search described below.

Here, before describing the algorithm, variables used in the following are defined. Let X be a partial corpus that includes a sequence f (X⊆S). The sets Y, H, and G that are used as indexes are defined as follows.
That is, in X, a value obtained by adding 1 to the last word position matching the sequence f in the sentence corresponding to the sentence number d is set to r. A set of pairs (d, r) of the sentence number d and the value r is Y. However, if r is larger than the word position at the end of the sentence, the pair (d, r) is not included in the set Y.
In all pairs (d, s) in the partial corpus X, a pair (d, r) of a sentence number d and a value r such that (d, r) εY and a word g at a word position r in the sentence Let H be the set of r, g).
A set of words g included in the set H is defined as G.

  FIG. 5 is a schematic diagram illustrating the operation of a process for acquiring a phrase starting with the word “a” when ζ = 2. In other words, this figure is an example of a phrase search with depth priority in a single language.

The processing of searching for a phrase with depth priority is as follows.
Step 1: As initial values, f ₁ = a and f = f ₁ are set.
Step 2: A set X of a pair (d, s) of a sentence number d and a sentence s of a sentence including f is generated, and a pair (s) of the sentence number d and the end position where f appears plus a value r of 1 ( A set Y of d, r) is generated.
Step 3: If | X |, which is the radix of X, is greater than or equal to ζ (here, ζ = 2), the f is output as a phrase. If | X | is less than ζ, the process for the current f ends.
Step 4: Set H and set G are generated from set X and set Y. And for each word w _i (w _i ∈G) following f, we added w _i to the end of f,

And then

Create And these created,

As new f, X, and Y, the process returns to step 3 above.
The “new f” is obtained by concatenating the current f with the word g at the word position r of the sentence with the sentence number d in the corpus X for each element (d, r) of the set Y.

The processing from step 1 to step 3 described above is processing for searching for a sub-tree connected to each branch extending from the current node in the tree when the search space to be searched is regarded as a tree structure. Is called recursively. By repeating this, all phrases starting from a and satisfying c (p) ≧ 2 can be extracted.
In this search, a depth-first search is performed. By making the depth-first search in this way, it is possible to prevent an explosive increase in the amount of information to be stored during the search process.

A specific process when the above procedure is applied to the illustrated example will be described.
Node (1): In the leftmost node in the figure, f = a is set as the initial value of the search. Then, in the corpus S, a set X corresponding to the sequence f is obtained. Then, a set having four pairs of elements (d = 1, s = abcd), (d = 2, s = dabc), (d = 3, s = acab), (d = 4, s = bdac) as elements. X is obtained. Since | X | = 4 (| X | ≧ ζ), f = a is output as a phrase. Then, the following search is performed for f = ab (appears when d = 1, 2, 3) and f = ac (appears when d = 3, 4) as new f.
Node (1-1): The first node of the upper branch in the figure corresponds to f = ab. Here, since | X | = 3 (| X | ≧ ζ), f = ab is output as a phrase. Then, the next search is performed for f = abc (appears when d = 1, 2) as a new f. Here, the sentence s corresponding to d = 3 is not included in the set Y because there is no word after ab, and therefore is not subjected to further search.
Node (1-1-1): The next node corresponds to f = abc. Here, since | X | = 2 (| X | ≧ ζ), f = abc is output as a phrase. Then, the next search is performed for f = abcd (appears when d = 1) as a new f.
Node (1-1-1-1): The next node corresponds to f = abcd. Here, since | X | = 1 (| X | <ζ), f = abcd is not output. Then, the processing of this branch is finished.
Node (1-2): Next, the process moves to node (1-2), which is another node immediately below node (1). This node corresponds to f = ac. Here, since | X | = 2 (| X | ≧ ζ), f = ac is output as a phrase. Then, the next search is performed for f = aca (appears when d = 3) as a new f.
Node (1-2-1): The next node corresponds to f = aca. Here, since | X | = 1 (| X | <ζ), f = aca is not output. Then, the processing of this branch is finished.
This completes all searches when f = a is the initial value.

FIG. 6 is a pseudo code showing a processing procedure of the phrase depth-first search in the above-described single language. This pseudo code is in a procedural language having a block structure. In this figure, the number on the left side of the code is the line number.
The code on the first line calls the procedure DepthFirstSearch described below using φ (empty series), corpus S, and set Y _init as actual arguments. Here, the set Y _init is a set of pairs of d and (positions of all words in the sentence s) −1 in (d, s) ∈S.
The code on the second line is a declaration of the procedure DepthFirstSearch and its parameters f, X, and Y.
The third to fourteenth lines are the execution part of the procedure DepthFirstSearch.
The third line is a begin statement representing the beginning of the block.
The fourth line is a process for creating a set H and a set G based on the set X and the set Y.
The fifth line is a control statement “foreach” for executing the processing from the sixth line to the thirteenth line for each word w _i included in the set G.
From the 6th line to the 8th line, about the current w _i ,

It is a process for making.
From the 9th line to the 12th line,

If the cardinal number of the is greater than or equal to the threshold ζ (if the condition of the if statement on the ninth line is true),

As a phrase ("output" on line 10)

Is a process of recursively calling the procedure DepthFirstSearch (line 11).
If the condition of the if statement on the 9th line is false, nothing is done.
The code on the 13th line is an end sentence corresponding to “foreach” on the 5th line.
The code on the 14th line is an end statement corresponding to “begin” on the 3rd line.

<1.2 Method of extracting parallel translation phrase candidates>
Next, a procedure of bilingual phrase candidate extraction processing by the bilingual document pair data analysis processing unit 2 according to the present embodiment will be described. This procedure extends the phrase search processing procedure in the single language described above with reference to FIGS.
In the following, the enumeration algorithm for phrase pairs based on the number of co-occurrence sentences will be described first, then the statistical index of parallelism will be described, and finally the pruning technique based on statistics will be described.

<1.2.1 Phrase Pair Enumeration Algorithm Based on Co-occurrence Sentence>
First, in order to distinguish each language of parallel data, the variables introduced at the time of phrase search in a single language are expanded as follows. That is, a character for identifying the language is added to the previously introduced variable. The variable for the first language, language J (for example, Japanese here) is appended with “j” on the right shoulder of the variable, and the second language, language E (for example, English here). “E” is attached to the right shoulder of the variable. That is, for example, the variable s ^j indicates a Japanese sentence, and the variable s ^e indicates an English sentence. Each of the corpora S and X is a set of sets (d, s ^j , s ^e ) of a sentence number d, a sentence s ^{j in} language J, and a sentence s ^{e in} language E. In parallel data, sentences in both languages (language J and language E) in a parallel translation relationship share sentence numbers. In other words, the sentence s ^{j of} language J and the sentence of language E correspond to a sentence number d. Since s ^e exists, it is not necessary to add information for distinguishing languages to the sentence number itself. This group (d, s ^j , s ^e ) corresponds to the bilingual expression document pair group data stored in the bilingual expression document pair group data storage unit 1 and corresponds to the data configuration described above.

FIG. 7 is a pseudo code showing a processing procedure by the bilingual document pair group data analysis processing unit 2. This code extracts a frequent phrase pair based on a depth-first search from the bilingual document pair group data. It is realized. In this figure, the number on the left side of the code is the line number.
As shown in the figure, this code includes the definition of the procedure ExpandJ, the definition of the procedure ExpandE, and the main processing part. The procedure ExpandJ is called from the main processing portion, the procedure ExpandE is called from the procedure ExpandJ, the procedure ExpandJ is recursively called, and the procedure ExpandE is recursively called from the procedure ExpandE.

The code on the first line is a main processing part, and calls the procedure ExpandJ with φ (empty series), corpus S, and set Y ^j _init as actual arguments. Here, the set Y ^j _init is the sentence number d in (d, s ^j , s ^e ) εS and (the positions of all words in the sentence s ^j of the language J corresponding to the sentence number d) − A set of pairs with a value r ^{j of} 1.
The code on the second line is a declaration of the procedure ExpandJ and its parameters f ^j , X, Y ^j .
The third to fifteenth lines are the execution part of the procedure ExpandJ.
The third line is a begin statement representing the beginning of the block.
The fourth line is a process of creating a set H ^j and a set G ^j based on the set X and the set Y ^j .
The fifth line is a control statement “foreach” for causing the processes from the sixth line to the 14th line to be executed for each word w ^j _i included in the set G ^j .
From the 6th line to the 8th line, about the current w ^j _i ,

It is a process for making.
From the 9th line to the 12th line,

If the cardinal number of the is greater than or equal to the threshold ζ (if the condition of the if statement on the ninth line is true)

(Here, the set Y ^e _init is the sentence number d in (d, s ^j , s ^e ) εS and all of the sentences s ^{e in} the language E corresponding to the sentence number d) word position) with a value of -1 ^{r e,} as well as a set of pairs), calls the procedure ExpandE (11 line call), and recursively calling the procedure ExpandJ (12 line call). When calling procedure ExpandE here, as an actual argument,

give. When calling the procedure ExpandJ, as an actual argument,

give.
If the condition of the if statement on the 9th line is false, nothing is done.
The code on the 14th line is an end sentence corresponding to “foreach” on the 5th line.
The code on the 15th line is an end statement corresponding to “begin” on the 3rd line.

The code on the 16th line is a declaration of the procedure ExpandE and its parameters f ^e , X, Y ^e , and f ^j .
The 17th to 28th lines are the execution part of the procedure ExpandE.
The 17th line is a begin statement representing the beginning of a block.
The 18th line is a process for creating a set H ^e and a set G ^e based on the set X and the set Y ^e .
Line 19 is for the word ^w _{e i} of each included in the set ^{G e,} a control statement for executing the processing from the 20th line to line 27 "foreach".
From line 20 to line 22, for the current ^w _{e i,}

It is a process for making.
From the 23rd line to the 26th line,

If radix is not less than the threshold value ζ of (if the condition is true 23 line if statement), and f ^j

As a phrase pair and the procedure ExpandE is recursively called (call on line 25). Here, when calling the procedure ExpandE, as an actual argument,

give.
If the condition of the if statement on the 23rd line is false, nothing is done.
The code on the 27th line is an end sentence corresponding to “foreach” on the 19th line.
The code on the 28th line is an end statement corresponding to “begin” on the 17th line.

  In other words, in the processing procedure described above using the pseudo code, the phrase of the other language (language E) is depth-first searched in the procedure ExpandJ that searches for the phrase in depth priority in the single language (language J). The depth-first search is performed in two stages, such as calling the procedure ExpandE (the 11th line code). In addition, the feature of this processing procedure is that, when the procedure ExpandE is called in the procedure ExpandJ (the 11th line code), the second actual argument is not the corpus S corresponding to the entire bilingual document pair group data,

It is a point that passes. As a result, on the called procedure ExpandE side, only the sentence in which f ^j appears is searched. As a result, in the code on the 23rd line in the procedure ExpandE

Corresponds to f ^j passed as a parameter and the node to be searched by the procedure ExpandE

The number of sentences in which the phrase pair co-occurs, and the phrase pairs having the co-occurrence sentence number equal to or greater than the threshold ζ can be directly listed.

  In other words, by executing the procedures ExpandJ and ExpandE described above, by searching the word sequence in the data of the first language (language J), all the parallel document pair data (corpus S) For each word sequence in the first language in which the appearance frequency of the count result is equal to or higher than the frequency threshold value ζ, the procedure ExpandE is called and another language (language E) By searching for word sequences in the data, the co-occurrence frequency of the word sequence of the first language and the word sequence of another language is counted, and whether or not the co-occurrence frequency is equal to or higher than the frequency threshold ζ. In accordance with the above, bilingual phrase pair candidates are extracted.

  Further, in the processes of the procedures ExpandJ and ExpandE described above, a depth-first search is performed for each language. In other words, when the search for a word sequence is performed sequentially, In addition, a new word obtained by adding one word to the current word sequence rather than a new word sequence obtained by replacing one word in the current word sequence (this new word sequence spreads in the width direction). This means that the sequence (this new word sequence spreads in the depth direction) is preferentially searched first.

<1.2.2 Statistical indicators of parallelism>
The bilingual document pair group data analysis processing unit 2 performs a process of calculating a statistical index value of the phrase pair at the same time as performing the process of extracting the phrase pair in the search process described above. Hereinafter, statistical indicators will be described.

  The enumeration algorithm described above enumerates phrase pairs based on the number of co-occurrence sentences, but there are many phrase pairs enumerated by this that are unlikely to be actually translated. End up. Therefore, by using the four types of statistics described below as an index indicating the likelihood of parallel translation, it is possible to select those that are actually highly likely to be parallel translations as parallel phrase candidates.

Therefore, the bilingual document pair data analysis processing unit 2 calculates these four types of statistics, extracts only phrase pairs for which the calculated statistics are equal to or greater than a predetermined threshold as bilingual phrase candidates, Use phrase data. The parallel document pair data analysis processing unit 2 writes the calculated statistic in the parallel phrase candidate data storage unit 3 in association with the phrase pair. In other words, the bilingual document pair data analysis processing unit 2 calculates a statistic indicating the probability of being a bilingual phrase pair for each bilingual phrase pair candidate to be extracted, and outputs the calculated statistic in association with the bilingual phrase pair candidate. (Write to the parallel translation phrase candidate data storage unit 3).
As a result, the value of the statistic associated with the phrase pair can be used later. These four types of statistics are as shown in FIG. 4 and described as part of the parallel phrase candidate data.

These four types of statistics in the present embodiment are significance probability, Dice coefficient, phrase average generation probability, and phrase generation probability.
<1.2.2.1 Significance probability>
A one-sided test of Fisher's Exact Test, which is a statistical hypothesis test, is performed on the number of co-occurrence sentences in a phrase pair, and the significance (p-value) is doubled to obtain a negative logarithm. Values are used as statistics. In other words, when expressed as an expression:
-Log (p-value x 2)
It is.

Here, FIG. 8 is a 2 × 2 (2 rows / 2 columns) contingency table having the number of data (number of sentences) a ₁ , a ₂ , a ₃ , a ₄ in the corpus as elements. The definitions of a ₁ , a ₂ , a ₃ , and a ₄ are as follows.
a ₁ is the number of data in the corpus of sequences f ^e sequence f ^j and Language E language J is neither appear. In other words, the position of the row of ^{f j} in the row of the ^{f e} in the contingency table is _{a 1.}
a ₂ is the number of data in which the series f ^j does not appear and the series f ^e appears. In other words, the position of the column of the row with ¬ (not) ^{f j} of ^{f e} in the contingency table is _{a 2.}
a ₃ is the number of data in which the sequence f ^j appears and the sequence f ^e does not appear. That is, in the contingency table, the position of the column f ^{j in} the row ƒ ^e is a ₃ .
a ₄ is the number of data in which neither the sequence f ^j nor the sequence ^fe appears. In other words, the position of the row of ¬F ^j in the row of ¬F ^e in contingency table is _{a 4.}

  At this time, p-value can be calculated by the following equation (1).

但し、

However,

In the above formula, min (a ₂ , a ₃ ) is a function whose value is the smaller of a ₂ or a ₃ .
When the parallel document pair data analysis processing unit 2 calls the procedure ExpandE on the 8th line in the procedure ExpandJ, as the second argument,

However, it is possible to search ^fe for the entire corpus S by calling the procedure ExpandE using S as the second argument, and as a result, the value of a ₁ + a ₂ is obtained. Can be obtained.

<1.2.2.2 Dice coefficient>
Using the values of the 2 × 2 contingency table shown in FIG. 8, the bilingual document pair group data analysis processing unit 2 calculates a Dice coefficient represented by the following equation (2).

<1.2.2.3 Phrase average generation probability>
The bilingual document pair data analysis processing unit 2 calculates a phrase average generation probability defined by the following equation (3).

The conditional probability P of the word in the above formula uses IBM model 1, which is a probability model that is most likely estimated by the EM algorithm. The IBM model 1 is described in the following document. Literature: Franz Josef Och, Hermann Ney, “A Systematic Comparison of Various Statistical Alignment Models”, Computational Linguistics, 2003, volume 29, number 1, pp. 19-51, March 2003.
<1.2.2.4 Phrase generation probability>
The bilingual document pair data analysis processing unit 2 calculates a phrase generation probability defined by the following equation (4).

  The conditional probability P of the word in the above formula uses IBM model 1, which is a probability model that is most likely estimated by the EM algorithm.

<1.2.3 Pruning method based on statistics>
According to the processing procedure described above, the bilingual document pair data analysis processing unit 2 can search for phrase pairs in the corpus and extract phrase pair candidates. By using the method described here, further search processing can be performed. It is also possible to reduce time.
Specifically, the bilingual document pair group data analysis processing unit 2 prunes the space to be searched using a statistic during searching in order to reduce the amount of calculation.

  In the processing procedure described above, among the phrase pair candidates of language J and language E, those having the number of co-occurrence sentences equal to or greater than the threshold ζ are listed. Here, even if the number of co-occurrence sentences is greater than or equal to the threshold ζ, the amount of calculation is reduced by excluding those whose statistic is less than the predetermined threshold from the search space. However, it is difficult to directly enumerate phrase pairs whose four types of statistics (significance probability, Dice coefficient, phrase average generation probability, and phrase generation probability) are equal to or greater than a threshold value during the search process.

  Therefore, during the search, it is predicted whether a phrase pair whose statistic is equal to or greater than the threshold value will appear in the space of the range to be searched in the future. Mow. Specifically, it is as follows.

  The bilingual document pair group data analysis processing unit 2 performs a depth-first search by recursively calling the procedure ExpandE, and the current sequence (phrase candidate) of the language E is in the middle of the search.

If the statistic of is less than the threshold (statistics threshold) and becomes smaller than the statistic of the previous ^fe (in other words, one step higher in the search tree), the search is further deeper. Even so (in other words, even if a word is added and the sequence is further extended), the statistic is predicted not to be sufficiently large, and the search is terminated at that stage without further deep search. In other words, a deeper branch (that is, the search space following the branch) is mowed.

The reason why the above prediction is effective is as follows. If p ^e bilingual phrase p ^j bilingual a is at the Language E language J is long representation, partial sequence f ^e from the beginning of the p ^e is the with the search in the depth direction it is expected that the length becomes larger statistics closer to the bilingual of p ^e. In other words, if the statistic is less than the predetermined threshold value and the statistic becomes smaller even if the sequence is extended by one word, the statistic will remain even if the sequence is further extended. It is expected that it will not be more.

  FIG. 9 is a pseudo code showing a procedure of search processing to which the pruning technique is applied. In this figure, the number on the left side of the code is the line number. By replacing the code of the procedure ExpandE described in FIG. 7 with the code of the procedure ExpandE shown in this figure and performing a search process, the above-described search while pruning can be performed. In this pseudo code, a statistic is simply expressed as one variable u, but a plurality of statistic may be used for determination. Even when determining whether or not to perform pruning using a plurality of statistics, the variable u is regarded as a vector, and a magnitude relationship between the vectors is appropriately defined and a threshold value (a vector of a plurality of statistics) is defined. Therefore, the code processing procedure shown in this figure is essentially applicable.

The code on the first line is a declaration of the procedure ExpandE and its parameters f ^e , X, Y ^e , f ^j and u. U received as a parameter by the procedure ExpandE is a statistic corresponding to the sequence ^fe .
The second to the 18th lines are the execution part of the procedure ExpandE.
The second line is a begin statement representing the beginning of the block.
The third line is a process of creating a set H ^e and a set G ^e based on the set X and the set Y ^e .
The fourth line, the word ^w _{e i} of each included in the set ^{G e,} a control statement for executing the processing from the fifth line to the 17th line "foreach".
From fifth to seventh lines, for the current ^w _{e i,}

It is a process for making.
The eighth line is

This is an “if” statement with a conditional clause that makes a condition whether or not the cardinal number of the above is greater than or equal to a threshold value ζ. When this condition is true, the processing from the 9th line to the 15th line is executed.
The ninth line is the series f ^j and the series

Is the statistic when pairing

Is a process for calculating.
The 10th line is an “if” statement having a conditional clause that makes a condition whether or not the statistic calculated in the 9th line is equal to or greater than a predetermined threshold τ (statistical threshold). If this condition is true, the sequence f ^j and the sequence

Is output as a phrase pair (line 11) and the procedure ExpandE is recursively called (line 12).
If the condition on line 10 is false, the if statement on line 13

Is true (in other words, the statistic calculated in the 9th line is larger than the statistic u received as a parameter from the previous line), the current phrase pair will not be output, but a deeper search will be performed. To do this, the procedure ExpandE is called recursively (line 14).
The actual argument when calling procedure ExpandE in the processing of the 12th or 14th line is

It is.
If the condition of the if statement on the 8th line is false, nothing is done.
The code on the 17th line is an end sentence corresponding to “foreach” on the 4th line.
The code on the 18th line is an end statement corresponding to “begin” on the 2nd line.
As described above, the feature of the processing procedure shown here is that even if the number of co-occurrence sentences of a phrase pair candidate is equal to or greater than the threshold ζ, the phrase pair is not output unless the feature amount is equal to or greater than the threshold τ. Furthermore, when the feature value is lower than that of the previous one, no further depth-first search (recursive call of procedure ExpandE) is performed on the branch (pruning). is there.

  When determining whether or not to prun, both the significant probability (the value of -log (p-value x 2)) and the Dice coefficient related to the next series are those values related to the current series. It is preferable that the branch of the next series is cut when it becomes smaller. In this way, pruning is performed only when both of these two types of statistics are worse (smaller), so that excessive pruning can be prevented, and therefore it is necessary while reducing the amount of calculation. Can be extracted sufficiently.

  As described above, in other words, the processing of the procedure ExpandE that performs pruning is performed, while the word sequence of the first language (language J) and the language of the other language are searched during the search of the word sequence in the data of another language (language E). A statistic representing the probability that the pair with the word sequence is a parallel phrase set is calculated (line 9 of the pseudo code in FIG. 9), and the probability represented by the statistic calculated with respect to the current language E word sequence is: A statistic that has already been calculated for a word sequence obtained by removing one word from the current language E word sequence (that is, the word sequence corresponding to the parent node in the tree of the search space) (in the pseudo code, procedure ExpandE Variable u) which is the fifth parameter of (the comparison in the if statement of the 13th line of the pseudo code) and lower than the probability represented by the threshold (statistic threshold τ) In the comparison of the if code on the 10th line of the similar code, the search for a new word sequence obtained by adding one word to the current language E word sequence is suppressed (that is, the pseudo code 14). Do not recursively call the procedure ExpandE described in the line).

<2. Phrase pair alignment>
Next, processing for aligning phrase pairs in a given bilingual document pair based on bilingual phrase candidate data will be described.
FIG. 10 is a block diagram showing a functional configuration of the parallel expression processing apparatus 200 for performing the alignment process. As shown in the figure, the bilingual expression processing apparatus 200 includes a bilingual phrase candidate data storage unit 3, a bilingual dictionary data storage unit 4, a bilingual document pair data input unit 5, a bilingual phrase candidate acquisition unit 6, and a bilingual phrase candidate rank. It includes an attaching processing unit 7 and an alignment processing unit 8, and outputs the result of the alignment processing.

  The bilingual phrase candidate data storage unit 3 stores bilingual phrase candidate data created by the process described above. In other words, the parallel phrase candidate data includes a plurality of parallel phrase pair (group) candidates in which word sequences in a plurality of languages are associated with each other. The configuration of the bilingual phrase candidate data is as described with reference to FIG.

The bilingual dictionary data storage unit 4 stores bilingual dictionary data that holds bilingual bilingual phrases associated with each other. This data has sufficient reliability in terms of language. However, the priority based on the frequency of appearance of phrases in a single language, the frequency of co-occurrence of phrase pairs between two languages, the length (number of words) of phrases, etc. are additionally stored in association with the phrase pairs. You may do it. The reason why this priority also depends on the length of the phrase is, for example, a phrase that is formed by combining a Japanese word corresponding directly to an English idiom with a Japanese word corresponding to each individual word constituting the idiom This is because there are cases where priority should be given to the translated phrase rather than the case.
That is, the bilingual dictionary data storage unit 4 stores bilingual dictionary data having a plurality of bilingual phrases associated with word sequences in a plurality of languages (language J and language E).

  The bilingual document pair data input unit 5 has a function of receiving input of a bilingual document to be subjected to alignment processing. More specifically, the bilingual document pair data input unit 5 is a keyboard or the like that receives input from a user, a reading unit that reads text from a hard disk device, or an OCR (optical character that is read and recognized). It is means for capturing text to be processed, such as an optical character recognition (Optical Character Recognition) device.

The bilingual phrase candidate acquisition unit 6 refers to the data read from the bilingual phrase candidate data storage unit 3 and the bilingual dictionary data storage unit 4, thereby matching the bilingual document pair captured by the bilingual document pair data input unit 5. Get all candidates. The phrase “matched” with the parallel phrase candidate is a state in which the phrase of each language of the parallel phrase candidate is included in each language in the parallel document pair. At this time, the translation phrase candidate acquisition unit 6 also holds information for distinguishing between the translation phrase candidates acquired from the translation phrase candidate data and the translation phrase candidates acquired from the translation dictionary data. In addition, the bilingual phrase candidates acquired from the bilingual phrase candidate data are also stored in association with their statistical values. Further, when the bilingual dictionary data has the above-mentioned priority data, the bilingual phrase candidates acquired from the bilingual dictionary data are also stored in association with the priority values.
In other words, the bilingual phrase candidate acquisition unit 6 reads the bilingual phrase candidate data from the bilingual phrase candidate data storage unit 3 based on the bilingual document pair (group) data representing the bilingual document formed by associating documents in each of a plurality of languages. For all of a plurality of languages (that is, for both the language J and the language E), a bilingual phrase set candidate in which a word series exists in the bilingual document is selected and acquired.
The parallel phrase candidate acquisition unit 6 reads the parallel dictionary data from the parallel dictionary data storage unit 4 and is an entry registered in the parallel dictionary, and includes the word of the entry in the parallel document pair for all of a plurality of languages. Select and acquire bilingual phrases that have a series.

  The parallel phrase candidate ranking processing unit 7 performs ranking obtained by the phrase candidate obtaining unit 6. Ranking is based on a score or the like. Details of the ranking and score will be described later. The bilingual phrase candidate ranking processing unit 7 ranks all the translated phrase candidates acquired by the bilingual phrase candidate acquiring unit 6 in the entire order. In other words, about two bilingual phrase candidates arbitrarily selected from all the bilingual phrase candidates acquired by the bilingual phrase candidate acquiring unit 6, whether one is higher and the other is lower or they are in the same rank? Become decisive.

  The alignment processing unit 8 performs inclusive alignment in a predetermined procedure according to the order of the parallel translation phrase candidates assigned by the parallel translation phrase candidate ranking processing unit 7 and outputs the processing result. The definition of “inclusive” and the alignment processing procedure will be described later.

  As will be described later, in this embodiment, a score is calculated based on a statistic (translation phrase candidate alignment rate) using the alignment result, and ranking of the medical phrase candidates is performed based on this score. . Therefore, the process of the bilingual phrase candidate ranking processing unit 7 and the process of the alignment processing unit 8 are repeated a plurality of times. At this time, since there is no alignment processing result in the first ranking, the score is calculated without using the statistic based on the alignment processing result, ranking is performed, and the alignment processing is performed based on the ranking. As a result, since the alignment processing result is obtained, the alignment processing can be performed by using a statistic that uses the alignment result after the second time. Since the second alignment processing result is more reliable than the first alignment result, a statistic that uses the alignment processing result is calculated again from the second alignment processing result, and the third alignment is performed based on the ranking of the result. To do. As described above, the above-described repetition may be performed until the processing result converges within a predetermined range, or the repetition may be stopped at a predetermined number of times (for example, three times).

  Next, the details of the processing by the bilingual expression processing device 200 will be described. In the following, the alignment method will be described first, then the statistical feature amounts used in the alignment will be described, the score calculation method will be described, and finally the position selection method will be described.

<2.1 Alignment method>
The bilingual phrase candidate ranking processing unit 7 ranks the parallelism of the bilingual phrases acquired by the bilingual phrase candidate acquiring unit 6. The order of parallelism is determined as follows. That is, first, the translation phrase candidates registered in the aforementioned translation dictionary data are collectively referred to as Tier 1 for convenience. Next, candidates for bilingual phrases that are not included in tier 1 and that include content words on both sides of the phrase pair and whose scores are equal to or higher than a predetermined threshold are collectively referred to as tier 2. Next, candidates for parallel phrases that are not included in tiers 1 and 2 and whose scores are equal to or higher than a predetermined threshold are collectively referred to as tier 3.
Bilingual phrase candidates that are not included in any of Tiers 1 to 3 (that is, those whose scores are not in the above thresholds) are not used.

  Here, content words are words having content meanings such as general nouns, verbs, and adjectives in Japanese. On the other hand, what is not a content word is a function word, and a function word has a grammatical function but has no content meaning.

The order among the above three tiers is tier 1, tier 2, and tier 3 in order from the top.
The ranks of the parallel phrase candidates within Tier 1 are as follows. That is, when the bilingual dictionary data has the above-mentioned priority, the order of each bilingual phrase candidate is the order of the priority. However, if the priority happens to be the same, the ranking is the same. When the bilingual dictionary data does not have this priority, all the bilingual phrase candidates in the tier 1 have the same rank.
The ranking of the translation phrase candidates within the tier 2 is the order of the scores of the respective translation phrase candidates. If the scores happen to be the same, they will be tied.
The ranking of the translation phrase candidates in the tier 3 is the order of the scores of the respective translation phrase candidates. If the scores happen to be the same, they will be tied.
As described above, all the target parallel phrase candidates are ranked.

  In other words, the bilingual phrase candidate ranking processing unit 7 ranks the likelihood of the bilingual phrase pair (group) candidates based on the statistics read from the bilingual phrase candidate data storage unit 3.

  The alignment processing unit 8 performs alignment by discriminatingly selecting a parallel translation phrase from the parallel translation phrase candidates that appear in the parallel sentence pair. The phrase pairs to be aligned here are limited to those that are not connected to other alignments (not connected) or that have an inclusive relationship with other alignments. The definitions of “unconnected” and “inclusive” here will be described later.

  An alignment that satisfies the above restrictions is called an aligned alignment. The alignment processing unit 8 tries the alignment in order from the translated phrase candidate having the higher rank. When trying to align using a parallel phrase candidate in a certain order, if the alignment is not connected to an already established alignment or if it is inclusive but connected, the alignment is established, otherwise The alignment is not established (that is, the translated phrase candidate is not used). And the alignment process part 8 outputs the result of having tried the alignment of all the target parallel translation phrase candidates according to order. By performing the processing in such a procedure, all the alignments included in the processing result are aligned with each other. Also, by trying to align according to the rank, phrase pairs that are highly likely to be translated are preferentially aligned.

  FIG. 11 is a schematic diagram illustrating an example of the alignment processing result output by the alignment processing unit 8. This example of processing results is the result of processing the bilingual text example shown in FIG. 3 by inputting it from the bilingual document pair data input unit 5. FIG. 11 first shows the position of a word in the sentence in each language of the input bilingual sentence as a number in parentheses immediately after the word. For example, the word “typhoon” in the Japanese sentence is at position “1”, the word “ha” is at position “2”, the word “tomorrow” is at position “3”, and so on. The last punctuation mark “.” Of the Japanese sentence is also treated as a word here for convenience, and its position is “20”. Also, the word “The” in the corresponding English sentence is at position “1”, the word “typhoon” is at position “2”, the word “is” is at position “3”, and so on. The last period “.” Of the English sentence is also treated as a word here for convenience, and its position is “15”.

FIG. 11 also shows alignment processing results. In the first line of the processing result, the Japanese phrase “typhoon” (position “1”) is aligned with the English phrase “typhoon” (position “2”). In the second line, the Japanese phrase “typhoon” (position “1”) is aligned with the English phrases “The” (position “1”) and “typhoon” (position “2”). In the third line, the Japanese phrases “As” (position “3”) and “no” (position “4”) are aligned with the English phrase “tomorrow” (position “13”). The same applies to the alignment between phrases shown in the fourth and subsequent lines.
As described above, the data of the alignment processing result includes information on the adopted parallel phrase pair (group) candidates.

  FIG. 12 is a schematic diagram for explaining alignment consistency. Here, FIGS. 12A to 12C will be described with reference to the alignment processing result of FIG.

  First, FIG. 12A conceptually shows the alignment of the first and second rows shown in FIG. 11 in a plan view. In this figure, the symbol J1 corresponds to the Japanese word “typhoon” at position “1”. The symbol E1 corresponds to the English word “The” at the position “1”, and the symbol E2 corresponds to the English word “typhoon” at the position “2”. The symbol A corresponds to the alignment of the first line. The frame of “A” encloses the symbols J1 and E2 indicates that the Japanese phrase “typhoon” and the English phrase “typhoon” correspond in the alignment of the first line. Similarly, the symbol i corresponds to the alignment of the second line, and the frame of “I” surrounds the symbols J1, E1, and E2 because the Japanese phrase “Typhoon” and the English phrases “The”, “ "typhoon" shows that it corresponds. As shown in this figure, the symbol “I” includes the symbol “A”. Therefore, the alignments represented by these symbols “a” and “a” are consistent.

  Next, FIG. 12 (b) shows the third, fourth, fifth, and sixth lines of the alignment shown in FIG. 11, which correspond to the symbols K, K, K, and K, respectively. doing. Similar to the above, J3 represents the Japanese word “Tomorrow”, J4 represents the word “no”, J5 represents the word “daytime”, J6 represents the word “around”, and E13 represents the English word “tomorrow”. E14 represents the word “afternoon”. In this figure, the symbol “K” includes the symbol “K”. The symbol “K” includes the symbols “K”, “K”, and “K”, respectively. In addition, the symbol “K” and the symbol “K” are not connected, and the symbol “K” and the symbol “K” are not connected. Thus, since the symbols “K” to “K” are either inclusive of each other or not connected, the alignments represented by these symbols “K” to “K” are consistent. Yes.

  Next, FIG. 12C shows an example in which the two alignments do not match each other. In this figure, the alignment indicated by the solid frame includes the words J3 and E13. On the other hand, the alignment indicated by the dashed frame includes the words J4 and E13. In other words, these two are not inclusive of each other and are not in an unconnected relationship, so they are not consistent with each other. Conversely, if the alignment indicated by the solid frame has already been established, the alignment processing unit 8 described above adopts this candidate even if there is a parallel translation candidate candidate indicated by the dashed frame. Therefore, the alignment indicated by the dashed frame is not established.

The definitions of the inclusion relationship and the non-connection relationship illustrated using FIG. 12 are as follows.
Alignment A and B are inclusive (inclusive) means that all elements (words) included in alignment A are included in alignment B, or all elements included in alignment B are all This is the case where at least one of the elements included in the alignment A holds, and only in that case.
The fact that the alignments A and B are in an unconnected relationship means that none of the elements (words) included in the alignment A are elements of the alignment B, and none of the elements included in the alignment B are elements of the alignment A. It is a case and only in that case.

  Then, as a processing procedure, the alignment processing unit 8 can establish a new alignment by using the translation phrase candidate because the new alignment is inclusive or non-existent with respect to any already established alignment. This is the case when any of the connection relationships is established, and only in that case.

In summary, the alignment processing unit 8 selects a plurality of parallel phrase pair (group) candidates from a plurality of parallel phrase pair (group) candidates acquired by the parallel phrase candidate acquisition unit 6 based on the parallel document pair (group) data. Determine whether or not to adopt each of the above-mentioned translated phrase pair (set) candidates so that they have a matching relationship (either inclusive relationship or non-linked relationship), and the adopted translated phrase pair (Set) Outputs alignment processing results including candidate information.
In addition, the alignment processing unit 8 preferentially adopts a probable translation phrase pair (group) candidate based on the ranking by the translation phrase candidate ranking processing unit 7.
The alignment processing unit 8 prioritizes the parallel phrase acquired from the parallel translation dictionary data storage unit 4 by the parallel phrase candidate acquisition unit 6 over the parallel phrase pair (group) candidate acquired from the parallel phrase candidate data storage unit 3. The alignment processing result including the information of the adopted parallel translation phrases is output. This is because the bilingual phrase candidate ranking processing unit 7 gives a higher rank as the tier 1 to the bilingual phrases registered in the bilingual dictionary data as described above.

<2.2 Statistical features>
Next, the feature amount used for calculating the bilinguality score for the bilingual phrase candidate will be described. Here, six types of feature quantities can be used. They are significance probability, Dice coefficient, phrase average generation probability, phrase generation probability, word alignment result content rate, and parallel phrase candidate alignment rate. Of these six types, the first four types of feature quantities have already been described, and thus description thereof is omitted here.

<2.2.1 Word alignment result content rate>
In the word-by-word alignment result, the word alignment result content rate is the word alignment rate among the words in the corresponding translation phrase candidate phrase. Here, the alignment result of IBM model 4 which is the standard setting of the computer program “GIZA ++” is used as the alignment result in units of words. Since word alignment aligns one word in one language with an arbitrary number of words in the other language, the result differs depending on which language is used for alignment. Here, as a result of performing word alignment for both Japanese and English and English-Japanese bi-directional, the content rate (referred to as “AND” for convenience) when only word pairs aligned in either direction are the alignment results, The content rate (referred to as “OR” for convenience) when word pairs aligned in at least one of the directions are used as the alignment result can be calculated and used as a statistic.

<2.2.2 Bilingual phrase candidate alignment rate>
As a result of aligning parallel data, (q _s +1) / (q _a +1) is converted into a parallel phrase candidate alignment by using the number q _{s in} which the parallel phrase candidates are selected as the alignment and the total number q _{a in} which the parallel phrase candidates appear. It is defined as rate.
Thus, the translation phrase candidate alignment rate is a statistic (second statistic) calculated based on the alignment processing result.

<2.3 Score Calculation Method>
Of the six types (of which the word alignment result content rate is further divided into two types, AND and OR), −log (p-value × 2) is set to h ₁ , and the Dice coefficient is set to h ₂ . the phrase average generation probability and h _3, the phrase generation probability and h _4, word alignment results content of (aND) and h _5, the word alignment results content of (OR) and h _6, the bilingual phrase candidate alignment factor h ₇ The bilingual phrase candidate ranking processing unit 7 uses the feature values of h _{1 to} h ₇ to calculate a score according to the following equation (5).

Here, λ _i is a parameter for weighting each feature amount, and γ _i is a smoothing parameter. These parameter values may be set using any means. For example, the parameter values can be set by error rate minimization learning. The error rate minimization learning is described in the following document. Literature: Franz Josef Och, “Minimum Error Rate Training in Statistical Machine Translation”, ACL, pp. 160-167, 2003.

<3. Bilingual phrase candidate extraction and phrase pair alignment>
Next, a bilingual expression processing apparatus 300 having a form that includes both parallel phrase candidate extraction processing and phrase pair alignment processing will be described.
FIG. 13 is a block diagram illustrating a functional configuration of the parallel expression processing apparatus 300. As illustrated, the parallel translation expression processing device 300 includes a parallel document pair group data storage unit 1A, a parallel document pair group data analysis processing unit 2A, a parallel phrase candidate data storage unit 3A, a parallel dictionary data storage unit 4A, A bilingual document pair data input unit 5A, a bilingual phrase candidate acquisition unit 6A, a bilingual phrase candidate ranking processing unit 7A, and an alignment processing unit 8A are included.

Among the functions of the bilingual expression processing device 300, the bilingual document pair group data storage unit 1A, the bilingual document pair group data analysis processing unit 2A, and the bilingual phrase candidate data storage unit 3A include bilingual phrase pairs from bilingual document pair group data. This is a process of extracting candidates, and the specific processing procedure and the like are the same as those of the bilingual expression processing apparatus 100 described above. Also, the bilingual document pair data input unit 5A, the bilingual phrase candidate acquisition unit 6A, the bilingual phrase candidate ranking processing unit 7A, and the alignment process based on the data of the bilingual phrase candidate data storage unit 3A and the bilingual dictionary data storage unit 4A The processing of the part 8A is alignment processing, and the specific processing procedure is the same as that of the bilingual expression processing apparatus 200 described above.
Thus, the parallel translation expression processing device 300 performs the parallel translation phrase candidate extraction process, writes the parallel translation phrase candidate data in the parallel translation phrase candidate data storage unit 3A, and performs the alignment process while reading the parallel translation phrase candidate data.

If the data input from the parallel document pair data storage unit 1A shown in FIG. 13 and the data input from the parallel document pair data input unit 5A are the same, the bilingual phrase candidate is extracted as shown in FIG. By outputting and storing the values of Y ^j and Y ^e that appear in the algorithm as a set together with bilingual phrase candidates, it is already known which candidate appears at which position in which bilingual document pair. The parallel phrase candidate acquisition process can be started, that is, the process of the parallel phrase candidate acquisition unit 6A in the configuration of FIG. 13 can be made efficient.

  In addition, you may make it implement | achieve a part or all function of the bilingual expression processing apparatus in embodiment mentioned above with a computer. In that case, the program for realizing the bilingual document processing may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into the computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It is also possible to include those that hold a program for a certain time, such as a volatile memory inside a computer system serving as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

As mentioned above, although embodiment of this invention was described, this invention can also be implemented also in the following modifications.
For example, in the above-described embodiment, the bilingual document and the bilingual phrase are targeted only in two languages (language J and language E), but the same processing method is essentially applicable to a combination of roles in three languages or more. You can extract and align bilingual phrase candidates. For example, in the case of language F in addition to language J and language E, the pseudo code shown in FIGS. 7 and 9 is extended to provide a procedure called ExpandF, and instead of outputting a phrase pair in ExpandE, procedure ExpandF , The phrase pair is output in the procedure ExpandF and the procedure ExpandF is called recursively. The same is true for four or more languages.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

For example, in the above-described embodiment, significant probability, Dice coefficient, phrase average generation probability, phrase generation probability, word alignment result content rate, bilingual phrase candidate alignment rate are used as the statistical amount. It may be used. Further, in the present embodiment, since a statistic based on the alignment processing result is used in part, the processing result by the alignment processing unit 8 is fed back to the parallel phrase candidate ranking processing unit 7 for ranking processing and alignment processing. Is repeated multiple times, but if ranking is performed based only on statistics that do not depend on the result of alignment processing when ranking parallel phrase candidates, such multiple iteration processing is necessary. Absent.
In the above-described embodiment, when depth-first search processing is performed, a new sequence is created by adding words to the end of the sequence when moving from a parent node to a child node in a tree-like search space. However, the location where the word is added is arbitrary, and a new sequence may be generated by concatenating the words at the beginning, or a new sequence may be inserted by inserting a word at an arbitrary position in the existing sequence. It may be generated.
In the above-described embodiment, the depth-first search process is performed, but a breadth-first search may be performed instead. In the case of the breadth-first search, a large amount of memory (spatial calculation amount) for storing the state during the search is required, but if the apparatus has sufficient memory, the same result as the depth-first search can be obtained.

It is a block diagram which shows the function structure of the bilingual expression processing apparatus (extraction process of a bilingual phrase candidate) by embodiment of this invention. It is the schematic which shows the structure of the parallel translation expression document pair group data by the embodiment. It is the schematic which shows an example of the parallel translation expression document pair in the embodiment. It is the schematic which shows the structure and data example of bilingual phrase candidate data by the embodiment. FIG. 6 is a schematic diagram illustrating the operation of processing of a phrase depth priority search in a single language (searching for a phrase starting with the word a, threshold ζ = 2) according to the embodiment. It is a pseudo code which shows the processing sequence of the phrase depth priority search in a single language by the embodiment. It is a pseudo code which shows the process which extracts the frequent phrase pair based on a depth priority search from parallel document pair group data by the embodiment. 4 is a 2 × 2 contingency table of the number of data (number of sentences) in the corpus according to the embodiment. It is a pseudo code which shows the procedure of the search process which applied the pruning method of search space by the embodiment. It is a block diagram which shows the function structure of the parallel translation expression processing apparatus (alignment process) by embodiment of this invention. It is the schematic which shows the example of the alignment process result output by the alignment process part by the same embodiment. It is the schematic explaining the alignment consistency in the same embodiment. It is a block diagram which shows the function structure of the bilingual expression processing apparatus (the extraction process and the alignment process of a bilingual phrase candidate) by embodiment of this invention.

Explanation of symbols

1,1A Bilingual document group data storage unit (Bilingual document group data storage unit)
2,2A Bilingual document group data analysis processing unit (Bilingual document group data analysis processing unit)
3, 3A Bilingual phrase candidate data storage unit 4, 4A Bilingual dictionary data storage unit 5, 5A Bilingual document pair data input unit 6, 6A Bilingual phrase candidate acquisition unit 7, 7A Bilingual phrase candidate ranking processing unit 8, 8A Alignment processing unit 100 parallel translation expression processing device (processing for extracting parallel phrase candidate data)
200 Parallel expression processing device (Process to align parallel phrases)
300 Bilingual expression processing device (Branch phrase candidate data extraction process and Bilingual phrase alignment process)

Claims (5)

A bilingual document group data storage unit for storing a plurality of bilingual document group data which is a set of bilingual documents in a plurality of languages;
Based on the bilingual document set data read from the bilingual document set data storage unit, the co-occurrence frequency between the plurality of languages of word sequences appearing in a single bilingual document set data is counted, Bilingual document set group data analysis processing for extracting and outputting the set of word sequences in the plurality of languages as a bilingual phrase set candidate such that the total value of the co-occurrence frequencies in the bilingual document set data is equal to or greater than a predetermined frequency threshold And
A translated expression processor you equipped with,
The bilingual document set group data analysis processing unit searches the word series in the first language data of the plurality of languages to search for the word series in the first language in all the bilingual document set data. Counting the appearance frequency, and searching for word sequences in the data of other languages of the plurality of languages for each word sequence in the first language such that the appearance frequency is equal to or higher than the frequency threshold, Counting the co-occurrence frequency of the word sequence of the first language and the word sequence of the other language, and making this a co-occurrence frequency between the plurality of languages ,
The bilingual document set group data analysis processing unit obtains a new word sequence obtained by replacing one word in the current word sequence after the current word sequence when sequentially searching for the word sequence. Rather, the new word sequence obtained by adding one word to the current word sequence is preferentially searched first.
The bilingual expression processing apparatus characterized by this. The bilingual expression processing device according to claim 1,
The bilingual document set group data analysis processing unit calculates a statistic indicating the probability of being a bilingual phrase set for each of the bilingual phrase set candidates to be extracted, and outputs the calculated statistic together with the bilingual phrase set candidate.
The bilingual expression processing apparatus characterized by this. A bilingual document group data storage unit for storing a plurality of bilingual document group data which is a set of bilingual documents in a plurality of languages;
Based on the bilingual document set data read from the bilingual document set data storage unit, the co-occurrence frequency between the plurality of languages of word sequences appearing in a single bilingual document set data is counted, Bilingual document set group data analysis processing for extracting and outputting the set of word sequences in the plurality of languages as a bilingual phrase set candidate such that the total value of the co-occurrence frequencies in the bilingual document set data is equal to or greater than a predetermined frequency threshold And
A bilingual expression processing apparatus comprising:
The bilingual document set group data analysis processing unit searches the word series in the first language data of the plurality of languages to search for the word series in the first language in all the bilingual document set data. Counting the appearance frequency, and searching for word sequences in the data of other languages of the plurality of languages for each word sequence in the first language such that the appearance frequency is equal to or higher than the frequency threshold, Counting the co-occurrence frequency of the word sequence of the first language and the word sequence of the other language, and making this a co-occurrence frequency between the plurality of languages,
The bilingual document set group data analysis processing unit converts a pair of the first language word series and the other language word series into a bilingual phrase set during a search for a word series in the other language data. A statistic representing the certainty is calculated, and the probability represented by the statistic calculated for the current word sequence in the other language is a word obtained by subtracting one word from the current word sequence in the other language. If it is lower than the probability expressed by the statistic already calculated for the sequence and lower than the probability expressed by the predetermined statistic threshold, it is not matched with the current word sequence of the other language. Suppress the search for new word sequences obtained by adding words,
The bilingual expression processing apparatus characterized by this. A computer having a parallel document group data storage unit that stores a plurality of parallel document group data that is a set of parallel documents in a plurality of languages.
Based on the bilingual document set data read from the bilingual document set data storage unit, the co-occurrence frequency between the plurality of languages of word sequences appearing in a single bilingual document set data is counted, Bilingual document set group data analysis processing for extracting and outputting the set of word sequences in the plurality of languages as a bilingual phrase set candidate such that the total value of the co-occurrence frequencies in the bilingual document set data is equal to or greater than a predetermined frequency threshold process,
A computer program for executing the process of
In the bilingual document set group data analysis process, the word series in the first language in all the bilingual document set data is searched by searching for the word series in the data of the first language of the plurality of languages. Counting the appearance frequency, and searching for word sequences in the data of other languages of the plurality of languages for each word sequence in the first language such that the appearance frequency is equal to or higher than the frequency threshold, Counting the co-occurrence frequency of the word sequence of the first language and the word sequence of the other language, and making this a co-occurrence frequency between the plurality of languages ,
In the bilingual document set data analysis process, a new word sequence obtained by replacing one word in the current word sequence after the current word sequence when sequentially searching for the word sequence Rather, the new word sequence obtained by adding one word to the current word sequence is preferentially searched first.
A computer program characterized by the above.   A computer having a parallel document group data storage unit that stores a plurality of parallel document group data that is a set of parallel documents in a plurality of languages.
  Based on the bilingual document set data read from the bilingual document set data storage unit, the co-occurrence frequency between the plurality of languages of word sequences appearing in a single bilingual document set data is counted, Bilingual document set group data analysis processing for extracting and outputting the set of word sequences in the plurality of languages as a bilingual phrase set candidate such that the total value of the co-occurrence frequencies in the bilingual document set data is equal to or greater than a predetermined frequency threshold process,
  A computer program for executing the process of
  In the bilingual document set group data analysis process, the word series in the first language in all the bilingual document set data is searched by searching for the word series in the data of the first language of the plurality of languages. Counting the appearance frequency, and searching for word sequences in the data of other languages of the plurality of languages for each word sequence in the first language such that the appearance frequency is equal to or higher than the frequency threshold, Counting the co-occurrence frequency of the word sequence of the first language and the word sequence of the other language, and making this a co-occurrence frequency between the plurality of languages,
  In the bilingual document set group data analysis process, a set of the first language word sequence and the other language word sequence is converted into a parallel phrase set during the search of the word sequence in the data of the other language. A statistic representing the certainty is calculated, and the probability represented by the statistic calculated for the current word sequence in the other language is a word obtained by subtracting one word from the current word sequence in the other language. If it is lower than the probability expressed by the statistic already calculated for the sequence and lower than the probability expressed by the predetermined statistic threshold, it is not matched with the current word sequence of the other language. Suppress the search for new word sequences obtained by adding words,
  A computer program characterized by the above.

Images