JP2015095182A - Character string processing device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire a character position of an input character string before replacement even when the character position is changed by performing replacement processing for the input character string.SOLUTION: A character processing device 10 includes a replacement part 14 that: replaces a first partial character string included in an input character string by a second partial character string on the basis of a collection of predetermined replacement rules for replacing the first partial character string by the second partial character string; and generates and provides character position information and length information of the second partial character string in the input character string after the replacement, on the basis of character position information and length information of the first character string in the input character string before the replacement.

Description

  The present invention relates to a character string processing device, method, and program, and more particularly, to a character string processing device, method, and program for processing a character string before translation in machine translation.

  2. Description of the Related Art Conventionally, a machine translation technique for mechanically converting a sentence in one natural language into a sentence in another natural language using a computer program is known. In recent years, statistical machine translation that automatically realizes machine translation using a statistical model is known (see, for example, Non-Patent Document 1).

  In statistical machine translation, for example, when an English sentence is translated into a Japanese sentence, the character position of each character in the Japanese sentence, which is the translation result, is the character in the English sentence before translation. There are many requests to know if it corresponds to a position.

  In statistical machine translation, it is common to perform a substitution process that converts mathematical formulas, chemical formulas, long proper nouns, and the like into virtual words or deletes articles in pre-translation processing.

P.Koehn et al. 2007. "Moses: Open Source Toolkit for Statistical Machine Translation"

  However, in general statistical machine translation, only the phrase correspondence before and after translation is supported, so if the character position is changed by executing the above pre-translation processing on the input sentence to be translated Thus, it is difficult to associate the character position of the input sentence with the translated sentence after translation.

  The present invention has been made in view of the above circumstances, and even when the character position is changed by executing a replacement process on the input character string, the character position of the input character string before replacement can be obtained. An object of the present invention is to provide a character string processing apparatus, method, and program.

  In order to achieve the above object, the character string processing device according to the present invention inputs according to the replacement rule based on a set of predetermined replacement rules for replacing the first partial character string with the second partial character string. The first partial character string included in the character string is replaced with the second partial character string, and based on character position information and length information of the first partial character string in the input character string before replacement A replacement unit that generates and assigns character position information and length information of the second partial character string in the input character string after replacement.

  In the replacement unit, the first partial character string in the replacement rule includes an empty character, and for each character obtained by dividing the input character string in character units, the first character position of the input character string is determined. The character position information indicating the character position of the character as 0 and the length information in which the length of each character of the input character string is set to 1 are replaced according to the replacement rule. When the first partial character string is an empty character, the character position information of each character of the replaced second partial character string is generated as 0, and the character of each character of the second partial character string is When the length information is generated as 0 and the first partial character string to be replaced according to the replacement rule is not a null character, the character position information of each character of the replaced second partial character string is the first character string. 1 for each character of the substring A character string processing unit that generates a character position information as a minimum value and generates the length information of each character of the second partial character string as the length of the first partial character string. It is good.

  The character string processing method according to the present invention is a character string processing method executed in a character string processing device including a replacement unit, wherein the replacement unit converts the first partial character string into a second partial character string. Based on a set of predetermined replacement rules to be replaced, the first partial character string included in the input character string is replaced with the second partial character string according to the replacement rule, and the input character before replacement Based on the character position information and length information of the first partial character string in the string, the character position information and length information of the second partial character string in the input character string after replacement are generated and attached. Includes steps.

  The character string processing program of the present invention is a program for causing a computer to function as each part of the character string processing device according to claim 1 or claim 2.

  As described above, according to the character string processing device, method, and program of the present invention, even when the character position is changed by executing the replacement process on the input character string, the input character string before replacement is changed. The effect that the character position can be obtained is obtained.

It is a block diagram which shows the functional structural example of the character string processing apparatus which concerns on this Embodiment. It is a figure which shows an example of the character string before substitution, character position information, and length information. It is a figure which shows an example of the character string after substitution, character position information, and length information. It is a figure which shows an example of the character string before substitution, character position information, and length information. It is a figure which shows an example of the character string after substitution, character position information, and length information. It is a flowchart which shows the character string processing routine in this Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<Summary of invention>
First, an outline of an embodiment of the present invention will be described.
In the present embodiment, as the pre-translation processing, the first partial character string included in the pre-substitution character string as the input sentence is replaced with the second partial character string. At this time, based on the character position information and length information of the first partial character string, the character position information and length information of the second partial character string after replacement are generated and attached. This makes it possible to appropriately associate the character position of the input sentence with the translated sentence after translation even when the character position changes by executing pre-translation processing on the input sentence to be translated. .

<System configuration>
The character string processing apparatus 10 according to the present embodiment includes an input unit 12, a replacement unit 14, and an output unit 16.

  The input unit 12 receives input of an input sentence to be translated. The input unit 12 is realized by an input device such as a keyboard, a mouse, and a storage device.

  The replacement unit 14 functionally includes a replacement rule storage unit 18, an information generation unit 20, and a character string processing unit 22. The replacement unit 14 includes, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory) storing a character string processing program and a replacement rule for executing a character string processing routine described later, and Are realized by a computer equipped with. Note that a nonvolatile memory may be used instead of the ROM.

  The replacement rule storage unit 18 includes, as pre-processing for statistical machine translation, a replacement rule that describes a rule for replacing the first partial character string included in the input sentence to be translated with the second partial character string. Remember the set. Note that the “characters” in the present embodiment include not only characters representing languages of each country such as Roman characters and Chinese characters, but also numbers and symbols. The “partial character string” includes a case of one character. The “partial character string” includes a null character. Hereinafter, an input sentence input by the input unit 12 may be referred to as a pre-translation character string, and a character string in which the pre-translation character string is replaced according to a replacement rule may be referred to as a post-substitution character string.

  In this embodiment, two replacement rules will be described. For example, the first replacement rule is “if the specific character does not exist at the end of the input sentence, the specific character is inserted at the end of the sentence. The character position information of the inserted specific character is“ 0 ”, and the length of the inserted specific character. The information is “0”.

  The specific character is a character that represents a period that is usually written at the end of a sentence, such as “.” And “.”. In other words, the first replacement rule can be said to be a rule for replacing the last empty character (first partial character string) of the character string before replacement with a specific character (second partial character string).

  Further, for example, the second replacement rule is “a specific character string in which at least one of a symbol and a number continues is replaced with a virtual word. All character position information of each character of the inserted virtual word is the minimum of the specific character string. The length information of each character of the inserted virtual word is all (the character position of the last character of the specific character string + the length of the last character−the length of each character of the specific character string). The smallest character position among the character positions).

  The virtual word is a word such as “Suisuki” in the case where the specific character string is a mathematical expression including a number and an arithmetic symbol, for example. In addition, (the character position of the last character of the specific character string + the length of the last character−the minimum character position among the character positions of each character of the specific character string) is, in other words, the length of the specific character string. It is.

  The information generation unit 20 divides the input sentence input by the input unit 12, that is, the character string before substitution into character units, and generates and assigns character position information and length information to each divided character. For example, as shown in FIG. 2, when the pre-replacement character string is “This is a pencil”, it is composed of 16 characters including an empty character (space), so it is divided into 16 characters. Then, for the first character (leftmost character) of the pre-replacement character string, character position information with the character position set to “0” is generated, and one character is advanced toward the last character (rightmost character). In addition, character position information obtained by adding character positions one by one is generated for each character. As a result, as shown in FIG. 2, “0” is generated as the character position information for the first character “T”, and “15” is set as the character position information for the last character “l”. Generated. As the length information, “1” is generated for all characters as shown in FIG.

  The character string processing unit 22 replaces the pre-replacement character string input by the input unit 12 based on the replacement rule stored in the replacement rule storage unit 18. For example, a sentence that does not have a period as a specific character at the end of the pre-translation character string, such as “This is a pencil” shown in FIG. 2, follows the first substitution rule, “This is a pencil” shown in FIG. Insert a period at the end. That is, the last empty character (first partial character string) in the character string before replacement is replaced with a specific character (second partial character string).

  In addition, as shown in FIG. 3, the character string processing unit 22 generates and assigns “0” as character position information and “0” as length information to a period as an inserted specific character.

  Also, for example, as shown in FIG. 4, a sentence including a specific character string (first partial character string) representing a mathematical expression such as “2 + 3 = 5” follows the second replacement rule as shown in FIG. The specific character string is replaced with a virtual word (second partial character string) such as “SUSUKIKI”.

  Further, the character position of “5” which is the last character of the specific character string is “4”, the length of “5” which is the last character of the specific character string is “1”, The minimum character position among the character positions of each character, that is, the character position of “2” that is the first character of the specific character string is “0”. For this reason, the character string processing unit 22 generates “0” as character position information and “5” (= 4 + 1−0) as length information for each character of the virtual word. That is, the length information of each character of the virtual word is the same as the length of the specific character string.

  Thus, when the first partial character string to be replaced is an empty character, the character string processing unit 22 sets the character position information of each character of the replaced second partial character string to “0” and the length of each character. Information is generated and assigned as “0”. On the other hand, if the first partial character string to be replaced is not a null character, the character position information of each character of the replaced second partial character string is set as the minimum value of the character position information of the first partial character string, Character length information is generated and assigned as the length of the first partial character string.

<Operation of character string processing device>
Next, a character string processing routine executed in the character string processing apparatus 10 according to the present embodiment will be described with reference to FIG.

  In step S100, an input of a character string to be translated is accepted. In the present embodiment, an example will be described in which an input character string is an English character string, and the English character string is replaced by the first replacement rule and the second replacement rule.

  In step S102, the input sentence input in step S100 is divided into character units, and character position information indicating the character position of each divided character and length information indicating the character length are generated and attached. For example, as shown in FIG. 2, when the input sentence as the pre-replacement character string is “This is a pencil”, it is divided into 16 characters, and character position information and length as shown in FIG. Information is generated.

  In step S104, the pre-replacement character string is replaced based on the set of replacement rules stored in the replacement rule storage unit 18. If the first partial character string to be replaced is an empty character, the character position information of each character of the replaced second partial character string is set to “0”, and the length information of each character is set to “0”. On the other hand, if the first partial character string to be replaced is not a null character, the character position information of each character of the replaced second partial character string is set as the minimum value of the character position information of the first partial character string, Character string processing is performed using the character length information as the length of the first partial character string.

  For example, when the pre-replacement character string is a character string as shown in FIG. 2, there is no period at the end of the sentence. Therefore, according to the first replacement rule, a period is added to the end of the character string before replacement as shown in FIG. That is, the last empty character (first partial character string) in the character string before replacement is replaced with a specific character (second partial character string).

  Since the first partial character string to be replaced is an empty character, “0” is generated as character position information and “0” is generated as length information for the period as the replaced second partial character string. To do.

  Note that the pre-replacement character string shown in FIG. 2 does not include a character string in which at least one of a symbol and a number continues, so that the replacement according to the second replacement rule is not performed.

  For example, when the pre-replacement character string is a character string as shown in FIG. 4, a period is present at the end of the sentence, so that the replacement based on the first replacement rule is not performed. On the other hand, since the character string shown in FIG. 4 includes the mathematical expression “2 + 3 = 5” as a specific character string (first partial character string) in which at least one of a symbol and a number continues, according to the second replacement rule, The mathematical expression is replaced with “Suisuki” as a virtual word (second partial character string).

  Further, since the mathematical expression as the first partial character string to be replaced is not a null character, “0” as the character position information, long for each character of “Suisuki” as the replaced second partial character string “5” is generated as the information.

  In step S106, the replaced character string replaced in step S104 is output to the subsequent translation apparatus. The latter-stage translation device is a device that converts, for example, a translated character string into another language, that is, a translation.

  Thus, according to the present embodiment, as a pre-translation process, even when the first partial character string included in the pre-substitution character string as the input sentence is replaced with the second partial character string, Based on the character position information and length information of the first partial character string, the character position information and length information of the second partial character string after replacement are generated. For this reason, even when the character position is changed by executing pre-translation processing on the input sentence to be translated, the character position of the input sentence can be appropriately associated with the translated sentence after translation.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, the first partial character string in the second replacement rule is not limited to a mathematical formula, but may be a chemical formula or a proper noun longer than a predetermined length. Further, the replacement rule is not limited to the first replacement rule and the second replacement rule described above, and other replacement rules may be stored.

  Further, the above-described character string processing apparatus 10 has a computer system therein, but the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. Shall be.

  Further, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium such as a CD-ROM or a memory card. It is.

DESCRIPTION OF SYMBOLS 10 Character string processing apparatus 12 Input part 14 Replacement part 16 Output part 18 Replacement rule memory | storage part 20 Information generation part 22 Character string processing part

Claims (4)

Based on a set of predetermined replacement rules for replacing the first partial character string with the second partial character string, the first partial character string included in the input character string according to the replacement rule is changed to the second partial character string. The second partial character string in the input character string after replacement is replaced with a partial character string and based on the character position information and length information of the first partial character string in the input character string before replacement. A character string processing apparatus including a replacement unit that generates and assigns character position information and length information. The first partial character string in the replacement rule includes a null character,
The replacement part is:
For each character obtained by dividing the input character string in character units, the character position information indicating the character position of the character with the leading character position of the input character string being 0, and the length of each character of the input character string An information generation unit that generates the length information with a length of 1;
When the first partial character string to be replaced according to the replacement rule is an empty character, the character position information of each character of the replaced second partial character string is generated as 0, and the second part When the length information of each character of the character string is generated as 0 and the first partial character string to be replaced according to the replacement rule is not a null character, each character of the replaced second partial character string The character position information is generated as a minimum value of the character position information of each character of the first partial character string, and the length information of each character of the second partial character string is generated as the first part. A character string processing unit for generating the length of the character string;
The character string processing device according to claim 1. A character string processing method executed in a character string processing device including a replacement unit,
The first partial character string included in the input character string according to the replacement rule based on a set of predetermined replacement rules in which the replacement unit replaces the first partial character string with the second partial character string. Is replaced with the second partial character string, and based on the character position information and length information of the first partial character string in the input character string before replacement, the second partial character string is replaced with the second partial character string. A character string processing method comprising: generating and assigning character position information and length information of the partial character string of 2.   A character string processing program for causing a computer to function as each part of the character string processing device according to claim 1.

Images