JPH08212225A - Language judgement device - Google Patents

Abstract

PURPOSE: To provide a language judgement device which can precisely judge a language name to which a character string belongs when the character string where the belonging language name is unknown. CONSTITUTION: A partial character string generation part 202 generates plural partial character strings in accordance with a prescribed rule from the given input character string. A retrieval part 203 retrieves the character strings stored in a feature dictionary 106b, searches the character string matched with the generated partial character string and obtains the language name to which the respective partial character strings possibly belong and an evaluation value showing the possibility. A language decision part 204 decides the language name to which the given input character string belong by calculating all the obtained evaluation values in accordance with the prescribed judgement rule.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a language determination device for determining a language name to which a character string belongs, when the character string to which the language name belongs is unknown.

[0002]

2. Description of the Related Art With the recent increase in international exchange, there are increasing opportunities to translate documents written in one language into another language. It is not uncommon for some documents to contain sentences and words written in multiple languages within a single document, and even in that case, a multilingual automatic translator that can appropriately translate into a document in one language Is expected.

However, in order to translate a document in which such multilingual sentences and words are mixed, prior to translation of the document, for each sentence or word constituting the document, the language name to which those sentences or words belong. It is necessary to perform a pre-process of accurately specifying As a conventional technique for language determination at that time, for example, “Japanese Patent Laid-Open No. 61-139884” discloses a “language type determination method”.

The contents of this prior art will be described with reference to the flow chart shown in FIG. Here, it is assumed that the character string to be determined is α1α2 ... αn and the candidate language names are A language, B language, ..., Z language. Then, a table in which unique characters and symbols belonging only to each language are collected is created in advance as follows.

A table that collects characters and symbols that belong only to the A language {a1, a2, ..., Ax} a table that collects characters and symbols that belong only to the B language {b
1, b2, ..., by} A table that collects characters and symbols that belong only to the Z language is {z
1, z2, ..., zz}.

When the character string to be judged is acquired (step 1001), the character string is compared with the characters in the table one by one from the beginning (steps 1002-1).
009). When a matching character is found, it is determined that the language corresponding to the table including the character is the desired language name (steps 1004, 1006, 100).
8).

[0007]

However, the above-mentioned prior art has the following problems. For example, if the A language is English and the B language is German, all the characters and symbols that belong to English belong to German, so the table that collects the characters and symbols that belong only to English is an empty set. I will end up.

Therefore, there is a problem that even if the language determination is performed on a certain English word, it cannot be determined that the word is English. Further, even if neither the A language table nor the B language table is an empty set, for a character string consisting only of characters commonly included in both A language and B language, the character string is A Since it belongs to both the language and the B language, there is a problem that the language name cannot be accurately determined.

Therefore, the present invention has been made in view of the above problems, and even in the case where a language name to which a given character string belongs must be determined for a language whose features are close to each other. Further, even if the language name has to be determined for a character string consisting of only characters that commonly belong to a plurality of languages, a language determination device that can accurately determine the language name to which the character string belongs is provided. The purpose is to do.

[0010]

In order to achieve the above object, a language determining apparatus according to claim 1 is a language determining apparatus for determining a language name to which a given input character string belongs, the input character string And a character string group having the same number of characters as the partial character string, and a language name to which each character string may belong, A feature dictionary storing candidate language information including an evaluation value indicating the possibility and a character string stored in the feature dictionary are searched, and candidate language information corresponding to a character string matching the partial character string is searched. It is characterized by comprising a searching means for acquiring and a language determining means for determining a language name of the input character string based on the candidate language information obtained as a result of the search.

According to a second aspect of the present invention, there is provided a language determination device according to the first aspect.
The described language determination device further includes a partial character string generation rule storage unit that stores a plurality of rules for generating a partial character string in advance, and the partial character string generation unit stores the partial character string generation rule memory. It is characterized in that the partial character string is generated according to one rule selected from a plurality of rules stored in the means.

According to a third aspect of the present invention, there is provided a language determination device according to the first aspect.
Alternatively, the language determination device according to the second aspect further includes a determination rule storage unit that stores a plurality of rules for determining a language name in advance, and the language determination unit is configured to store a plurality of rules stored in the determination rule storage unit. It is characterized in that the language name of the input character string is determined according to one rule selected from the rules.

[0013]

According to the language determining apparatus having the above-mentioned structure, when the input character string is given, the partial character string generating means generates one or more partial character strings. Then, the search unit searches for a character string in the feature dictionary that matches each generated partial character string, and the candidate language information corresponding to the matching character string, that is, the partial character string may belong. An evaluation value indicating a certain language name and its possibility is obtained. Further, the language determining means collects the candidate language information for each partial character string, and determines one language name to which the input character string is most likely to belong.

According to the language determining apparatus of the second aspect, the language determining apparatus of the first aspect further includes a partial character string generation rule storage means for storing a plurality of rules in advance for generating the partial character strings. And when generating a substring,
The partial character string is generated according to one rule selected from the plurality of rules. According to the language determination apparatus of claim 3, the language determination apparatus of claim 1 or 2 further comprises determination rule storage means for storing a plurality of rules for determining a language name in advance, Upon determination, the language name of the input character string is determined according to one rule selected from the plurality of rules.

[0015]

Embodiments of the present invention will be described in detail below with reference to the drawings. (Description of Configuration) FIG. 1 is a block diagram showing a hardware configuration of a language determination device according to an embodiment of the present invention.
FIG. 2 is a block diagram showing the functional configuration of this device.

The present apparatus has a CPU 103 as a center, an input device 101, a display device 102, a ROM 104, and a RAM.
105 and an external storage device 106. The input device 101 is composed of a keyboard and the like, and is mainly used for giving a character string to be judged (hereinafter referred to as “input character string”) to this device. The display device 102 is a CRT
It is mainly used to confirm the result of the judgment by this device.

The CPU 103 is composed of a microprocessor and the like, performs input / output with the input device 101 and the display device 102, and processes a character string according to a program (not shown) stored in the ROM 104. C
As shown in FIG. 2, the processing of the character string by the PU 103 is performed in accordance with the function of the input character string acquisition unit 201, the partial character string generation unit 202, the search unit 203, and the language determination unit 204.
are categorized.

The ROM 104 is composed of a semiconductor memory or the like, and stores a program corresponding to the flowchart shown in FIG. The RAM 105 is made up of a semiconductor memory or the like, and stores temporary information generated as the character string is processed by the CPU 103. The RAM 105, as shown in FIG. 2, according to the contents of the stored information,
Input character string storage unit 105a, partial character string storage unit 105
b, the candidate language information storage unit 105c.

The external storage device 106 is composed of a hard disk or the like, and depending on the contents of the stored information, as shown in FIG. 2, a partial character string generation rule storage unit 106a, a feature dictionary 106b, and a determination rule storage unit 106c are stored. being classified. As shown in FIG. 4, the partial character string generation rule storage unit 106a stores a plurality of generation rules each having a rule number and a generation rule as a pair. These generation rules are registered in advance based on past experience and the like. Note that the column of the specific example in the table shown in FIG. 4 is provided for convenience of description, and does not constitute the partial character string generation rule storage unit 106a. The column of this specific example shows only the character string corresponding to the corresponding generation rule in the partial character string starting from the i-th character.

Here, the meaning of the partial character string is shown in FIG.
Will be described with reference to. FIG. 5A shows an input character string consisting of n characters. αi represents the i-th character from the beginning of the input character string. FIG. 5 (b) shows
All substrings starting with αi (Si1, Si2, ..., Si
l) is shown. There are a total of (n-i + 1) types of substrings that start with αi. Here, the partial character string refers to a character string composed of a continuous character sequence of a part or all of the input character string. Therefore, all the partial character strings of the input character string consisting of n characters are α
All substrings that start with 1, all substrings that start with α2, and all substrings that start with αn.

As shown in FIG. 6, the feature dictionary 106b stores a plurality of pieces of information each including a character string and candidate language information. The candidate language information is made up of a language name to which the character string in the left column may belong and a pair of evaluation values indicating the possibility. These character strings and candidate language information are registered in advance based on statistical information regarding the appearance frequency of character strings for each language such as English, German, and French. As statistical information, for example, “n-Gram Statistics for Natural Language Understan by CYSuen
ding and Text Processing, IEEE Trans. Vol PAME1, N
o.2, Apr. 1979 ”can be adopted.

As shown in the table of FIG. 7, the determination rule storage unit 106c stores a plurality of determination rules each having a rule number and a determination rule as a pair. These determination rules are registered in advance based on past experience and the like. In addition, the number () in the column of the determination rule in the table shown in FIG. 4 indicates the procedure for the determination.

(Explanation of Operation) The operation of the language judgment apparatus configured as described above will be described with reference to FIGS. 2 and 3. FIG. 3 is a flowchart showing an operation procedure of the language determination device according to the embodiment of the present invention. The input character string acquisition unit 201 acquires the input character string (α1α2 ... αn) given from the input device 101, and the input character string storage unit 1
It is stored in 05a (step 301).

Subsequently, the partial character string generation unit 202 generates all the corresponding partial character strings from one input character string stored in the input character string storage unit 105a according to a predetermined rule, and generates the generated partial character strings. The string is a partial character string storage unit 105b.
(Step 303). Here, the predetermined rule refers to a rule corresponding to one rule number designated by the operator from the partial character string generation rules stored in the partial character string generation rule storage unit 106a. Therefore, the type and number of generated partial character strings differ depending on the content of the selected partial character string generation rule.

The order of generating the partial character strings is as follows. First, all the partial character strings (S
11, S12, ..., S1l) and then the processing of loop 2 (steps 304 to 306) is performed, and then all the partial character strings (S21, S) that start with the second character of the input character string are generated.
22 ..., and then the processing of loop 2 (step 3
04-306) is repeated, and finally, all the partial character strings (Sn1) starting with the last character of the input character string are generated, and then the processing of loop 2 (step 304-). 306) is performed (step 302-
307).

Next, the search unit 203 uses the feature dictionary 106b.
By searching for the corresponding candidate language information for each partial character string stored in the partial character string storage unit 105b, and storing the acquired candidate language information in the candidate language information storage unit 105c (step 305). . Specifically, for one partial character string stored in the partial character string storage unit 105b, a character string in the feature dictionary that matches the partial character string is searched for, and the corresponding candidate language information is candidate together with the character string. It is stored in the language information storage unit 105c. The above search is sequentially repeated for all the partial character strings stored in the partial character string storage unit 105b (step 30).
4-307). As a result, candidate language information about all the generated partial character strings is obtained.

Finally, the language determination unit 204 determines the language name to which the input character string is most likely to belong, based on all the candidate language information stored in the candidate language information storage unit 105c according to a predetermined rule. Then, the result is output to the display device 102 (step 308). Here, the predetermined rule means a judgment rule corresponding to the rule number designated by the operator from the judgment rules stored in the judgment rule storage unit 106c.

Specifically, the language determination unit 204 calculates a comprehensive evaluation value for each language by performing a calculation process on all the evaluation values obtained by the search of the feature dictionary 106b according to a predetermined rule, and calculates the largest overall evaluation value. The language name corresponding to the evaluation value is determined as the language name to which the input character string belongs. (Specific Example) Next, in the case where an input character string "school" is given, the process up to the final determination of one language name will be described with a focus on the processing by the language determination unit 204. explain.

The given input string is "school"
The rule for substring generation to be used is rule number “2” in the table shown in FIG. 4, and the decision rule to be used is shown in FIG.
Rule number “3” in the table shown in FIG. First, the input character string acquisition unit 201 acquires the input character string "school" from the input device 101 (step 301).

Next, the partial character string generation unit 202 causes the partial character string (“sc”, “sch”, “scho”, “scho”) of two or more characters starting from the first character “s”.
o "," school ") is generated (step 30)
3). Subsequently, the search unit 203 searches the feature dictionary 106b for each of the above five partial character strings, and acquires the corresponding candidate language information (steps 304 to 30).
6).

When the acquisition of the candidate language information for the partial character string starting with the first character "s" is completed by the above processing (steps 303 to 306), then the second character "c" is used. The same processing is repeated for the partial character string that starts (steps 303 to 306) In this manner, the same processing is repeated for all partial character strings generated from the input character string "school" (step 3).
02-307).

The language determination unit 204 follows the input character string "sc" according to the determination rule of rule number 3 in the table shown in FIG.
The language name of "tool" is determined (step 308). When determining the language, the language determining unit 204 first reproduces the input character string by using all the partial character strings generated by the partial character string generating unit 202 (Rule 2 referred to by Rule 3 in FIG. 7). Here, the reproduction of the input character string is 1
Alternatively, it means reassembling into the same character string as the input character string by connecting two or more partial character strings in series.
A combination of reproduced partial character strings is called a chain.

All chains reproduced by the language determination unit 204 are shown in the left column of the table of FIG. It can be seen that all chains are made up of partial character strings consisting of two or more characters. Here, the partial character strings forming the chain are connected by a hyphen ("-"). Also, for the substrings that make up these chains,
The candidate language information acquired by the search unit 203 is shown in the table of FIG. The data shown in the table of FIG. 9 is an example of candidate language information, and is unrelated to the CYSuen statistical information.

Next, the language determination unit 204 adds the evaluation values of the partial character strings for each language for each of the five types of chains shown in FIG. 8 (Rule 2 referred to by Rule 3 in FIG. 7), Calculate the overall evaluation value for each chain. In this case, the addition method is weighted addition according to the character length of each partial character string (provided in Rule 3 in FIG. 7). Specifically, the overall evaluation value of the chain "sc-ho-ol" for English is (English evaluation value of "sc") * (character length of "sc") + (English of "ho" Evaluation value) * (character length of "ho") + (evaluation value of English "ol") * (character length of "ol") = 0.1 * 2 + 0.2 * 2 + 0.2 * 2 = 1.0.

Similarly, the comprehensive evaluation values for French and German in this chain are "1.
8 "and" 2.2 ". The results of similar calculations performed for the other chains are shown in the table on the right side of FIG. Finally, the language determining unit 204 finds the maximum value from all the comprehensive evaluation values shown in FIG. 8 and determines the language name corresponding to the maximum value as the language name to be finally determined (FIG. 7). Rule 2) referred to by Rule 3 in. Since the maximum comprehensive evaluation value in the table of FIG. 8 is “4.8” for English, the language determination unit 204 causes the input character string “scho
The language name of “ol” is determined to be “English”, and the determination result is output to the display device 102.

(Other Configurations) The language determining apparatus according to the present invention has been described above based on the embodiment, but the present invention is not limited to this embodiment. That is, (1) In the present embodiment, the input character string acquisition unit 201 acquires the input character string from the input device 101 one by one, but the present invention is not limited to such an acquisition method. For example, a document may already be provided in the storage device, and the input character string acquisition unit 201 may acquire the words, phrases, clauses, or sentences constituting the document as one input character string.

Further, although the language determining section 204 outputs only the language name regarding the determination result to the display device 102, it is not limited to such information and output method. For example,
It is also possible to output not only the language name but also the comprehensive evaluation value as information, and also not only one type of language but also information about a language such as the second candidate. further,
These judgment results may be automatically delivered to a system different from the present invention, such as an automatic translation system. (2) In this embodiment, the external storage device 106 has the partial character string generation rule storage unit 106a, and one partial character string generation rule to be adopted is determined by the operator's selection. It is not limited to the configuration. For example, without the partial character string generation rule storage unit 106a,
It is also possible to have a simple configuration in which one type of fixed partial character string generation rule that seems to be optimal is provided inside the partial character string generation unit 202.

Similarly, the external storage device 106 may not have the determination rule storage unit 106c, but may have a simple structure in which one type of fixed determination rule is provided inside the language determination unit 204. (3) In the present embodiment, not only the language name but also the evaluation value is stored as the candidate language information of the feature dictionary 106b, but it is not limited to such a configuration. It is also possible to have a simple configuration in which only the character string and the language name to which the character string may belong are registered. This case corresponds to the case where all the evaluation values registered in the feature dictionary 106b in the present embodiment are the same value, but the effort for creating the feature dictionary 106b is reduced by such a simple configuration. There is an advantage.

Further, in this embodiment, one type of feature dictionary 1
Although 06b is used, it may be composed of a plurality of feature dictionaries. For example, by creating a special feature dictionary for each specific field such as electricity or chemistry, and limiting the search to only the feature dictionary that corresponds to the specific field to which the input character string belongs, it is possible to perform accurate search in a shorter time. A language determination device that can perform language determination can also be used.

[0040]

As is apparent from the above description, according to the language determining apparatus of the first aspect, a combination of various character sequences forming a partial character string, that is, an input character string to be subjected to language determination is combined. Each time, the information necessary for the language determination is obtained, and the language determination is performed based on the information. Therefore, as compared with the conventional language determination device that determines the language by using only one character that constitutes the input character string, this device enables more accurate determination based on the information regarding the language determination.

Further, the information obtained for each partial character string includes not only the language name to which the partial character string belongs but also the numerical value indicating the possibility, so that it is based on the information of only the language name. It is possible to perform more detailed and highly accurate language determination as compared with a conventional language determination device. According to the language determination apparatus of claim 2, the language determination apparatus of claim 1 further comprises a plurality of partial character string generation rules, and in language determination, one selected from the plurality of rules is selected. Substrings are generated according to one rule.

Therefore, by selecting an appropriate partial character string generation rule in advance, it is possible to restrict the generated partial character string from exceeding the range of the character string group described in the feature dictionary. become. As a result, it is possible to perform flexible language determination according to the contents of the feature dictionary included in this device. Further, by limiting the number of partial character strings to be generated, the processing time can be changed so that the language determination processing ends within the requested processing time range.

According to the language determining apparatus of claim 3, the language determining apparatus of claim 1 or 2 is further provided with a plurality of determination rules, and when determining the language, one of the plurality of rules is selected. The language name is determined according to the one rule. Therefore, depending on the specialized field in which the input character string is used, the linguistic characteristics of the candidate language itself, the hit rate in the past language judgment performed by this device, etc.
You can select the most appropriate decision rule. This makes it possible to make more accurate language determination and improve the accuracy of language determination.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a language determination device according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a functional configuration of the device according to the embodiment.

FIG. 3 is a flowchart showing an operation procedure of the language determination device according to the embodiment.

FIG. 4 is a partial character string generation rule storage unit 10 according to the embodiment.
It is a table which shows the content and the specific example of 6a.

FIG. 5A is a diagram for explaining the concept of an input character string. FIG. 5B is a diagram for explaining the concept of the partial character string.

FIG. 6 is a table showing the contents of a feature dictionary according to the embodiment.

FIG. 7 is a table showing the contents of a determination rule storage unit according to the embodiment.

FIG. 8 is a table showing a chain reproduced from a partial character string of an input character string “school” and an evaluation value for each chain.

FIG. 9 is a table showing candidate language information about a partial character string of an input character string “school”.

FIG. 10 is a flowchart showing an operation procedure of a conventional language determination device.

[Brief description of reference numerals]

 106a Partial character string generation rule storage unit 106b Feature dictionary 106c Judgment rule storage unit 201 Input character string acquisition unit 202 Partial character string generation unit 203 Search unit 204 Language determination unit

Claims (3)

[Claims] 1. A language determination device for determining a language name to which a given input character string belongs, which is a partial character string generation means for generating a partial character string consisting of part or all of the character strings of the input character string. And a character string group consisting of the same number of characters as the partial character string, a feature dictionary storing candidate language information consisting of a language name to which each character string may belong and an evaluation value indicating the possibility, Searching means for searching a character string stored in the feature dictionary to obtain candidate language information corresponding to a character string matching the partial character string; and the input character based on the candidate language information obtained as a result of the search. A language determining device, comprising: a language determining unit that determines a language name of a string. 2. The language determination device further includes a partial character string generation rule storage unit that stores a plurality of rules for generating a partial character string in advance, wherein the partial character string generation unit includes the partial character string. 1 selected from a plurality of rules stored in the column generation rule storage means
The language determination device according to claim 1, wherein the partial character string is generated according to one rule. 3. The language determination device further comprises determination rule storage means for storing a plurality of rules for determining a language name in advance, and the language determination means is stored in the determination rule storage means. 3. The language determination device according to claim 1, wherein the language name of the input character string is determined according to one rule selected from a plurality of rules.