JP2000148754A - Multilingual system, multilingual processing method, and medium storing program for multilingual processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To identify the language and coding system of a character code and to add the identification result to a character code string and output the obtained string. SOLUTION: An appearance probability table wherein the appearance probability of a character code is described for each character is prepared for every combination of languages and character code systems. An inputted character code string is divided into individual characters (step 21) and the appearance probability of the character code is obtained by referring to the appearance probability table (steps 23, 25, 27, 29, and 31). The product of appearance probability values is calculated for every combination of the languages and character code systems (steps 24, 26, 28, 30, and 32) and the combination of the languages and character code systems as to the input character string is judged from the obtained product. Information representing this judgement result is added to the character code string and outputted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multilingual system, a multilingual processing method, and a recording medium storing a program for multilingual processing that handle a plurality of languages, and more particularly to a character code string (encoded text data, keyword Language) and the type of character code (character code system)
TECHNICAL FIELD The present invention relates to a system having a function of judging the number, a method for realizing the function, and a program recording medium.

[0002]

BACKGROUND OF THE INVENTION Currently, Japan, China (People's Republic of China),
The character code for kanji (or Hangul) used in South Korea and Taiwan (Republic of China) expresses one character with two bytes. These character codes (systems) are defined independently for each language (Japanese, Chinese, Korean, etc.). If the encoding method (character code system, code type or encoding rule) is different, characters in the same language are represented by different character codes. Information representing a language is not usually added to character code data. For this reason, when a series of character codes are given, it is not easy to know in which language the character codes are encoded and obtained.

A language information processing system such as a database search system, a translation system, a speech synthesis system, and the like is created on the premise of a specific language and a character code system. Even when considering a language information processing system that can be used in a plurality of languages, it is necessary to know the language of a given keyword or text data because language information is different for each language type. Appropriate processing cannot be expected if the given keyword, text data language and character code system are unknown.

[0004]

DISCLOSURE OF THE INVENTION The present invention makes it possible to identify the language of a given character code string and its character code system, so that even if the language and character code system of an input keyword or text data are unknown, An object of the present invention is to enable various language information processing suitable for each language.

In the multilingual system according to the present invention, an input means for receiving an input character code string (encoded input text data, a keyword, etc.), a character code in each combination of a language and a character code system, Storage means for storing a plurality of occurrence probability tables each describing an occurrence probability; reading out the occurrence probabilities from the plurality of occurrence probability tables for one or a plurality of character codes included in the input character code string received by the input means; Means for obtaining evaluation data for each combination of language and character code system, means for determining the combination of the language and character code system of the input character code string based on the obtained evaluation data, and Information representing the combination of the determined language and the character code system is In which comprises an output means for outputting with a column.

In the multilingual processing method according to the present invention, for each combination of a language and a character code system, an appearance probability table describing the probability of occurrence of a character code in the combination is created in advance, and an input character code string is created. Is received, the appearance probabilities are read out from the plurality of occurrence probability tables for one or more character codes included in the received input character code string, and evaluation data is obtained for each combination of language and character code system. Based on the evaluation data, a combination of the language of the input character code string and the character code system is determined, and information representing the combination of the determined language and the character code system is output together with the character code string.

[0007] The present invention further provides a recording medium storing a program for executing the above method. This recording medium is used for each combination of language and character code system.
An input character code string is received to identify the combination of the language of the input character code string and the character code system using an appearance probability table describing the probability of occurrence of the character code in the combination, and the received input character code The appearance probabilities are read from the plurality of appearance probability tables for one or a plurality of character codes included in the column, evaluation data is obtained for each combination of language and character code system, and input is performed based on the obtained evaluation data. A program for controlling a computer to determine a combination of a language and a character code system of a character code string and to output information indicating the determined combination of the language and the character code system together with the character code string. .

Preferably, the appearance probability table is further stored in the recording medium. The recording medium refers to a magnetic disk storage device, a magneto-optical disk storage device, an optical disk storage device, a magnetic tape, a semiconductor memory, or the like.

The appearance probability of a character code depends on the combination of the language of the character represented by the character code and the character code system. Even with the same character code, the appearance probability of the character code differs for each language. Also, even in the same language, the appearance probability of the same character code is different if the character code system is different. The present invention discriminates a language represented by a character code and a type of the character code system by paying attention to an appearance probability of a character code unique to a combination of a language and a character code system. Here, character codes are numbers,
Codes that represent the alphabet, kanji, hiragana, katakana, hangul, and all other characters, symbols, and signs.

According to the present invention, the appearance probability is read from the appearance probability table for each character code of the received input character code string (text data, keyword, etc.), and the evaluation data is a combination of a language and a character code system. It is created for each. If the evaluation data related to the appearance probability is low, it is determined that the input character code string is not likely to be related to the combination of the language and the character code system, and if the evaluation data is high, the input character code string is It is highly likely that it is related to the combination of the language and the character code system. In this way, the combination of the language of the input character code string and the character code system is determined based on the evaluation data.

From the viewpoint of accuracy, it is preferable to calculate the product of the appearance probabilities read from the appearance probability table and determine the language and character code system (encoding method) of the input character code string based on the calculated values. . If the appearance probability of any one of the character codes is 0 or a value very close to 0, the product is also a very small value, and such a combination of the language and the character code system is clearly excluded.

Further, according to the present invention, information indicating the combination of the determined language and the character code system is output together with the character code string (all or part of the input character code string). Therefore, a device that uses this output data,
The system can recognize the language type and character code system of the character code string.

The present invention provides, for example, a multilingual morphological analysis system, a multilingual search system, a multilingual output (eg, print) system, a multilingual translation system, a multilingual word processor, a multilingual speech synthesis system, and a multilingual database system. , Multilingual processing systems such as multilingual document management systems. In these multilingual processing systems,
When processing character code strings (text data)
Using information representing a combination of a language and a character code system, processing suitable for the combination of the language and the character code system can be performed.

For example, a multilingual search system is provided with search means suitable for each of a plurality of languages. Based on information representing a combination of a language and a character code system provided from the output means, a search means suitable for at least the language represented by the information is selected, and a search process is performed in accordance with a character code string provided from the output means. Is executed.

[0015] In the multilingual database system, the character code string is added to the character code string provided from the output means, while the character code string is added to the character code string provided from the output means. Registered in.

In the multilingual document management system, a plurality of document processing means suitable for at least a plurality of languages are provided. Document processing means suitable for at least the language represented by the information is selected based on the information representing the combination of the language and the character code system given from the output means, and the document processing of the character code string given from the output means is performed. Is executed.

The input character code string accepting means may be an input device such as a keyboard or a communication device (including software) for receiving transmitted data. Similarly, the output means may directly control a processing system (retrieval system, database, document management system) or the like, or may be a communication device for transmitting a character code string and identification information. Therefore, a system for language identification and a system for processing character code strings based on the result of language identification can be connected by a network. That is, the multilingual processing system is realized on a network.

[0018]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows the appearance of a multilingual system having a function of character code language identification, and FIG. 2 shows an outline of the electrical configuration.

A multilingual system having a function of character language identification includes a computer 10. The computer 10 is connected to a CRT display (or liquid crystal display panel) 11, a printer 12, and input devices (keyboard 13A and mouse 13B). FD drive 14 and CD-RO inside computer 10
An M drive 15 and an HD unit 16 are provided. The FD drive 14 writes data to an FD (floppy disk) 19 and reads data from the FD 19. The CD-ROM drive 15 is
D-ROM (Compact Disc-Read Only)
Data from the memory (18). The HD unit 16 writes data to an HD (hard disk) (not shown) and reads data from the HD. The computer 10 further includes an internal memory (such as a semiconductor memory) 17. The computer 10 can further communicate with other systems via a network according to a communication control program.

The CD-ROM 18 includes a multi-lingual program including a language identification program for character code language identification.
Stores occurrence probability data used for program and language identification. The contents are shown in FIG. The appearance probability data represents the probability that a character code (ie, a character) appears. The appearance probability for each character is obtained in advance by statistically processing characters appearing in various past documents. The appearance probability data may be obtained only for representative characters, or may be obtained for all characters. The appearance probability data is stored in the form of a table for each combination of a language and a character code system (character code type or encoding method). In this embodiment, a table (probability table) for Chinese (for continent and Taiwan) and a table for Japanese (E
UC (Extended UNIX Code) code and Shift-JIS
(Japanese Industrial Standards) code) and a table for Korean. Japanese is generally encoded by EUC code or Shift-JIS code. The EUC code or Shift-JIS code is a character code system or an encoding method. Therefore, not only the language type but also the character code system can be identified. The same applies to languages other than Japanese. However,
Since the Japanese Shift-JIS code can be converted into a Japanese EUC code, only an EUC code table may be provided for Japanese as in an example of language identification processing described later.

The programs and data stored in the CD-ROM 18 are stored in the CD-ROM when the system is started up.
The data is read from the ROM 18 and stored in the HD. In the multilingual processing including the language identification processing of the character codes, some of these programs and data are temporarily stored or expanded in the internal memory 17 as necessary.

FIG. 4 conceptually shows the relationship between input processing (program), language identification processing (program), output processing (program), and appearance probability data of text data (character code string) to be processed. Is shown. All of these programs are called multilingual programs.

The text input processing software such as a browser and communication software (or a keyboard 1)
Text data input (via 3A) is cut out character by character by a language identification program, and for each cut out character, a combination of language (Japanese, Chinese, Korean) and character code system The appearance probability data is obtained for each of the cases. The obtained appearance probability is subjected to statistical processing (multiplication processing described later) in the evaluation value work area (part of the internal memory 17), and finally the combination of the language type of the input text data and the character code system is identified. Is done. Information representing the identification result is added to the input text data,
Output (for example, given or transmitted to another processing system).

FIG. 4 is a functional block diagram showing a multilingual program and data used for language identification processing, while FIG. 5 is a flowchart showing the multilingual program. When the input text data (character code string) is received (step 41), the combination of the language of the text data and the character code system is identified (step 42).
Finally, the identification result is added to the text data and output (step 43).

FIG. 6 is a flowchart showing the procedure of the character code identification process (step 42 in FIG. 5). In this processing, a character code string representing a phrase "plum blossoms" is used (for example, this character code string is 0xC7DF, 0xB2D6, 0xA4CB, 0xB in Japanese EUC code).
0x in Japanese Shift-JIS code represented as 2A9
947E, 0x89D4, 0x82C9, 0x89A7
It is expressed as Here, 0x indicates a hexadecimal number. ) Is input from the keyboard, an example will be described in which the character code string identifies a combination of a language and a character code system. FIG. 7 shows the appearance probabilities of the character codes constituting this phrase "Ume ni ni Ugui" for each combination of language and character code system. In each appearance probability table, the maximum occurrence probability is normalized to 100%. Values are shown.

Data of 2 bytes (one character) (0xC7DF or the like in the above example) is extracted from the input text data (step 21).

The appearance probabilities corresponding to the extracted 2-byte data (character codes) are read from the appearance probability table for each combination of language and character code system (steps 23, 25, 29, 31). On the other hand, the extracted 2-byte data is converted from the Shift-JIS code to the EUC code (step 22), and the appearance probability of the character code by the converted EUC code is represented by the Japanese appearance probability table (EUC code). (Step 2
7).

The character code of the first character "ume" of "plum blossoms" is 0.0948% in Japanese EUC code system, 0% in Japanese Shift-JIS code system, and EU of Chinese (continent).
0.0129% for C code system, 0.0022% for Chinese (Taiwan) Big5 code system, 10.9 for Korean EUC code system
Has a 41% appearance probability.

The product of the read appearance probability and the already calculated evaluation value is calculated, and this product is used as a new evaluation value (evaluation value update) (steps 24, 26, 28, and 3).
0, 32). This calculation is also performed for each combination of the language and the character code system (that is, for each appearance probability table).
1 is set as the initial value of the evaluation value. In the case of the first character code, the read appearance probability is multiplied by 1.

The other evaluation values are normalized with the evaluation value having the maximum value among the evaluation values updated in this way as 100 (step 33). This corresponds to step 35 described later.
This is for performing a comparison process with the threshold value.

If the above processing has not been completed for all the character codes constituting the input text data (step 34), the sum of all the evaluation values except the evaluation value having the maximum value is calculated. You. If the calculated total value is equal to or less than a predetermined threshold value (YE
S), the input text data is determined to be a combination of a language and a character code system that gives an evaluation value having the maximum value. If the calculated total value exceeds a predetermined threshold value (NO in step 35), the processes in steps 21 to 34 are repeated.

Regarding the character code of the second character “flower” of “plum blossoms”, the appearance probability in the Japanese EUC code is
3.2740%, the probability of occurrence of Japanese Shift-JIS code is 0
%, The probability of appearance in the EUC code of Chinese (continent)
0.1118%, the appearance probability in the Chinese (Taiwan) Big5 code is 0.2874%, and the appearance probability in the Korean EUC code is 0%.

The appearance probability of the character code of the third character “Ni” of “Plum blossoms” is 59.155% in the Japanese EUC code,
0% for Japanese Shift-JIS code, 0.0001% for Chinese (continent) EUC code, Big5 for Chinese (Taiwan)
The code is 0%, and the Korean EUC code is 0.0001%.

The appearance probability of the character code of the fourth character “Ugui” of “Plum blossoms” is 0.0001% in Japanese EUC code,
0% for Japanese Shift-JIS code, 0.3717% for Chinese (continent) EUC code, Big5 for Chinese (Taiwan)
0.0048% for code, 0.0299 for EUC code in Korean
%.

The product of the appearance probabilities of the character codes for the four characters that make up "Umebana to Ugui" is obtained as the final evaluation value.
This means that the above processing has been completed for all the character codes constituting the input text data (YES in step 34). The final evaluation value is 0.000000001836% for Japanese EUC code, 0% for Japanese Shift-JIS code, and 0.000000 for Chinese (continent) EUC code.
000005366%, 0 for Big5 code in China (Taiwan)
% And 0% for the EUC code of Korea. When these values are compared, since the evaluation value in the Japanese EUC code is the largest, it is determined that “Plum blossoms” is expressed in Japanese and in the EUC code. In this way, the language of the character represented by the character code and the type of the character code (character code system or encoding method) are identified. When the input text data includes a large number of character codes, normally, when the processing for three to four characters is completed, YES is obtained in step 35, and the combination of the language of the input text data and the character code system is set. The decision will be over.

FIG. 8 shows the overall configuration of the multilingual search system. Multilingual system 5
1 corresponds to the system described with reference to FIGS.

The multilingual system 51 receives, for example, the following questions. The language of this question (% C7 to% A9 part; this is the character code of "Plum blossoms" described above) is unknown.

/ Cgi-bin / recognize? Query =% C7% DF% B2% D6% A4% CB% B2% A9

Cgi-bin indicates that an execution command follows. Is the execution command recognize after that? This is a command for identifying the language and character code system of the subsequent character code string (search parameter) and adding the identification result to the search parameter.

The multilingual system 51 executes this instruction, It recognizes that the subsequent character string is Japanese represented by the EUC code. The multilingual system 51 creates and outputs a question to which the following language parameter is added.

/ Cgi-bin / search? Query =% C7% DF% B2% D6% A4%
CB% B2% A9 & lang = ja & encode = EUC

"Search" is a command for searching.
lang = ja & encode = EUC is a language parameter, the language type is Japanese (ja) (compliant with ISO-639), and the code system is E
It is a UC code.

Such a question with language parameter is transmitted to the search system 52 via a network (for example, the Internet). The search system decomposes the received query with language parameter into a search sentence part and a language parameter part.

Query sentence query =% C7% DF% B2% D6% A4% CB% B2% A9 Language parameter lang = ja & encode = EUC

The search system 52 selects an index specified by a language parameter, that is, the Japanese index 53, from the Japanese index 53, the Chinese index 54, and the Korean index 55, and performs a search process according to the search sentence. Execute Search results are sent from the search system 52 to the multilingual system 5
1 is notified.

It is preferable that the various indexes 53 to 55 be provided for each combination of the language type and the code system. However, if the language type is known, the code system can be converted (for example, as described above). Shift-JIS and E
It is only necessary to provide conversion for each language type.

FIG. 9 shows an example of a multilingual database system. The multilingual system 61 receives text data whose language type and character code are unknown, and determines the language type and character code. A language tag (representing the combination of the determined language type and character code) is added to the text data and registered in the database 62.

FIG. 10 shows an example of the database. This is a relational database with language tags added to each document. The language tag is the language (e
n, ja, fr, zh etc.) and code system (ASCII, EUC, I
SO-8859-1).

As described above, by adding a language tag to the data registered in the database, only a document in a specific language can be extracted, a full-text search can be performed only on a document in a specific language, and a document in a specific language can be retrieved. It is possible to perform processing for each language (including a combination with a character code, if necessary), such as creating only an abstract.

FIG. 11 shows an example of a document management system. The document management system 72 provides, for each language type,
Summary generation program 73, keyword search program 7
4, a grammar / spell check program 75 and the like. According to the text data with a language tag provided from the multilingual system 71, a program suitable for the language represented by the language tag is selected, and processing such as summary generation, keyword search, grammar and spell check according to the language is performed. Done.

[Brief description of the drawings]

FIG. 1 shows the appearance of a multilingual system having a language identification function for character codes.

FIG. 2 shows an outline of an electrical configuration of the multilingual system.

FIG. 3 shows a data structure of a recording medium used in the multilingual system.

FIG. 4 is a block diagram functionally showing a mutual relationship among an input process, a language identification process, an output process, and appearance probability data. .

FIG. 5 is a flowchart showing an overall flow of a multilingual program.

FIG. 6 is a flowchart illustrating a procedure of a language identification process of a character code;

FIG. 7 shows an appearance probability table in which the appearance probabilities of character codes representing predetermined phrases are stored in correspondence with languages and character codes.

FIG. 8 is a block diagram showing the entire multilingual search system.

FIG. 9 is a block diagram showing the entire multilingual database system.

FIG. 10 shows an example of a database.

FIG. 11 is a block diagram showing the entire multilingual document management system.

[Description of Signs] 10 Computer 13A Keyboard 15 CD-ROM Drive 51, 61, 71 Multilingual System 52 Search System 62 Database 72 Document Management System

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Hidenobu Kanaoka 10 Odron, Hanazono Todocho, Ukyo-ku, Kyoto, Kyoto Prefecture F term in the corporation (reference) 5B009 KA04 QA01 VB01 5B091 AA01 CB01 CC01

Claims (7)

[Claims] An input means for receiving an input character code string,
For each combination of a language and a character code system, storage means for storing a plurality of appearance probability tables each describing the probability of occurrence of a character code in the combination, one of the input character code strings received by the input means, Means for reading out the appearance probabilities from the plurality of appearance probability tables for a plurality of character codes, obtaining evaluation data for each combination of language and character code system, and determining the language of the input character code string based on the obtained evaluation data Means for determining the combination of the language and the character code system, and output means for outputting information indicating the combination of the language and the character code system determined by the determination means together with the character code string;
Multilingual system with. 2. A search device suitable for at least a language represented by the information based on information provided from the output device and representing a combination of a language and a character code system, the search device comprising: 2. The multilingual system according to claim 1, further comprising a search system for selecting a unit and executing a search process according to a character code string provided from said output unit. 3. A data base for registering a character code string in a state where information representing a combination of a language and a character code system given from the output means is added to a character code string given from the output means. The multilingual system of claim 1. 4. A plurality of document processing means suitable for at least a plurality of languages, and based on information provided from the output means and representing a combination of a language and a character code system, at least a document processing means suitable for the language represented by the information. 2. The multilingual system according to claim 1, further comprising a document management system that selects a document processing unit and executes a document process on the character code string provided from the output unit. 5. For each combination of a language and a character code system,
An appearance probability table describing the probability of occurrence of a character code in the combination is prepared in advance, an input character code string is accepted, and a plurality of occurrence codes are obtained for one or more character codes included in the received input character code string. The appearance probabilities are read out from the probability tables, evaluation data is obtained for each combination of language and character code system, and the combination of the language of the input character code string and the character code system is determined based on the obtained evaluation data. A multilingual processing method for outputting information indicating a combination of a determined language and a character code system together with a character code string. 6. For each combination of a language and a character code system,
An input character code string is received to identify the combination of the language of the input character code string and the character code system using an appearance probability table describing the probability of occurrence of the character code in the combination, and the received input character code The appearance probabilities are read from the plurality of appearance probability tables for one or a plurality of character codes included in the column, evaluation data is obtained for each combination of language and character code system, and input is performed based on the obtained evaluation data. A recording medium storing a program for controlling a computer to determine a combination of a language of a character code string and a character code system, and to output information representing the determined combination of the language and the character code system together with the character code string. 7. The recording medium according to claim 6, wherein said appearance probability table is further stored.