KR20010098422A - Method to convert unicode text to mixed codepages - Google Patents

Abstract

The present invention relates to a method and system for converting a source string encoded according to the UNICODE standard into a target string to be encoded according to mixed code pages.

Associate a predetermined priority with each of the sub-code pages 14, 15, 16, 17, and assign the target character and the target character to any one of a plurality of sub-code pages 14, 15, 16, 17; Strictly convert the characters according to the priority sequence without using a mapping table to find out if its encoding is stored. Preferably, sub-codepage 14 containing the most frequently used characters is associated with the highest priority and sub-codepage 17 with the least frequently used characters is associated with the lowest priority. (Fig. 1).

Description

How to convert a source string to a target string, its computer system and program product {METHOD TO CONVERT UNICODE TEXT TO MIXED CODEPAGES}

The present invention relates to a system and method for converting between character codes associated with computer-readable characters. In particular, the present invention relates to a method and system for converting a source string encoded according to a Unicode standard into a targetstring to be encoded according to a mixed code page.

Computers and other electronic devices typically use text to interact with a user. Text is typically displayed on a monitor and some other display device. Because text must be represented in digital form within a computer or other electronic device, character set encoding must be used. Generally speaking, character set encoding is operated to encode each character of the character set with a unique digital representation. Characters (encoded) correspond to letters, numbers and various text symbols. They are assigned numeric codes for use by a computer or other electronic device. The most popular character used in computers and other electronic devices is the American Standard Code for Information Exchange (ASCII). ASCII uses 7-bit sequences for its coding. In other countries, different character sets are used. In Europe, the dominant character encoding standard is the ISO 8859-X family, in particular ISO 8859-1 (called "Latin-1") developed by the International Standards Organization (ISO). For Japan, the dominant character encoding standard is JIS X0208, where JIS refers to the Japanese information standard, which was developed by the Japan Standards Association (JSA). Examples of other existing character sets include Mac ^TM OS Standard Roman encoding (by Apple Computer, Inc.), Shift-JIS (Japan), Big5 (Taiwan), and the like.

The character set described above is stored in a so-called codepage, which is a kind of table representing the coding of each character of the character sets. Therefore, for each character, the numerical code associated with it is given such that a unique mapping exists between the two. Most code pages associate a one-byte long numeric code for each character. But more than that, for example, there are code pages that have numeric codes that are at least two bytes or three bytes long. Code pages that contain characters having the same code length are called simple code pages.

In order to better accommodate the complexity of individual language-specific national requirements, so-called mixed code pages also exist. A hybrid code page consists of at least two sub-code pages whose coding may differ in length. The sub-code page is also called a code set. They are numbered from 0 to 3. For example, the mixed Japanese codepage IBM-33722 is codeset IBM-953 (1 byte, codeset 0), IBM-952 (2 bytes, codeset 1), codepage IBM -896 (escape character 8E + 1 byte, codeset 2) and IBM-953 (escape character 8F + 2 bytes, codeset 3).

Due to the ever-increasing globalization of businesses and networks and the increasing influence of the Internet connecting all countries around the world, any data conversion between computers using different kinds of code pages should be as fast as possible. The choice should be as simple as possible.

In order to simplify the code conversion, so-called Unicode standards have been developed and internationally recognized. Unicode provides a single scheme that represents all existing code sets. The design of the Unicode encoding scheme is independent of the design of the basic text processing algorithm, except for directionality. The Unicode implementation is assumed to include some suitable text processing and / or rendering algorithms. Any characters encoded according to Unicode standardization are represented as 2-byte long numeric codes.

The problem now is to obtain a very efficient way of converting from the Unicode standard to the hybrid code page described above, ie the source string is represented by the Unicode standard and this is very simply and quickly a plurality of code pages, for example It is desirable to convert to a code system including a plurality of four code pages as described above.

A prior art conversion method for converting from Unicode to a plurality of code pages is shown in US Pat. No. 5,793,381. The transcoding system maps a single source character or character sequence to a single target character or target character sequence by looking up the location of the associated target character in the mapping table. When reading the source character the mapping table is accessed to determine which of the sub-code pages can be used for code conversion. The particular sub-codepage to be continued for code conversion is retrieved until a source character is retrieved from an input character string that cannot be converted to the sub-codepage. In this case, the auxiliary mapping table is accessed again to retrieve the correct sub-codepage. Additionally, the prior art code conversion system provides fallback mapping for text elements when the look-up handler cannot identify one or more characters in the target encoding for the text elements. A fallback handling that operates with a mapping table to identify one or more characters in the target encoding that can be used as. However, prior art approaches use additional lookup tables that make it slower and more complicated than necessary.

It is therefore an object of the present invention to provide a method and system for code conversion from Unicode text to mixed code pages that can be executed with better performance.

These objects of the invention are achieved by the features described in the appended independent claims. Subsequently, other advantageous arrangements and embodiments of the invention are described in the respective subclaims.

In summary, the basic concept of the present invention relates to a predetermined priority and to each sub-code page, and to which sub-code page of the plurality of sub-code pages the target character and its encoding are stored. Strictly convert characters according to the priority sequence without using a mapping table to find out. Preferably, the sub-codepage containing the most frequently used characters is associated with the highest priority and the sub-codepage with the least frequently used characters is associated with the lowest priority. Therefore, in the case of four sub-code pages, a priority sequence between the sub-code pages can be set. Each priority is a measure of the probability of searching for a particular character in each sub-codepage.

In addition to this basic approach, it is also proposed to access the sub-code page with the highest priority that has not yet been accessed for that character if the character is not retrieved in a particular sub-code page. Applying the manner of the present invention described above has the following advantages.

First, the performance is significantly increased compared to the prior art level conversion method described above because there is no separate mapping table that must be accessed whenever a character is not retrieved in the currently used sub-codepage. .

Second, the secondary mapping table does not need to be created at all.

Third, a priority sequence given to a plurality of sub-codepages can be set such that country-specific knowledge about that language is available. Therefore, the conversion method of the present invention is easily applicable to special circumstances added according to a constant, country-specific code page system.

In contrast, in each individual case, the above-described priority sequence is used to adapt the code conversion of the present invention to the designation requirements imposed by the designation text to be converted when it is known in advance that the text does not represent an average. It can be changed dynamically from standard settings to individual settings before running the transformation. The new priority sequence may for example be given in the header of the file to be converted.

Another significant advantage of the present invention is that it provides an easy concept to take advantage of the special advantages of modern computer systems that hardware instructions can be used to process multiple characters instead of only one character at a time. will be. These modern hardware instructions require a linear table to look up the target character without additional checking access to any kind of mapping table.

The present invention can preferably be used on the Internet when any code conversion is needed. In addition, the tools of the present invention can be employed in database applications when some of the content of the database is converted from Unicode text to mixed code pages.

When the method of the present invention is applied when the probability of searching for a specific character in one of a plurality of sub-codepages is the same as that in which all sub-codepages can be searched, when four sub-codepages exist Only a statistical average of two additional accesses is needed. This value is reduced to 1.5 for three sub-code pages and to 1 for two sub-code pages. If 70% of all characters are Japanese EUC-tables searched in the remaining code sets 2 and 3, 30% are searched in codeset 0 and less than 1% are searched in the remaining code sets 2 and 3. The mean value is slightly larger than 1.

In addition, the present invention is preferably at least partly employed in hardware implementations that are burnt-in directly into hardware chips. This chip means then comprises hardware circuitry for implementing and reflecting at least some of the code conversion method steps of the present invention. In view of their increasingly increasing function range, which includes the diversity and increasingly technical features of the ever-growing telecommunications devices, such chips may later be used in a variety of devices. In view of the devices available today, such devices can be preferably used in any device that forms part of certain international communications. For example, certain network types, such as the Internet, TV or radio receiving devices, in particular digital TV or radio, set-top boxes for mobile phones, certain hand-held computing and / or remote Routers in communication device types and the like have an input interface that handles any foreign-language data.

The invention is illustrated by way of example and is not limited by the shape of the drawings in the accompanying drawings.

1 is a schematic logic diagram illustrating the basic elements of the method of the present invention;

2 is a schematic diagram of an arbitrarily selected embodiment showing for each of a plurality of 230 source characters from which a plurality of four sub-code pages may be retrieved, each character being searched for;

3 is a logic diagram illustrating a codeset access sequence when the method is applied in accordance with a preferred embodiment of the present invention during code conversion.

Explanation of symbols for the main parts of the drawings

10: Unicode 12: Priority Rule

14,15,16,17: code set

Overall, with reference now to FIG. 1 in particular, the entire Unicode characters in box 10 are represented symbolically under the inventive transformation method.

According to a preferred embodiment of the method of the present invention, any priority rules for establishing a predetermined well-defined priority sequence between a plurality of sub-codepages used. 12 is established. The term 'codeset n'-n is an integer-is used herein to mean the same as the term 'sub-codepage n'. As shown in Fig. 1, four sub-code pages are shown, which are codesets 1 and 14, codesets 0 and 15, codesets 2 and 16 and codesets 3 and 17. In box 10 four exemplary selected characters are shown and their encodings are each located in different separate sub-codepages as shown on the right side of FIG. As shown in each of the tables 10, 14, 15, 16 and 17 from the figure, numerical codes are stored for each character.

Referring now to FIGS. 2 and 3, a preferred embodiment of the present invention will be described in more detail in an example code conversion from Japanese UNICODE to a mixed Japanese EUC sub-codepage.

Prior to starting the code conversion, the existing estimate of the Japanese EUC sub-codepage is that in this particular case the sub-codepage has codeset 1 containing approximately 70% of all occurrence source characters, Including approximately 29%, codeset 2 comprising approximately 0.6% and codeset 3 being configured to comprise approximately 0.4% of the total occurrence characters. The codeset probability distribution is shown in FIG. 1, where the most frequently used codeset 14 is shown first and the rarest used codeset is shown as the last set 17 of codesets' stacks. .

Codeset 1, codeset 0, codeset 2, codeset 3.

In FIG. 2, a schematic diagram of a randomly selected embodiment shows for each of a plurality of 230 source characters where each character can be retrieved from among a plurality of four sub-code pages.

All of the 230 source characters are translated in a single example conversion process. In order to improve the clarity of the process, it should be understood that the 230 numbers are chosen very small.

Thus, all 230 source characters consist of an input set symbolically represented with reference to reference numeral 10 in FIG. 1. The new numeric codes that need to be issued by the method of the present invention are stored in four sub-code pages 14, 15, 16 and 16, referring to the right side of FIG. 1 as follows.

Codeset 1 contains characters 1 through 171,

Codeset 2 contains the characters 172,173,

Again, codeset 1 contains the characters 174-196,

A very rarely used character, character 197, is located in codeset 3.

Again, codeset 1 contains characters 198 through 210,

Characters 211 through 215 are stored in codeset 0,

Very rarely used codeset 2 has the characters 216,217 and

Codeset 1 contains the characters 218-230.

The conversion scheme processes the source characters described above sequentially. In a preferred method of applying the method of the present invention, a hardware instruction capable of processing a plurality of characters at a time may be used. An example of this is the IBM OS / 390 hardware instruction "Translate Two to One", which is abbreviated as TRTO, which converts a string of 2-byte characters into an output buffer containing characters as 1-byte characters. The hardware instruction has the following variables.

The string to be converted,

The target buffer where the converted string is to be stored,

Character that indicates that certain input characters cannot be converted,

A conversion table addressed as the character to be converted, where the converted character is stored in its addressed location.

However, for the sake of clarity and for the purpose of the present invention, the above-described input character sequence is submitted to a single-character conversion process, that is, a process for processing each character separately.

According to a preferred feature of this embodiment, a set of processing rules derived from the above-described priority sequence is set. The processing rule is as follows.

1. Access the highest priority code set first.

2, when a particular character is not retrieved from the highest priority code set, then proceeds to the code set with a lower priority than the next.

3, if a character is not found in the codeset, accesses the codeset with the highest priority that has not yet been accessed for this character.

Applying these rules generates the scheme given in FIG.

3 includes rows. The first row reflects the sequence of sub-codepage 1 to be subsequently accessed if a particular character is not retrieved when starting access in sub-codepage 1, ie the sub-codepage with the highest priority. Thus, when a character is not retrieved in sub-codepage 1, codepage 0 will be accessed to retrieve the current character. If the current character is retrieved in sub-codepage 0, the process continues with the next character to be converted in the sub-codepage. In the case of this next character, the second line will be applied for the search. Alternatively, if the current character described above is not searched in sub-codepage 0, then sub-codepage 2 will be accessed for another search later. Subsequently, the countermeasure will follow for sub-code 2.

If the current character is retrieved in sub-codepage 2, the associated numeric code, i.e. the converted code, is issued and the next character will be retrieved according to the third column shown in FIG. Otherwise, ie, if the current character is not retrieved in sub-codepage 2, the last sub-codepage 3 will be accessed for the search. The text will then be searched and the search proceeds to the fourth column shown in FIG.

As can be appreciated from the above details according to this embodiment of the present invention, the search always proceeds in the particular sub-codepage where the last character has been successfully detected.

In particular, referring to the second, third and fourth lines, codepage 1, shown here by reference numeral 14 in FIG. 1, is typically used when the next character is not retrieved in each current sub-codepage. Accessed for the next character to be converted.

In particular, referring to the character string illustrated in FIG. 2, a method of processing, i.e., accessing another sub-codepage will now be described in more detail. The arrows of FIGS. 2 and 3 are shown from A) to G) indicating respective access changes from one codeset to another.

The search begins with access to sub-codepage 1 because this is the highest priority access. Thus, character 1 is retrieved and its numeric code is converted by outputting the numeric code stored in sub-codepage 1. The conversion process then takes the second character as input and the same procedure is repeated because the second character is stored in sub-codepage 1.

The current character 172 cannot be retrieved in sub-codepage 1. Therefore, as indicated by arrow A), sub-codepage 0 will be accessed next because it is the sub-codepage with the next highest priority. Incidentally, the character 172 is searched for in sub-codepage 0. Thus, its numerical code will be issued as described above. Sub-codepage 0 will now continue to apply character 173. Incidentally, as shown in Fig. 2, it is also stored in sub-codepage 0. The character 174 is then processed. This time, this character is not found in codeset 0. Thus, the second row of FIG. 3 is implemented. As indicated by arrow b), codeset 1 is reaccessed because the probability of searching for a character is highest when retrieved from this codeset.

Incidentally, referring again to FIG. 2, the character 174 is searched again in codeset 1. FIG. Therefore, after issuance, the first row is again implemented. Characters 175 through 196 proceed as described above without changing the code set.

Later, a very rarely used character, character 197, is not retrieved from codeset 1. Thus, as indicated by arrow c), codeset 0 of FIG. 3 is accessed and retrieved, not retrieved here, and finally codeset 3 is accessed. Here, the character 197 is retrieved and its numeric code is issued. The search then continues in codeset 3.

Character 198 is not searched in codeset 3. Thus, the fourth row shown in FIG. 3 is implemented and codeset 1 is accessed next, as indicated by arrow d). From here, the search for character 198 is successful until no character 211 is found. Therefore, the first row is again implemented. As indicated by arrow E), codeset 0 is accessed next. Characters 211 to 215 in codeset 0 are searched.

However, character 216 cannot be retrieved, so the second line is implemented and codeset 1 is re-accessed for retrieval. However, since it is not retrieved here, codeset 2 is accessed as indicated by arrow F). Here it is retrieved and after the character character 217 is also successfully processed from codeset 2.

Thereafter, character 218 is processed and it is not retrieved from codeset 2, so codeset 1 is re-accessed as shown from the third column of FIG. Character 218 and all subsequent characters remaining in the input character set to be converted are again retrieved from codeset 1. Thus, they are processed as described above and after the last character 230 is converted, the conversion process stops. Therefore, all source character codes are converted successfully.

In the foregoing detailed description, the invention has been described with reference to specific exemplary embodiments. However, it will be apparent that various modifications and changes may be made without departing from the spirit and scope of the invention as set forth in the appended claims. Accordingly, the detailed description and drawings are only illustrative rather than limiting of the invention.

For example, the search may proceed differently after hit in a rarely used code set. Alternatively, the search may proceed automatically to the highest priority codeset without the effort to search for it in the same rarely used codeset. This situation may occur after character 197 has been processed in the detailed description given above. Statistically, it can be achieved that a small performance gain.

The invention can be implemented in hardware, software, or a combination of hardware and software. The code conversion tool according to the present invention is realized in a centralized fashion of a given computer system or in a distributed fashion in which different elements are distributed across multiple interconnected computer systems. Any type of computer system or other apparatus that performs the methods described herein is suitable. A computer program that controls a computer system when a typical hardware and software combination is loaded and executed can be a general purpose computer system and thus performs the methods described herein.

The invention may also be embedded in a computer program product, including all the features that enable the implementation of the methods described herein, and making it possible to perform these methods when loaded in a computer program.

A computer program means or computer program in this context means that a system having information processing capability in any language, code or notation is directly or a) converted to another language, code or notation b. ) Any instruction set representation intended to perform a particular function after either or both reproductions in different material forms.

The present invention is effective to provide a method and system for code conversion from Unicode text to mixed code pages that can be executed with better performance.

Claims (12)

A method of converting a source string including a plurality of source characters into a target string, wherein the source string is encoded according to Unicode code pages. And the target string is encoded according to a mixed code page comprising a plurality of sub-code pages 14, 15, 16 and 17. Associating a predetermined processing priority with each sub-codepage 14,15,16,17 to generate a processing priority sequence; Converting the characters strictly according to the priority sequence; How to convert a source string into a target string. The method of claim 1, And the priority sequence reflects the probability of retrieving a source character from one of the subcode pages (14, 15, 16, 17). The method of claim 1, Accessing the sub-code page with the highest priority that has not yet been accessed for the character if the character is not retrieved from the current subcode page. Method of conversion. The method of claim 1, A method of converting a source string into a target string where one or more characters are processed by a single hardware instruction. The method of claim 1, And said priority sequence is dynamically changed from a standard setting to an individual setting before said code conversion is executed. A computer system having program means installed for carrying out the steps of the method according to any of the preceding claims. The method of claim 6, The computer system is arranged for use as an Internet server with program means installed for carrying out the steps of the method according to any of the preceding claims. Chip means comprising hardware circuitry for implementing at least some of the steps of the method according to any of the preceding claims. A device comprising a chip according to claim 8. A computer program product for execution in a data processing system, comprising a portion of computer program code for performing the steps of the respective method according to any one of the preceding claims. The method of claim 10, The computer program is a browser program. A computer program product stored on a computer usable medium comprising computer readable program means for causing a computer to perform the method according to any one of claims 1 to 5.