DE60131490T2 - Method for converting UNICODE text into mixed codepages - Google Patents

Description

1. BACKGROUND OF THE INVENTION

1.1. FIELD OF THE INVENTION

The The present invention relates to a method and system for conversion between character codes associated with computer readable characters. In particular, it relates to a method and system for conversion a source string encoded according to the UNICODE standard, into a target string that encodes mixed codepages shall be.

1.2. OBJECTS OF THE INVENTION

Computers and other electronic devices typically use text to interact with users. The text is usually displayed on a monitor or other type of screen. Since the text must be rendered in digital form within the computer or other electronic device, character set encoding must be used. In general, a font encoding is used to encode each character of the font with a unique digital representation. The characters (which are encoded) correspond to letters, numbers and different text symbols. They are assigned numerical codes for use by computers or other electronic devices. The most common character set for use in computers and other electronic devices is the American Standard Code for Information Interchange (ASCII). ASCII uses 7-bit sequences for its encodings. In other countries, different character sets are used. In Europe, the prevailing coding standards are the ISO 8859-X family of standards, especially ISO 8859-1 (called "Latin-1"), developed by the International Standards Organization (ISO). In Japan, the predominant character encoding standard is JIS X0208, JIS being the Japanese Information Standard (JIS) developed by the Japan Standards Association (JSA). Examples of other existing fonts are Mac ^™ OS Standard Roman (by Apple Computer, Inc.), Shift-JIS (Japan), Big5 (Taiwan) and many more.

The mentioned above fonts are stored in a so-called code page, which is itself is a kind of table that represents the coding of each character and from the character sets consists. Thus, for each character has its associated numerical code of that kind indicated that between these two a unique association exist. Most codepages assign each character a numeric Code with a length from one byte too. However, there are also code pages with a numeric Code of more than one byte in length, for example with two bytes or three bytes in length. Code pages, the characters which all have the same code length are considered simple Code pages.

Around the complexity individual, language-specific, national needs better To take into account, there are also so-called mixed codepages. A mixed code page contains at least two subcode pages, their coding is in length can differentiate. These subcode pages are also called code sets. They are numbered 0 to 3. The Japanese mixed code page IBM-33722, for example, includes the IBM 895 code sets (1 byte, code set 0), IBM-952 (2 bytes, code set 1), code page IBM-896 (Escape 8E + 1 byte, code set 2) and IBM-953 (escape 8F + 2 bytes, code set 3).

in view of of the constantly increasing process of globalization of economic life and the Networks and given the ever-increasing influence of the Internet, practically all countries Connects around the world, needs any conversion of data between computers that use different types of code pages, so as soon as possible and optionally as easy as possible - done.

Around To simplify the code conversions, the so-called UNICODE standard has been developed and is now internationally recognized. UNICODE offers a single system for representing all existing code sets. Of the Structure of the UNICODE coding system is, apart from the directionality, independent of Structure of basic word processing algorithms. It is assumed, that UNICODE implementations have appropriate word processing and / or Include playback algorithms. Each according to the UNICODE standard coded characters are represented by a 2-byte numeric code shown.

The The problem now is a very efficient way to convert from the UNICODE standard to the above-mentioned mixed code pages to find, d. H. there is a source string given with the UNICODE standard and is very simple and fast Way into a code system that is supposed to be a multitude of code pages, for example, a variety of four, such as mentioned above has been.

A conversion method of UNICODE into multiple code pages in the prior art is in U.S. Patent 5,793,381 disclosed. The code conversion system allocates a single source character or sequence of characters to either a single destination character or a sequence of destination characters by searching the location of the associated destination character in a mapping table. When reading a source character, this mapping table is accessed to determine which of the partial code pages is used for code conversion. A certain sub-code page is found that will be used for code conversion until a source character is found in the input string that can not be converted with that sub-code page. In this case, the additional mapping table is accessed again to find the correct partial code page. In addition, the prior art code converting system includes a fallback handling that serves, together with the mapping table, to identify one or more characters in the target encoding that can be used as a replacement for the text element in cases involving the search routine was unable to identify one or more characters in the destination encoding for the text element.

This However, the prior art approach uses an additional one Lookup table that makes the system slower and more complicated than necessary.

1.3. OBJECT OF THE INVENTION

It Thus, an object of the present invention is a method and a system for code conversion of UNICODE text into mixed Provide code pages that perform better can.

2. OVERVIEW OF THE INVENTION AND ITS ADVANTAGES

These Objects of the invention are achieved with the features that in the appended main claims are. Further advantageous arrangements and embodiments of the invention are set forth in the corresponding subclaims.

in the short overview of the basic concepts of the present invention is intended assign a fixed priority to each sub-code page and the characters strictly according to this priority order without using a mapping table to determine in which of the plurality of sub-code pages the destination character and its Coding is stored. Advantageously, the sub-code page, the most common used characters, the highest priority assigned, and the partial codepage with the least commonly used characters becomes the lowest priority assigned. Thus, in the case of four sub-code pages, a priority order between the partial code pages. Every priority is one Measure of the probability to find a specific character in the corresponding sub-code page.

Next this fundamental Approach will continue for the case that a character is not in a particular sub-code page has been found, intends to access the highest priority subcode page, on the for this character has not been accessed yet.

The application of the above mentioned inventive measures gives the following advantages:
First, performance is significantly increased as compared to the prior art conversion method mentioned above, since there is no separate mapping table that must be accessed each time a character is not found in the currently used sub-code page.

Secondly needs the extra Mapping table not even created. This saves a lot of work.

thirdly can the priority order, which is lent to the plurality of sub-code pages, thus set up be that country specific knowledge of the Language is exploited. In this way, the conversion method according to the invention easily adapted to the specifics of a certain country-specific Code page system imposed.

ever after a particular individual case, the above-mentioned order of priority may be required the execution the code conversion alternatively also dynamically from a standard to an individual Setting changed become the code conversion method according to the invention to adapt to specific requirements, which depend on a specific to be converted text, if known in advance is that the text is not representative for the Average is. For example, the new priority order in the header of the file to be converted.

Another distinct advantage of the method of the invention is that it provides a concept that is open to exploiting the specific advantage of modern computer systems, which is that hardware instruction sets can be used, each having a plurality of characters rather than just one character to process. Such modern hardware instruction sets require a linear table to search for the target characters without any additional checking access to any kind of allocation table.

The The present invention can be advantageously used with the Internet when code conversion is required. Furthermore, can the tool according to the invention be integrated into a database application when the conversion Some contents of this database are mixed by a UNICODE text in mixed Code pages is likely.

Becomes the method of the invention applied in a case where the probability of a given Finding characters in one of the multitude of subcode pages is the same for all subcode pages is high, then in the presence of 4 sub-code pages is a statistical Mean of only 2 additional ones Access required. This value drops to 1.5 at 3 sub-code pages and 1 in the case of two sub-code pages. In one case with Japanese EUC tables, where 70% of all characters are found in code set 1, 30% in Code set 0 and less than 1% in the remaining code sets 2 and 3, is the statistical mean just over 1.

Furthermore The present invention may advantageously be at least partially be integrated into a hardware implementation directly in a hardware chip is branded. Such chip means include then hardware circuits that at least parts of the steps of the Code conversion method according to the invention implement and reflect. In view of the ever-increasing Variety of telecommunication devices and their ever increasing Functional bandwidth with more and more technical features can then such a chip is used in a wide variety of devices become. In view of the devices available today, such a chip can advantageously be used in every device be used as part of an international communication system is, for example, in routers in any type of network, eg. The internet, in set-top boxes for Television or radio receivers, especially for digital television and digital radio, in mobile phones, in any kind of computer or Telecommunication handset or in any other device with an input interface for processing foreign language data.

3. BRIEF DESCRIPTION OF THE DRAWINGS

The The present invention is exemplified and is not limited to the shape of the figures of the accompanying drawings. It demonstrate:

1 a schematic logical representation showing the basic elements of the method according to the invention,

2 a rough illustration of an arbitrary selected example showing, for each of a plurality of 230 source characters, in which subcode page of a plurality of 4 subcode pages each character can be found, and

3 a logic diagram showing the order of code set accesses when a method according to a preferred embodiment of the present invention is applied during code conversion.

4. DESCRIPTION OF THE PREFERRED CARRYING

As in the figures generally is in 1 in box 10 the entirety of UNICODE characters, which are subjected to the conversion method according to the invention, shown symbolically.

According to a preferred embodiment of this method according to the invention certain rules of priority will be imposed 12 which establish a well-defined priority order between the plurality of sub-code pages used. The term "code set n", where n is an integer, is sometimes used herein and basically means the same as the term "partial code page n". In the in 1 In the case illustrated, four partial code pages are used, which are referred to as code set 1, 14 , Code set 0, 15 , Code set 2, 16 and code set 3, 17 are designated. In the box 10 For example, four exemplary selected characters are shown whose coding is each on a different separate sub-code page than on the right-hand side of the code 1 is apparent.

As is apparent from the drawing, in each of the tables 10 . 14 . 15 . 16 . 17 the numeric code is stored for each character.

In the 2 and 3 For example, a preferred embodiment of the inventive method is described in greater detail in an exemplary code conversion from the Japanese UNICODE to Japanese mixed EUC sub-code pages.

Before code conversion begins, an already existing estimation is used for the Japanese EUC subcode pages, which estimates that the subcode pages in this particular case are organized such that the code set 1 comprises nearly 70% of all occurring characters, whereas the code set 0 about 29%, code set 2 about 0.6% and code set 3 about 0.4% of the total occurring characters comprises. This code set probability distribution is also in 1 in this regard, as the most commonly used code set 14 front and the least used as the last sentence 17 of the "stack" of code sets, so the order of priority mentioned above is:
Code set 1, code set 0, code set 2, code set 3.

In 2 FIG. 3 is a rough illustration in an arbitrarily selected example showing, for each of the plurality of 230 source characters, in which subcode page of a plurality of 4 subcode pages each character can be found.

In a single example conversion process is intended to be a whole be converted from 230 source characters. It should be understood that the fairly small number of 230 was chosen for clarity to improve the process.

The total of 230 source characters thus consists of the input sentence, the reference numeral 10 in 1 is shown symbolically. The new numerical code to be output by the method of the invention is in the four sub-code pages 14 . 15 . 16 . 17 , please refer 1 right, saved as follows:
Characters 1 to 171 in code sentence 1,
Characters 172, 173 in code set 0,
Signs 174 to 196 again in Codesatz 1,
Character 197, which is a rather rarely used character, is found in code sentence 3,
Characters 198 to 210 again in code sentence 1,
Characters 211 to 215 are stored in code set 0,
Characters 216, 217, which are also quite rarely used, in Codeset 2 and
Characters 218 to 230 in code set 1.

The conversion scheme serially processes the above-mentioned source characters. In a preferred embodiment of the inventive method, hardware instruction sets that process a plurality of characters simultaneously may be used. An example of this is the IBM OS / 390 "Translate Two to One" hardware instruction set, TRIO for short, for converting a string consisting of 2-byte characters into an output buffer comprising 1-byte characters. This hardware command set takes arguments as follows:
the string to convert
the target buffer into which the converted string can be stored
a character indicating that a particular input character can not be converted,
a conversion table, which is controlled by the character to be converted and in which the converted character is stored at the thus controlled position.

Around the clarity However, they want to and to focus on the very core of the present invention to concentrate, becomes the input string described above subjected to a single character conversion process, i. H. one Operation that treats each character separately.

• 1. greife zuerst auf den Codesatz mit der höchsten Priorität zu;
• 2. kann ein bestimmtes Zeichen nicht im Codesatz mit der höchsten Priorität gefunden werden, so fahre mit dem Codesatz fort, der die nächstniedrigere Priorität aufweist und so weiter;
• 3. für den Fall, dass ein Zeichen in einem Codesatz nicht gefunden wurde: greife auf den Codesatz mit der höchsten Priorität zu, auf den für dieses Zeichen noch nicht zugegriffen wurde.

According to a preferred feature of the present embodiment, a set of processing rules derived from the above-mentioned priority order is established. These processing rules are as follows:

• 1. First access the highest priority code set;
• 2. If a particular character can not be found in the highest priority code set, continue with the code set having the next lower priority, and so on;
• 3. In the event that a character was not found in a code set: access the highest priority code set that has not yet been accessed for that character.

Applying these rules yields the in 3 specified scheme.

3 includes four lines. The first line reflects the order of subcode pages to be accessed after a particular character could not be found when accessing subcode page 1, that is, the highest priority subcode page, is started , Thus, if a character can not be found on the partial code page 1, the partial code page 0 is accessed to search for the current character. If this current character is found in the partial code page 0, the process in this sub-code page continues with the next character to be converted.

For this next character, the second line is used for the search. Otherwise, if the above-mentioned current character can not be found in the partial code page 0, partial codepage 2 is accessed for the further search. Then a corresponding scheme for the partial code page 2 is followed:
If the current character in sub-code page 2 is found, the numerical code assigned to it, ie the converted code, is output, and the next character is written according to the in 3 searched third line searched. Otherwise, that is, if the current character is not found in the partial code page 2, the last partial code page 3 is accessed for the search. Then the character is found and the search is done with the in 3 continued fourth row.

As can be understood from the foregoing description according to this embodiment of the present invention, the search is always in continued the particular partial code page in which the last character was successfully detected.

With respect to the second, third, and fourth lines, the next highest-order sub-code page is accessed for the next character to be converted, here the in-line 1 with the reference number 14 displayed code page 1 if this next character was not found in the corresponding current sub-code page.

With reference to in 2 In the illustrated sequence, the manner of processing, ie, accessing the various sub-code pages, will now be described in more detail. As well in 2 as well as in 3 arrows are shown, labeled A) to G), to indicate corresponding access changes from one code set to another code set.

The Search starts with an access to the partial codepage 1, since this the partial code page with the highest priority is. Thus, character 1 is found and its numeric code becomes converted by the numeric code stored in sub code page 1 is issued. Then the conversion process takes the second one Characters as input and the same procedure is repeated because the second character is also stored in sub-code page 1. This Process continues until character number 172 is reached.

The current character 172 can not be found in the partial code page 1. Thus, as arrow A indicates, sub-code page 0 is next accessed because it is the next-highest-priority sub-code page. Randomly, character 172 is found in subcode page 0. Thus, its numerical code is output as described above. Part code page 0 is still applied to character 173. Coincidentally it is how out 2 can be seen, also stored in partial code page 0. Then character 174 is processed. This time, this character is not found in sub code page 0. Thus, the second line in 3 , As indicated by arrow B), it reverts to partial code page 1 because the probability of finding the character is highest when searching in this code set.

Coincidentally, as again turns off 2 it can be seen that the character 174 found again in partial code page 1. Thus, after the output again the first line. The characters 175 through 196 are processed as described above without changing the code set.

Then character 197, which is a rather rarely used character, is not found in code set 1. Thus, like the arrow C) in 3 indicates, accesses code set 0 and searches it. Then, since the character is not found there, code set 2 is accessed and searched, and since it is not found there, code set 3 is finally accessed. Here, the character 197 is found and its numeric code is output. Then the search continues in the code set 3.

Character 198 is not found in code set 3. Thus, the in 3 4, and next, as arrow D indicates, code set 1 is accessed. There, the search for character 198 succeeds until character 211 can not be found. Thus again the first line applies. As arrow E indicates, code set 0 is next accessed. In the code set 0, the characters 211 to 215 are found.

character However, 216 can not be found, and thus the second applies Line, and for the search is again accessed on codeset 1. But there it is is not found, codeset 2 is accessed, like the arrow F) indicates. Here it is found, and after the edition will be too the next Character 217 successfully processed from code set 2.

Then the character 218 is processed, and since it is not found in the code set 2, code set 1 is accessed again, as from the third line in FIG 3 is apparent. Characters 218 and all subsequent characters remaining in the input set of characters to be converted are found again in the code set 1. Thus, they are processed as described above, and after the last character 230 has been converted, the conversion process ends. Thus all source character codes were successfully converted.

In In the preceding description, the invention was based on a specific exemplary embodiment described. However, it is obvious that various modifications and changes can be done without from that in the attached claims deviated scope of the invention. The description and the drawings are accordingly exemplary and not in a restrictive To look at the senses.

For example can search for a hit in a rarely used code set continue differently: alternatively, the search can be automatic with the highest set of codes priority be continued without attempting to use the sign in the same rare used code set to find. This situation occurs after in the detail detailed above Description character 197 was processed. Statistically speaking So another small increase in performance can be achieved.

The The present invention may be in hardware, software or in combination be realized from hardware and software. A code conversion tool according to the present Invention can be implemented centrally in a computer system be, or distributed, with different elements over several, interconnected computer systems. Any kind from computer system or any other to the execution of the above Method adapted device is suitable. A typical combination hardware and software can be a universal computer system with a Computer program, which is in the loaded and running state controls the computer system so that it has the methods described above implements.

The The present invention can also embedded in a computer program product that has all the features to enable the implementation of the methods described herein comprises and which after loading into a computer system is able to do this Execute methods.

Of the Term computer program means or computer program is available in the present Context independent of language, code or notation for each expression of an instruction set, the one over a data processing functionality causing system, a special function either directly or after at least one of the following two operations execute: a) Conversion to another language, notation or another Code; b) reproduction in another material form.

Claims (12)

A method of converting a source string that includes a plurality of source characters into a target string, wherein the source string is encoded by UNICODE code pages, and the target string is to be encoded with mixed code pages containing a plurality of sub-code pages ( 14 . 15 . 16 . 17 ), characterized by the following steps: assigning a fixed processing priority to each sub-code page ( 14 . 15 . 16 . 17 ), resulting in a processing priority order, - convert the characters strictly according to this priority order. The method of claim 1, wherein the priority order is the probability of finding a source character in one of the sub-code pages ( 14 . 15 . 16 . 17 ). The method of claim 1, further comprising the following step includes: - Access to the highest-priority sub-code page, to the for a Character has not yet been accessed when this character in the current part code page was not found. Method according to claim 1, wherein more than one character is processed by a single hardware instruction set. Method according to claim 1, wherein the priority order before the execution code conversion dynamically from a standard to an individual Setting changed becomes. Computer system, on the program means to carry out the Steps of a method according to one of the preceding claims 1 to 5 are installed. Computer system according to the preceding claim, which is intended for use as an internet server on the program agent to carry out the Steps of a method according to one of the preceding claims 1 to 5 are installed. Chip means comprising hardware circuits which at least partially the steps of a method according to one of the preceding claims Implement 1 to 5. Device, which comprises the chip according to the preceding claim. Computer program for execution in a data processing system, the computer program code section for performing appropriate steps the method according to one of the claims 1 to 5. Computer program according to the preceding claim, which is a browser program. Computer program product based on a computer Medium is stored and include computer readable program means, which cause a computer to use the method after one of claims 1 to 5 perform.