KR20180002145A - Utf-8 code real-time lossless compression by using universal code method and appratus thereof - Google Patents

Abstract

The present invention is provided to achieve a real-time lossless compression. According to the present invention, [1] 110, a header of a first byte, is replaced with 101 and a bit compression effect is obtained; [2] next, 1110, a header of the first byte, is replaced with 1001 and a compression effect of 4 bits is obtained; [3] next, 11110, a header of the first byte, is replaced with 1101, a compression effect of 1 bit is obtained, and a compression effect of 6 bits is obtained; [4] next, 111110, a header of the first byte, is replaced with 10001, a compression effect of 1 bit is obtained, and a compression effect of 11 bits is obtained in overall; and [5] next, 1111110, a header of the first byte, is replaced with 11001, a compression effect of 2 bits is obtained, and a compression effect of 14 bits is obtained in overall.

Description

{UTF-8 CODE REAL-TIME LOSSLESS COMPRESSION BY USING UNIVERSAL CODE METHOD AND APPRATUS THEREOF USING UNIVERSAL CODE OF DATA CONTAINING UTF-

UTF-8 code, character set, universal code, compression technology

UTF-8 code, character set, universal code, compression technology

Specification to concrete contents to carry out invention

Specification to concrete contents to carry out invention

Specification to concrete contents to carry out invention

UTF-8 code is a global standard code to include both ASCII code and multilingual code. In Korea, Korean standard encoding method is used to represent Hangul, and Naver's search engine is very universal It is the code that is becoming. Below is the frequency of UTF-8 code usage on the web.

The present invention is an invention for achieving real-time lossless compression by combining an encoding scheme of such a UTF-8 code with a universal code. The following figure shows the coding rules of UTF-8.

The code pointer U + 0000 to U + 007F starts with 0, and the next 7 bits are the actual code.

On the other hand, the code pointer is composed of patterns 110, 1110, 11110, 111110, and 111110 in BYTE 1 according to the range, and the byte is variably read from 1 to 6 bytes.

If BYTE 1 begins with 110, the next 5 bits are read further to form byte 1, and then additional byte 2 is read.

When BYTE 1 starts with 1110, the next 4 bits are further read to construct BYTE 1, and then two additional bytes are read. The UTF-8 code is encoded as a rule according to the figure below.

After the actual code is read, the code marked with x is combined to decode the original data.

According to the present invention, the bytes starting with 0 in BYTE 1 are not subjected to a separate compression process. The main concern is that in Byte 1, the header part starting with 1 is replaced by the universal code,

[1] U + 0080 ~ U + 07FF The first byte corresponding to the pointer (BYTE 1)

 (One) The header 110 is set to 101  Instead,

(2) By removing the first 2 bits of the second byte , a 2-bit compression effect is obtained in U + 0080 to U + 07FF.

[2] Next, the first byte corresponding to the U + 0800 ~ U + FFFF pointer

(1) header 1110 To 1001  Instead,

(2) second  And third  First byte Two bits each  To obtain a compression effect of 4 bits.

[3] Next, the first byte corresponding to the U + 10000 ~ U + 1FFFF pointer

(1) By replacing the header 11110 with 1101, a compression effect of 1 bit is obtained

(2) The first two bits of the second, third, and fourth bytes, "10", are all removed, resulting in a 6-bit compression effect.

Overall, a compression effect of 7 bits is obtained.

[4] Next, the first byte corresponding to the U + 200000 ~ U + 3FFFFFF pointer

  (1) By replacing header 111110 with 10001, a compression effect of 1 bit is obtained

  (2) By removing all the most significant 2 bits of the second to fifth bytes, a 10-bit compression effect is obtained, thereby obtaining an 11-bit compression effect as a whole.

[5] Next, the first byte corresponding to the pointer U + 4000000 to U + 7FFFFFFF

  (1) By replacing the header 1111110 with 11001, a compression effect of 2 bits is obtained

  (2) By removing all the 2 most significant bits of the second to sixth bytes, a compression effect of 12 bits is obtained and a compression effect of 14 bits as a whole is obtained.

That is, if the first bit of the first byte starts with 1, the header is UTF-8

110

1110

11110

111110

1111110

Of course, there is also a possibility that 1 is added continuously and expanded.

At this time, the headers are converted into the following universal codes, and the header part of the first byte is compressed. Then, according to the type of the header of the first byte, the number of bytes to be read is changed variable. The most significant bit 2 of the subsequent variable bytes to be read Remove the bit.

101

1001

1101

10001

11001

The features of the universal code proposed in the present invention are as follows.

It is in the form of a universal code starting with 1 and ending with "01"

This code can be used in other code E.g  suffix as a direct binary code), and this code is then placed in the prefix.

Since it is a universal code starting with "1", it is completely separated from the character code starting with "0".

In the case of this universal code, the only decoding property is as follows.

For example, the transmitting side transmits as follows,

101/1001/11001/11101/101/1001

On the receiving side,

If received as 101100111001111011011001,

On the receiving side, you can divide the universal code equally by dividing each of them from "01" next to "01" each time they are first encountered.

101/1001/11001/11101/101/1001

As described above, it can be seen that the universal codes to be transmitted from the transmitting and receiving sides are directly separated.

These universal codes are as follows.

Order number M Universal code One 101 2 1001 3 1101 4 10001 5 11001 6 11101 7 100001 8 110001 9 111001 10 111101 11 1000001 12 1100001 13 1110001 14 1111001 15 1111101 16 10000001 17 11000001 18 11100001 19 11110001 ... ...

K group seed codes having the highest and lowest positions of "1" and "K" of "0" in the middle are arranged at the beginning of each group as in the case of 101, 1001, 10001 and 100001, , The universal code is developed by modifying this seed code,

For example, in the group k, when the sequence number is increased, "0" is changed to "1" one by one from the most significant bit next to the most significant bit, The seed code becomes the next universal code.

That is, for example, in the third group,

10001 is the third group seed code, and the next code is 11001, then the next code is 11101, and all the bits to the least significant 2 bits are found as "1".

The conversion table of headers whose standard first byte of UTF-8 starts with 1 is as follows.

The UTF-8 code system To convert the pressure to be compressed, 110 101 1110 1001 11110 1101 111110 10001 1111110 11001 ... . ... ..

On the other hand, the figure below shows the code system area where the Hangul is located in the UTF-8 code system. Since the header of the first byte is 1110 and Korean is used as a dictionary, if it is sequential as shown in Table 3, it should be mapped to a universal code of 1001 as shown in Table 4 below. The header is mapped to a universal code of 1001, and moreover, a higher compression rate can be achieved due to a promise as a compression and decompression protocol according to a character system or a complex system.

The UTF-8 code system To convert the pressure to be compressed, 110 1001 1110 101 11110 1101 111110 10001 1111110 11001 ... . ... ..

Now, a decompression method of the compressed UTF-8 code in the above-described manner will be described.

First, in the compressed UTF_8 code data,

[1] The most significant bit is important. When it starts with "0", it is uncompressed. It reads 7 bits as it is, and is immediately used as a UTF-8 code source corresponding to pointers from U + 0000 to U + 007F.

[2] If the most significant bit starts with "1", "01" is divided according to the decoding rule of the universal code at "01" next to the beginning, and then the universal code is first decoded.

The following example is the result of UTF_8 code compression of U + 0800 ~ U + FFFF

1001xxxx / xxxxxx / xxxxxx (/ is a virtual delimiter)

1001 xxxx / xxxxxx / xxxxxx (/ is a virtual delimiter)

Thus, if 1001 is divided first, this is because "1110" becomes the header of the original first byte as in Table 3, so the next 4 bits are read in accordance with the UTF-8 encoding rule to decode the first byte, and the variable byte reads 2 bytes (Since the header of the first byte is 1110, since it is compressed by 2 bits, it reads 2 times by 6 bits, adds 2 bits of "10" to the top of each 6 bits, Unpack it into UTF-8 code, which is shown below.

After reading and decoding the data, the above process is repeated according to whether the most significant bit value of the bit is "0" or "1", decompressing the data into the original UTF-8 code.

On the other hand, a universal code generalization method in the present invention is as follows.

That is, the code can be directly calculated from the sequence number directly from the sequence number.

Order number M Uni-buster sticking code code One 101 2 1001 3 1101 4 10001 5 11001 6 11101 7 100001 8 110001 9 111001 10 111101 11 1000001 12 1100001 13 1110001 14 1111001 15 1111101 16 10000001 17 11000001 18 11100001 19 11110001 20 11111001 21 11111101 22 100000001 23 110000001 24 111000001 25 111100001 26 111110001 27 111111001 28 111111101 29 1000000001 30 1100000001 31 1110000001 32 1111000001 33 1111100001 34 1111110001 35 1111111001 36 1111111101 37 10000000001 38 11000000001 39 11100000001 40 11110000001 41 11111000001 42 11111100001 43 11111110001 44 11111111001 45 11111111101 46 100000000001 47 110000000001 48 111000000001 49 111100000001 50 111110000001 51 111111000001 52 111111100001 53 111111110001 54 111111111001 55 111111111101 56 1000000000001 ... ...

On the other hand, it is necessary to find the K and X values as to the specific random integer M which is to be mapped to the universal code.

을 통해 K'를 계산한 뒤,i)

And then calculates K '

ii) if K 'is an integer ( positive integer ), then K = K'-1,

If K 'is not an integer (positive integer) then K = f (K'

(Where f (x) is a function that discards the decimal point of x, and x> = 0)

For reference, as one method for distinguishing whether K 'is an integer

  If f (K ') = K', K 'is an integer,

  If f (K ') ≠ K', then K 'is not an integer.

(Where f (x) is a function that discards the decimal point of x, and x> = 0)

Or i), ii) The process is expressed as a formula.

iii) Since K is obtained and M is given in this way, X is obtained by using the following formula.

As a result, the K value and the X value can be calculated from the M order by the above equation, and the universal code having the order M

Universal code = top level " 1 "  Then X consecutive "1" and K- X  Quot; 0 "and the lowest one" 1 ".

Universal code = "1" X consecutive "1" and K-X consecutive "0" s and the lowest one "1"

The method of decoding the generated universal code according to this formula is as follows.

That is, the value obtained by subtracting 2 from the total length of the universal code is K,

T = "0" in the lowest direction from the second bit at the top of the universal code

The number of "1" s until the meeting. At this time, it is important to be less than the second bit from the top.

The order M is as follows.

E.g,

In the case of the universal code "1101 ", since K = 2 and T = 1

Substituting into the above formula, M = 3, so that it can be seen that the same sequence number is derived as shown in Table 5.

In the case of the universal code "100001 ", since K = 4 and T = 0,

Since M = 7, it can be confirmed that the order is completely derived.

Claims (1)

Domestic priorities, and PCT application claims.