TW201322649A - Method for data compression - Google Patents

Abstract

A method for data compression is provided, including the steps of: dividing a source data into a plurality of blocks, the division being based on the consecutive repetitive data; compressing each block and storing compressed block in a segment format having a first length indicating length of consecutive repetitive data, a second length indicating length of non consecutive repetitive data and a data area; forming related segments into a section; and constructing complete compressed data.

Description

Data compression method

The invention relates to a data compression method, in particular to a compression method for continuous repetitive data.

Although the network bandwidth and the capacity of digital storage devices have continued to increase, with the growth in the number of users and the growth in data throughput and throughput, it seems that there is still a lack of room to fully meet the bandwidth and capacity requirements. . Therefore, the importance of data compression technology is still a long-lasting and still plays a decisive role. In particular, with the spread of short, thin and thin consumer electronic products, how to make the most effective use of the limited storage capacity of portable electronic products and to balance the capacity and use efficiency is an important issue.

For example, the first figure shows a schematic diagram of an access architecture of a conventional compressed data. As shown in the first figure, the CPU 110 reads the compressed data from the flash memory 120, and transmits the compressed data to the FPGA or DSP 130 through the Serial Peripheral Interface Bus (SPI Bus). And write to memory 140 again. The traditional Field Programmable Gate Array (FGPA) or Digital Signal Processing (DSP) loads data in a character, a character, and then reads it. . The larger the amount of data, the more times you read and write. In contrast, the time it takes to load the data will be lengthened. Therefore, the way to speed up the loading of data is to reduce the number of reads. If the data is compressed by the commonly used zip method, although the compression ratio can be achieved, the use of the memory is reduced, but when the data is loaded, the data must first be decompressed into the memory. Therefore, although the memory space is saved, the number of times the memory is read is not reduced. In other words, the time to load the data is shortened.

Another common data compression method is Run-Length Encoding (RLE), which is especially suitable for compressing data with runs of same value; in other words, using a single data value (value And a length (length) to replace the entire continuous repeating data string. For example, if the original material is:

WWWWWWWBWWWWWWWWWBBBWWWWWWWW

Then use RLE coding, the coding result is:

7W1B9W3B8W

The above code can be interpreted as: 7 W, 1 B, 9 W, 3 B, 8 W. Therefore, the original data is 28 characters long and 10 characters after compression. If there are more continuous data, the compression ratio can be higher, that is, the compression effect is better.

However, the worst case of the above RLE compression method is when the original data does not have any continuous duplicate data strings, for example, when the original data is:

WBWBWBWBWBWBWBWBWB

Its RLE coding result is:

1W1B1W1B1W1B1W1B1W1B1W1B1W1B1W1B1W1B

In other words, therefore, the length of the original data is 18 characters, which is 36 characters after compression, which is doubled. For the above examples, other similar compression methods, such as the common LZ77, LZ78, etc., or the sliding window, propose an improvement method, that is, the WB is regarded as a repeating unit. However, if the original data does not have any regular repeating data, the above improvement method is not sufficient to improve the low compression ratio of the irregular data. Therefore, a compression method that can effectively compress continuous repetitive data and can respond to non-continuous repetitive data strings is not only an important issue for researchers, but also a top priority for the relevant industry.

Based on the above-mentioned shortcomings of the prior art, the main object of the present invention is to provide a data compression method, which can effectively compress continuous repetitive data, achieve a high compression ratio, save memory space, and simultaneously reduce data loading time.

Another object of the present invention is to provide a data compression method, which can effectively respond to data without continuous repetitive data strings, without compression, and avoid wasting time and increasing data volume.

Another object of the present invention is to provide a data compression method, which can be used for decompression, that is, while decompressing, while using part of the data that has been decompressed, to increase efficiency; be usable.

To achieve the above object, the present invention provides a data compression method, comprising: cutting a raw data into a plurality of blocks; compressing each block separately, and compressing the data in a paragraph format; Form all consecutive related paragraphs into one chapter; and construct complete and compressed complete data.

The above and other advantages of the present disclosure will be described in detail below with reference to the following drawings, detailed description of the embodiments, and claims.

The second figure shows a schematic diagram of the compression principle used in a data compression method of the present invention. As in the example of the second figure, a raw data 201 contains continuous repetitive data S, and d1, d2, d3, d4, d5, d6, d1', d2', d3', d4, respectively, with non-continuous repetitive data. ', d5', d6', d7', d8', d9', etc. The compression method of the present invention uses a continuous repetitive data string and its subsequent non-continuous repetitive data string as a unit of compression, and the compressed data is stored in a segment format. Each of the paragraphs includes a first length, a second length, and a data segment; wherein the first length represents the length of the continuous repeating data string by a size of a byte, the second length The length of the non-continuous repetitive data string is represented by a size of one character, and the data segment is data for storing the non-continuous repetitive data string. In other words, the compression method used in the present invention can be defined as a compression mode of LLD, that is, Length+Length+Data (length+length+data). Therefore, as shown in the second figure, the original data 201 is compressed into two paragraphs 202, 203, wherein the contents of the first paragraph 202 are: a first length 7, a second length 6, and a data segment d1. D2, d3, d4, d5, d6; and the content of the second paragraph 203 is: first length 5, second length 9, data segment is d1', d2', d3', d4', d5', d6 ', d7', d8', d9'.

It should be noted that since the present invention uses only the size of one character to represent the length of the data string, the length of the data string that can be represented is up to 255. In the process of compressing the original data, it may be faced that the length of the continuous repetitive data string exceeds 255 or the length of the non-continuous repetitive data string exceeds 255. Another possibility is that there may be more than one continuous repetitive data string in the original data, and the repetitive data is not the same, for example, S1 and S2. In other words, the data repeated in each successive repetitive data string is not the same. In the third case, the original data does not contain any continuous repetitive data strings and can be compressed. The data compression method of the present invention is also included in the design considerations for the case where the above three kinds of original materials may occur.

For the case where the above length is exceeded, the data compression method of the present invention adopts a method of cutting the original data into a plurality of blocks. The third figure shows a schematic diagram of the invention cutting the original data into a plurality of blocks. As shown in the third figure, the original data 301 is a continuous repetitive data S of 500 characters followed by discontinuous repetitive data d1-d500 of 500 characters. Since the length of the data string that can be represented in each paragraph of the present invention is up to 255, the present invention first cuts the original data 301 into a plurality of blocks, wherein the block 302 includes 255 consecutive repetitive data S and blocks. 303 includes 245 consecutive repetitive data S and 10 non-continuous repetitive data d1-d10, block 304 includes 255 non-continuous repetitive data d11-d265, and block 305 includes 235 non-continuous repetitive data d266-d500 . In this way, the compressed corresponding paragraphs should respectively be paragraph 3021 including: first length 255, second length 0, no data segment; paragraph 3031 includes: first length 245, second length 10, data segment d1-d10 The paragraph 3041 includes a first length 0, a second length 255, and a data segment d11-d265; the paragraph 3051 includes a first length 0, a second length 235, and a data segment d266-d500.

Similarly, in the case where there may be more than one continuous repetitive data string in the original data, and the repetitive data is not the same, the present invention particularly adds a category of fields for indicating the situation. The compression category used to compress the data. For example, when the category is 0x00, it indicates that the compressed data contains only one continuous repetitive data S; when the category is 0x01, it indicates that the compressed data contains only at least two consecutive repetitive data S1, S2. The compressed full data format of the present invention will be described in detail later.

In addition, for the case where the original data does not contain any continuous repetitive data strings and can be compressed, the category field as described above also provides a solution for the above situation. For example, when the category is 0xFF, it means that the compressed data completely retains the original data and is not compressed at all.

The fourth A-fourth C diagram shows a schematic diagram of the complete compressed data format of the present invention. As shown in the fourth figure, the compressed data format of the present invention is classified according to its compression category (A) compression for only one continuous repetitive data, and (B) compression for at least two consecutive repetitive data, and ( C) The original data is completely preserved and is not compressed at all. Figure 4A shows the compressed data format for a continuous repetitive data, including a checksum, a version field, a category field, and a chapter ( Section); wherein the check field is used to check whether the compressed data is correct, and the version field indicates the software version used in the compressed calculation, and the category field is set to 0x00, indicating that only one type of continuous Repetitive data is compressed, so there is only one chapter at a time. The chapter further includes an S-byte field for storing the continuous repetitive data S, a compressed length field for storing the length of the chapter, and a plurality of paragraphs, each paragraph The format of the LLD is as described above. Figure 4B shows a compressed data format for at least two consecutive repetitive data, including a check field, a version field, a category field, and at least two chapters; wherein the check field As with the version field, as described above, the category field is set to 0x01, indicating that compression is only performed for at least two consecutive repetitive data, so there are at least two chapters. Each chapter further includes an S-byte field for storing the continuous repetitive data S in the chapter, a section length field, a compressed length field, and a plural number. Each paragraph is in the format of LLD as described above. In other words, each chapter is only compressed for one continuous repetitive data. The fourth C picture shows the complete retained original data, the completely uncompressed data format, including a check field, a version field, a category field, a reserved field (reserved), a compressed total length field and A data segment; wherein the check field and the version field are as described above, the category field is set to 0xFF, indicating that the original data is completely retained, and is not compressed at all. The reserved field is reserved for the corresponding S-byte field, and the data area is the complete copy of the original data block. It can be seen from the category field that the chapter format and number in the compressed complete data can be known.

The fifth figure shows a flow chart of a data compression method according to the present invention. As shown in the fifth figure, step 501 is to cut a raw data into a plurality of blocks, and the cutting method is based on continuous repetitive data, and each block can be a continuous repetitive data string. Then a non-continuous repetitive data string, a continuous repetitive data string, or a non-continuous repetitive data string, and the total length of each block is less than 255 characters. Step 502 is separately compressed for each block, and the compressed data is stored in a paragraph format. The compression method includes storing the continuous repetitive data in one character, the length of the continuous repetitive data string in one character, the length of the non-continuous repetitive data string in one character, and copying the non-continuous repetitive data to the data. Within the paragraph. Step 503 is to group all consecutive related paragraphs into one chapter. Step 504 constructs the complete compressed data, for example, including setting a check field, a version field, a category field, and filling in at least one chapter or data segment to complete the compressed complete data.

It is worth noting that when decompressing, the subsequent chapters can be interpreted according to their category fields, and each paragraph or data section in at least one of the subsequent chapters can be restored to the original data. For example, if the first length of the paragraph is 50 and the second length is 30, the paragraph can be decompressed and restored to its original data string, including: 50 consecutive repetitive data S (stored in S-byte), and stored in 30 non-continuous repetitive data in the data section of the paragraph.

The present invention has the following advantages over the above-described embodiments of the present invention as compared with the prior art:

1. Simple structure, for example, can be applied to the system boot program.

2. It can effectively compress continuous repetitive data and achieve high compression ratio, which can save memory space and reduce data loading time. It can also effectively respond to data that does not have continuous repetitive data strings, without compression, and avoid wasting time and increasing data volume.

3. Can be used while solving, that is, using some of the data that has been decompressed while decompressing to increase efficiency; rather than waiting for all data to be compressed before use.

Therefore, the method of data compression of the present invention can achieve the intended purpose and effect by the disclosed technology, and meets the requirements of novelty, advancement and industrial utilization of the invention patent.

The above is only the embodiment of the disclosure, and the scope of the disclosure is not limited thereto. All changes and modifications made to the scope of the patent application of the present invention are intended to fall within the scope of the invention.

110. . . CPU

120. . . Flash memory

130. . . FPGA/DSP

140. . . Memory

201. . . Source material

202. . . paragraph

203. . . paragraph

301. . . Source material

302. . . Block

303. . . Block

304. . . Block

305. . . Block

3021. . . paragraph

3031. . . paragraph

3041. . . paragraph

3051. . . paragraph

501. . . step

502. . . step

503. . . step

504. . . step

The first figure shows a schematic diagram of the access architecture of the conventional compressed data.

The second figure shows a schematic diagram of the compression principle used in a data compression method of the present invention.

The third figure shows a schematic diagram of the invention cutting the original data into a plurality of blocks.

The fourth A-fourth C diagram shows a schematic diagram of the complete compressed full data format of the present invention.

The fifth figure shows a flow chart of a data compression method according to the present invention.

501. . . step

502. . . step

503. . . step

504. . . step

Claims (7)

A data compression method comprises the steps of: cutting a raw material into a plurality of blocks, the cutting manner is based on continuous repetitive data; respectively compressing for each block, the compression method comprising continuous repeatability The data is stored in one character, the length of the continuous repetitive data string is represented by one character, the length of the non-continuous repetitive data string is represented by one character, and the non-continuous repetitive data is copied; the consecutive related paragraphs are grouped into one chapter. ; Construct the fully compressed data and fill in at least one chapter to complete the compressed data. The method of claim 1, wherein the cutting method generates each of the blocks as a continuous repetitive data string followed by a non-continuous repetitive data string, a continuous repetitive data string, or A non-continuous repetitive data string, one of the above three. The method of claim 1, wherein the total length of each block is less than 255 characters. The method of claim 1, wherein the compressed data is stored in a paragraph format, the paragraph format further comprising a first length, a second length, and a data segment, the first length representation The length of the continuous repetitive data string, the second length representing the length of the non-continuous repetitive data string, and the data segment replicating the non-continuous repetitive data. The method of claim 1, wherein the format of each chapter further comprises an S-byte field, a compressed total length field and at least one paragraph, the S-byte field storing the continuous Repeatable data, the total length field after compression stores the length of the chapter. The method of claim 1, wherein the format of each chapter further comprises a reserved field, a compressed total length field and a data area, wherein the chapter format is used when the original data is not When there is continuous repetitive data, the original data is retained without any compression. The method of claim 1, wherein the compressed complete data further includes a check field, a version field, and a category field, wherein the category field is used to determine the compressed complete The format and number of chapters in the data.