JP2016052046A - Compression device, decompression device and storage device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform lossless compression on data at a high compression rate and a high throughput.SOLUTION: A compression device includes a first compression part, a second compression part and a selection part. In the case where at least one of partial data streams obtained by dividing an input data stream in a predetermined size is matched with any dictionary registration data, the first compression part generates first compression data including a dictionary address associated with at least one of dictionary registration data. The second compression part generates second compression data in which no dictionary address is included, by compressing the input data stream on the basis of either comparison of the first compression data with past first compression data or past input data streams. The selection part selects either the first compression data or the second compression data having a smaller data size, thereby obtaining compression data of the input data stream.SELECTED DRAWING: Figure 1

Description

  Embodiments relate to data compression and decompression.

  In recent years, mobile terminals such as tablet terminals and ultrabooks are spreading. As a storage of such a mobile terminal, an SSD (Solid State Drive) that is superior in terms of power consumption, impact resistance and the like as compared with an HDD (Hard Disk Drive) may be adopted. Furthermore, the capacity of SSDs is increasing, and the host interface is shifting from SATA (Serial ATA) to PCI (Peripheral Component Interconnect) Express. As a result, the amount of data handled by the SSD and the access speed are increasing.

  In order to improve SSD responsiveness and reliability, lossless compression can be used in association with reading and writing data. Specifically, the SSD controller compresses the original data and writes the compressed data to the NAND flash memory in the SSD. Then, the SSD controller reads the compressed data from the NAND flash memory and decompresses the compressed data to restore the original data. By utilizing such lossless compression, the responsiveness of the SSD is improved especially when the host is suspended / resume. Furthermore, since the over-provisioning of the NAND flash memory becomes equivalently large, the life and reliability of the SSD are also improved. Therefore, the SSD controller preferably compresses the entire data stored in the storage (NAND flash memory) losslessly with a high compression rate and high throughput. The overall data stored in the storage is, for example, a system file such as an OS (Operating System) or a program, a memory image such as a hibernation file or a page file, or user data.

B. Sukwani, B.H. Abali, B.A. Brezzo, and S.M. Asaad, “High-Throughput, Lossless Data Compression on FPGAs,” in Field Programmable Custom Computing Machines (FCCM), 2011 IEEE 19th Annual. Symp. on. IEEE, 2011, pp. 113-116.

  The embodiment aims at lossless compression of data at a high compression rate and high throughput.

  According to the embodiment, the compression device includes a dictionary memory, a first compression unit, a second compression unit, and a selection unit. The dictionary memory stores a dictionary table that associates dictionary addresses with dictionary registration data of a predetermined size. The first compression unit compresses the input data string based on a comparison result between each of the partial data strings obtained by dividing the input data string by a predetermined size and the dictionary registration data, so that at least one of the partial data strings is If it matches any of the dictionary registration data, first compressed data including a dictionary address associated with at least one of the dictionary registration data is generated. The second compression unit compares the first compressed data with the past first compressed data, and compresses the input data string based on at least one of the past input data strings, thereby obtaining a dictionary address. Second compressed data not included is generated. The selection unit obtains compressed data of the input data string by selecting one of the first compressed data and the second compressed data having a smaller data size.

  According to the embodiment, the decompression device includes a dictionary memory, a first decompression unit, a second decompression unit, and a selection unit. The dictionary memory stores a dictionary table that associates dictionary addresses with dictionary registration data of a predetermined size. The first decompressing unit extracts, from the compressed data, a code word of each partial data sequence obtained by dividing the pre-compression data corresponding to the compressed data according to a predetermined division pattern, and the partial data sequence corresponds to one of the dictionary addresses If so, the first decompressed data is generated by treating the dictionary registration data associated with the dictionary address as the partial data string. The second decompressing unit creates second decompressed data by decompressing the compressed data based on at least one of the past compressed data and the past decompressed data. The selection unit selects the first decompressed data if the codec of the compressed data is the first scheme, and selects the second decompressed data if the codec of the compressed data is the second scheme, Obtain decompressed data of compressed data.

  According to the embodiment, the storage device includes a storage, a compression device, and a decompression device. The compression device includes a dictionary memory, a first compression unit, a second compression unit, and a selection unit. The dictionary memory stores a dictionary table that associates dictionary addresses with dictionary registration data of a predetermined size. The first compression unit compresses the input data string based on a comparison result between each of the partial data strings obtained by dividing the input data string by a predetermined size and the dictionary registration data, so that at least one of the partial data strings is If it matches any of the dictionary registration data, first compressed data including a dictionary address associated with at least one of the dictionary registration data is generated. The second compression unit compares the first compressed data with the past first compressed data, and compresses the input data string based on at least one of the past input data strings, thereby obtaining a dictionary address. Second compressed data not included is generated. The selection unit obtains compressed data of the input data string by selecting one of the first compressed data and the second compressed data having a smaller data size. The compressed data of the input data string is read / written in the storage. The decompression device includes a dictionary memory, a first decompression unit, a second decompression unit, and a selection unit. The dictionary memory stores a dictionary table that associates dictionary addresses with dictionary registration data of a predetermined size. The first decompressing unit extracts, from the compressed data, a codeword of each partial data string obtained by dividing the pre-compression data corresponding to the compressed data read from the storage according to a predetermined division pattern, and the partial data string is a dictionary If it corresponds to any of the addresses, the first decompressed data is generated by treating the dictionary registration data associated with the dictionary address as the partial data string. The second decompression unit generates the second decompressed data by decompressing the compressed data read from the storage based on at least one of the compressed data read from the storage in the past and the past decompressed data. To do. The selection unit selects the first decompressed data if the codec of the compressed data read from the storage is the first method, and selects the codec of the compressed data read from the storage is the second method. For example, by selecting the second decompressed data, decompressed data of the compressed data read from the storage is obtained.

1 is a block diagram illustrating a compression device according to a first embodiment. Explanatory drawing of the data compression process performed by the 1st compression part. The figure which illustrates the division | segmentation pattern of an input data sequence. Explanatory drawing of the data compression process performed by the 1st compression part. 2 is a flowchart illustrating a data compression process performed by the compression apparatus in FIG. 1. 1 is a block diagram illustrating a decompressing device according to a first embodiment. The flowchart which illustrates the data expansion | extension process performed by the expansion | extension apparatus of FIG. 1 is a block diagram illustrating a storage device according to a first embodiment. The block diagram which illustrates the modification of FIG. Explanatory drawing of the structure of 1st compression data. The block diagram which illustrates the 2nd compression part contained in the compression device concerning a 3rd embodiment. The figure which illustrates the 2nd compression data generated by the 2nd compression part of Drawing 11. The figure which illustrates the 2nd compression data generated by the 2nd compression part of Drawing 11. The block diagram which illustrates the 2nd compression part contained in the compression device concerning a 4th embodiment. Explanatory drawing of the data compression process which the compression device concerning 4th Embodiment performs. Explanatory drawing of the data compression process which the compression device concerning 4th Embodiment performs. Explanatory drawing of the structure of the 1st compression data produced | generated by the compression apparatus which concerns on 4th Embodiment, and 2nd compression data. FIG. 9 is a block diagram illustrating a hardware configuration of the storage apparatus of FIG.

  Hereinafter, embodiments will be described with reference to the drawings. Hereinafter, the same or similar elements as those already described are denoted by the same or similar reference numerals, and redundant description is basically omitted.

(First embodiment)
As illustrated in FIG. 1, the compression device according to the first embodiment includes a dictionary memory 110, a first compression unit 120, a second compression unit 130, and a selection unit 140. This compression apparatus generates compressed data 11 by lossless compression of an input data string 10 having a fixed size (for example, 8 bytes). In the following description, the fixed-size input data string 10 may be referred to as a block.

  The dictionary memory 110 stores a hash function, a hash table, and a dictionary table. The hash table associates a hash value obtained by applying a hash function to a partial data string obtained by dividing the input data string 10 by a predetermined size with a parameter called a dictionary address.

  The dictionary table associates the dictionary address with dictionary registration data. The dictionary registration data is a partial data string obtained by dividing a past input data string corresponding to compressed data by the predetermined size.

  Individual hash functions, hash tables, and dictionary tables may be prepared for a plurality of sizes. As illustrated in FIGS. 3 and 4, if the size of the input data string 10 is 8 bytes, a partial data string obtained by dividing the input data string 10 in units of 2 bytes, 4 bytes, and 8 bytes is registered. In addition, a hash function, a hash table, and a dictionary table for 2 bytes, 4 bytes, and 8 bytes may be prepared, respectively.

  As will be described later, the hash table and the dictionary table stored in the dictionary memory 110 are updated using the input data string 10 in order to efficiently compress the future input data string.

  The first compression unit 120 receives the input data string 10 from an external device (for example, a host) not shown. The first compression unit 120 obtains first compressed data by compressing the input data string 10 using a hash function, a hash table, and a dictionary table stored in the dictionary memory 110. The first compressed data includes a division pattern applied to the input data string 10, dictionary matching information indicating whether each partial data string corresponding to the division pattern matches dictionary registration data, and the partial data string. Each codeword. The first compression unit 120 outputs the first compressed data to the selection unit 140.

  Specifically, the first compression unit 120 performs data compression processing as illustrated in FIG. The first compression unit 120 divides the input data string 10 into one or more types of size units, and searches the dictionary table stored in the dictionary memory 110 for dictionary registration data that matches each partial data string.

  For example, as illustrated in FIG. 3, the first compression unit 120 divides the input data string 10 in units of 2 bytes, 4 bytes, and 8 bytes. Then, for example, as illustrated in FIG. 4, the first compression unit 120 obtains a hash value by applying a hash function for 8 bytes to the 8-byte partial data string. The first compression unit 120 reads the dictionary address associated with this hash value from the hash table for 8 bytes. Further, the first compression unit 120 reads out the dictionary registration data associated with the dictionary address from the dictionary table for 8 bytes, and compares it with the 8 bytes partial data string. The first compression unit 120 similarly performs a dictionary search process for a 4-byte partial data string and a 2-byte partial data string. Since there is no dependency between dictionary search processes for different sizes, the first compression unit 120 achieves high throughput by realizing dictionary search processes for multiple sizes as parallel pipeline processing. it can.

  If each partial data string matches any of the dictionary registration data, the first compression unit 120 outputs a dictionary address associated with the dictionary registration data as a code word of the partial data string. Note that both the first partial data string having the first size (for example, 2 bytes) and the second partial data string having the second size (for example, 4 bytes) including the first partial data string are dictionaries. If it matches any of the registered data, the first compression unit 120 encodes the second partial data string instead of the first partial data string into a dictionary address, thereby reducing the size of the first compressed data. Can be small.

  On the other hand, if the partial data string does not match any of the dictionary registration data, the first compression unit 120 outputs the partial data string (called dictionary mismatch data) as it is as a code word of the partial data string (that is, , Pass through the partial data string).

  That is, the first compressed data includes a division pattern applied to the input data string 10, dictionary matching information indicating a match / mismatch between each partial data string corresponding to the division pattern and dictionary registration data, and the part Each code word of the data string (that is, dictionary address or dictionary mismatch data) is included.

  Furthermore, the first compression unit 120 updates the hash table and the dictionary table stored in the dictionary memory 110 using the input data sequence 10 in order to efficiently compress the future input data sequence. That is, the first compression unit 120 registers each partial data string obtained by dividing the input data string 10 in the dictionary table in association with the dictionary address. Then, the first compression unit 120 associates the hash value corresponding to this partial data string and this dictionary address and registers them in the hash table.

  The second compression unit 130 receives the input data string 10 from an external device (for example, a host) not shown. The second compressing unit 130 compares the current first compressed data with the past first compressed data, and the input data based on at least one of the past input data string corresponding to the compressed data. By compressing the column 10, the second compressed data is obtained. The second compressed data is different from the first compressed data in that it does not include at least the above dictionary address. The second compression unit 130 outputs the second compressed data to the selection unit 140.

  The selection unit 140 receives first compressed data from the first compression unit 120 and receives second compressed data from the second compression unit 130. The selection unit 140 obtains the compressed data 11 by selecting one of the first compressed data and the second compressed data having a smaller data size. The selection unit 140 outputs the compressed data 11 to an external device (for example, a NAND flash memory) not shown.

The compression apparatus in FIG. 1 performs data compression processing illustrated in FIG.
The first compression unit 120 generates first compressed data by compressing the input data string 10 using a hash function, a hash table, and a dictionary table stored in the dictionary memory 110 (step S201). The first compression unit 120 updates the hash table and dictionary table stored in the dictionary memory 110 using the input data string 10 (step S202).

  The second compressing unit 130 compares the current first compressed data with the past first compressed data, and the input data based on at least one of the past input data string corresponding to the compressed data. Second compressed data is generated by compressing the column 10 (step S203). Note that step S201, step S202, and step S203 may be performed in a different order from FIG.

  The selection unit 140 obtains the compressed data 11 by selecting one of the first compressed data generated in step S201 and the second compressed data generated in step S202, which has a smaller data size (step S204). . After step S204, the data compression process of FIG. 5 ends.

  Note that the compression device in FIG. 1 can be modified to the compression device illustrated in FIG. 9. The compression apparatus of FIG. 9 is different from the compression apparatus of FIG. 1 in that a dividing unit 650 is provided.

  The dividing unit 650 receives the original data string 12 from an external device (for example, a host) not shown. The dividing unit 650 obtains the input data string 10 by dividing the original data string 12 into a predetermined fixed size.

  The first compression unit 120 and the second compression unit 130 in FIG. 9 receive the input data string 10 having a fixed size from the division unit 650 and compress it.

  As illustrated in FIG. 6, the decompression apparatus according to the first embodiment includes a determination unit 310, a dictionary memory 320, a first decompression unit 330, a second decompression unit 340, and a selection unit 350. including. This decompressing device creates decompressed data 22 of a fixed size (for example, 8 bytes) by decompressing the compressed data 21 generated by the compressing device according to the present embodiment.

  The determination unit 310 receives the compressed data 21 from an external device (for example, a NAND flash memory) not shown. The determination unit 310 determines whether the compressed data 21 is compressed by the first method or the second method. The determination unit 310 notifies the selection unit 350 of the determined method, and outputs the compressed data 21 to the first decompression unit 330 and the second decompression unit 340.

  The dictionary memory 320 stores a dictionary table. The dictionary table corresponds to the dictionary table stored in the dictionary memory 110 of the compression apparatus of FIG. That is, the dictionary table stored in the dictionary memory 320 associates a dictionary address with a partial data string (also called dictionary registration data) obtained by dividing past decompressed data by a predetermined size. That is, in the dictionary table, dictionary registration data is associated with the dictionary address. Individual dictionary tables may be prepared for a plurality of sizes. The dictionary table stored in the dictionary memory 320 is updated using the decompressed data 22 as will be described later.

  The first decompressing unit 330 receives the compressed data 21 from the determining unit 310. The first decompressing unit 330 obtains first decompressed data by decompressing the compressed data 21 using a dictionary table stored in the dictionary memory 320. Note that when the compressed data 21 is compressed by the second method, the first decompressing unit 330 cannot decompress the compressed data 21 correctly. The first decompression unit 330 outputs the first decompression data to the selection unit 350.

  When the compressed data 21 is compressed by the first method, the compressed data 21 includes the division pattern applied to the pre-compression data corresponding to the compression data 21 and each partial data string corresponding to the division pattern. Dictionary match information indicating the match / mismatch between the dictionary registration data and each code word of the partial data string (that is, dictionary address or dictionary mismatch data).

  The first decompressing unit 330 extracts dictionary match information from the compressed data 21, and further extracts codewords of each partial data string from the compressed data 21 based on the dictionary match information. Then, the first decompressing unit 330 determines whether the code word of each partial data string corresponds to a dictionary address or dictionary mismatch data based on the dictionary match information. If the code word of the partial data string corresponds to the dictionary address, the first decompressing unit 330 treats the dictionary registration data associated with the dictionary address in the dictionary table as the partial data string. On the other hand, if the code word of the partial data string corresponds to the dictionary mismatch data, the first decompressing unit 330 treats the dictionary mismatch data as the partial data string.

  The second decompression unit 340 receives the compressed data 21 from the determination unit 310. The second decompressing unit 340 obtains second decompressed data by decompressing the compressed data 21 based on at least one of past compressed data and past decompressed data. Note that when the compressed data 21 is compressed by the first method, the second decompression unit 340 cannot correctly decompress the compressed data 21. When the compressed data 21 is compressed by the second method, the compressed data 21 does not include at least the aforementioned dictionary address. The second decompression unit 340 outputs the second decompressed data to the selection unit 350.

  The selection unit 350 is notified of the method determined by the determination unit 310, receives the first decompressed data from the first decompressing unit 330, and receives the second decompressed data from the second decompressing unit 340. The selection unit 350 obtains the decompressed data 22 by selecting one of the first decompressed data and the second decompressed data that corresponds to the method determined by the determination unit 310. The selection unit 350 outputs the decompressed data 22 to an external device (for example, a host) not shown.

  Further, the selection unit 350 updates the dictionary table stored in the dictionary memory 320 using the decompressed data 22. That is, the selection unit 350 registers each partial data string obtained by dividing the decompressed data 22 in the dictionary table in association with the dictionary address.

  Note that the selection unit 350 may have the same or similar function as the determination unit 310. That is, the selection unit 350 determines whether the compressed data 21 is compressed by the first method or the second method, and corresponds to the determined method of the first decompressed data and the second decompressed data. One may be selected. In this case, the determination unit 310 can be omitted.

The compression apparatus in FIG. 6 performs data decompression processing illustrated in FIG.
The determination unit 310 determines whether the compressed data 21 is compressed by the first method or the second method (step S401). The first decompressing unit 330 obtains first decompressed data by decompressing the compressed data 21 using the dictionary table stored in the dictionary memory 320 (step S402).

  The second decompression unit 340 obtains the second decompressed data by decompressing the compressed data 21 based on at least one of the past compressed data and the past decompressed data (step S403). Note that step S402 and step S403 may be executed in a different order from FIG.

  The selection unit 350 selects one of the first decompressed data generated in step S402 and the second decompressed data generated in step S403, which corresponds to the method determined in step S401. Is obtained (step S404).

  The selection unit 350 updates the dictionary table stored in the dictionary memory 320 using the decompressed data 22 obtained in step S404 (step S405). After step S405, the data decompression process in FIG. 7 ends.

  As illustrated in FIG. 8, by connecting the compression device of FIG. 1 and the decompression device of FIG. 6 to the storage 500, a storage device for storing an input data string with lossless compression is formed. According to the storage apparatus of FIG. 8, the compressed data 11 generated by the compression apparatus of FIG. 1 is written to the storage 500, and the compressed data 21 read from the storage 500 is expanded by the expansion apparatus of FIG. The storage device in FIG. 8 corresponds to, for example, an SSD, and the storage 500 corresponds to, for example, a NAND flash memory. Note that this embodiment is applicable not only to SSDs but also to HDDs. For example, the storage 500 may be a magnetic disk, and the compression device of FIG. 1 and the decompression device of FIG. 6 may be incorporated in the hard disk controller.

  The hardware configuration of the storage apparatus of FIG. 8 is illustrated in FIG. 18 includes an SSD controller 901, an input / output I / F (interface) 904, and a NAND flash memory 905. The SSD controller 901, the input / output I / F 904, and the NAND flash memory 905 are connected to each other by a bus.

  The SSD controller 901 performs various controls such as reading and writing of the NAND flash memory 905. The SSD controller 901 includes a compression device 902 and a decompression device 903. The compression device 902 corresponds to, for example, the compression device in FIG. 1, and the expansion device 903 corresponds to, for example, the expansion device in FIG.

  The input / output I / F 904 exchanges data between the storage apparatus of FIG. 18 and an external apparatus (for example, a host) not shown. The input / output I / F 904 can be mounted using, for example, a USB. The NAND flash memory 905 corresponds to the storage 500 in FIG.

  As described above, the compression apparatus according to the first embodiment uses an input data string using a dictionary in which partial data strings obtained by dividing a past input data string corresponding to compressed data by a predetermined size are registered. Is compressed to generate first compressed data. Further, the compression apparatus compares the current first compressed data and the past first compressed data, and compresses the input data string based on at least one of the past input data strings, thereby generating the second data Generate compressed data. The compression apparatus selects one of the first compressed data and the second compressed data having a smaller data size. That is, when the compression rate of the first compressed data is low (for example, when most of the partial data sequence obtained by dividing the input data sequence does not match the dictionary registration data), the compression device converts the second compressed data. You can choose.

  Therefore, according to this compression apparatus, it is possible to losslessly compress the input data string at a high compression rate. In addition, this compression apparatus can be mounted with almost no decrease in throughput and increase in hardware compared to a conventional compression apparatus that generates first compressed data but does not generate second compressed data.

  The decompression device according to the present embodiment can decompress the compressed data generated by the compression device according to the present embodiment. The storage apparatus according to the present embodiment can achieve high responsiveness and reliability by storing this compressed data.

(Second Embodiment)
In the compression apparatus according to the second embodiment, the first compression unit 120 generates first compressed data having the structure illustrated in FIG. In the example of FIG. 10, 8 bytes data “ABCDEF12” is handled as the input data string 10.

  The first compression unit 120 performs dictionary search processing on the partial data strings “ABCDEF12”, “ABCD”, “EF12”, “AB”, “CD”, “EF”, and “12” of the input data string 10, respectively. . Then, it is assumed that the partial data strings “ABCD” and “12” match any of the dictionary registration data, and the remaining partial data string “EF” does not match any of the dictionary registration data.

  The first compression unit 120 divides the pattern applied to the input data string 10 (for example, 4 bytes | 2 bytes | 2 bytes), and matches / mismatches each partial data string corresponding to the divided pattern and the dictionary registration data ( For example, dictionary match information indicating “match | mismatch | match” is output. Specifically, when the 8-byte input data string 10 is divided in units of 2 bytes, 4 bytes, and 8 bytes, the appearance pattern of the partial data string and each partial data string corresponding to the divided pattern and the dictionary registration data The total number of matching pattern combinations is 26. Therefore, the dictionary match information can be expressed in 5 bits.

  In addition, the first compression unit 120 sets the dictionary address associated with the dictionary registration data corresponding to the partial data strings “ABCD” and “12” corresponding to the dictionary matching data to the partial data strings “ABCD” and “12”. Output as codeword. The data size of the dictionary address depends on the number of entries in the dictionary table. Specifically, if 256 dictionary registration data are managed by the dictionary table, the dictionary address can be expressed in 8 bits.

  Further, the first compression unit 120 outputs the partial data string “EF” corresponding to the dictionary mismatch data as it is as the code word of the partial data string. Specifically, since the data size of the partial data string “EF” is 2 bytes, the data size of the code word is also 2 bytes (16 bits).

  Therefore, according to the example of FIG. 10, the data size of the input data string 10 is 8 bytes (64 bits), whereas the data size of the first compressed data is about 3.6 bytes (37 bits). Therefore, the area saving ratio (1-data size after compression / data size before compression) of the first compressed data can be estimated to be about 42% (= 1-37 / 64).

  As described above, the compression apparatus according to the second embodiment uses the dictionary in which a partial data sequence obtained by dividing a past input data sequence corresponding to compressed data by a predetermined size is registered. Is compressed to generate first compressed data. Although the first compressed data needs to include dictionary matching information, a high compression ratio can be achieved when most of the partial data string obtained by dividing the input data string matches the dictionary registration data.

  Note that the occurrence frequency of dictionary match information and dictionary mismatch data is considered to be biased. Therefore, the first compression unit can improve the expected compression rate of the first compressed data by further applying variable length coding such as Huffman coding to the data.

(Third embodiment)
The compression apparatus according to the third embodiment is different from the compression apparatus of FIG. 1 or FIG. 6 in that the second compression unit 730 illustrated in FIG. 11 is provided. The second compression unit 730 receives the first compressed data from the first compression unit 120. The second compression unit 730 obtains second compressed data by compressing the input data sequence 10 based on the comparison between the current first compressed data and the previous first compressed data. The second compression unit 730 outputs the second compressed data to the selection unit 140. The second compression unit 730 includes a storage unit 731, a comparison unit 732, and an encoding unit 733.

  The storage unit 731 stores the previous first compressed data. The previous first compressed data stored in the storage unit 731 is read by the comparison unit 732. Note that the previous first compressed data stored in the storage unit 731 may be a part of the compressed data. For example, the dictionary match information and the dictionary address in the previous first compressed data may be stored in the storage unit 731, and the dictionary mismatch data in the previous first compressed data may not be stored in the storage unit 731.

  The comparison unit 732 receives the current first compressed data from the first compression unit 120 and receives the previous first compressed data from the storage unit 731. The comparison unit 732 compares the current first compressed data with the previous first compressed data. Specifically, the comparison unit 732 corresponds to each information element (for example, dictionary match information, dictionary address, and dictionary mismatch data) included in the current first compressed data included in the past first compressed data. It is determined whether or not the information element matches. The comparison unit 732 notifies the encoding unit 733 of the comparison result.

  In accordance with the comparison result notified by the comparison unit 732, the encoding unit 733 matches the corresponding information element included in the past first compressed data among the information elements included in the current first compressed data. Is replaced with a flag (referred to as a skip flag in the following description) to generate second compressed data. The information element replaced by the skip flag is skipped from being transmitted to the decompression device.

  Specifically, as illustrated in FIG. 12, among the information elements included in the current first compressed data, dictionary match information (Flag # 1), dictionary address (Adrs # 1), and dictionary mismatch data (Data). When # 1) matches the corresponding information element included in the past first compressed data (that is, dictionary match information (Flag # 0), dictionary address (Adrs # 0), and dictionary mismatch data (Data # 0)) To do. In this case, the encoding unit 733 generates the second compressed data by replacing the entire current first compressed data with the skip flag. For example, when initialization using a specific value (such as 0) is performed, the same value may appear continuously in the input data string 10. In such a case, the same dictionary match information, dictionary address, and dictionary mismatch data appear continuously in the first compressed data. Therefore, the selection unit 140 can achieve a high compression rate by selecting and outputting the second compressed data.

  Alternatively, as illustrated in FIG. 13, among the information elements included in the current first compressed data, the dictionary match information (Flag # 1) and the dictionary address (Adrs # 1) are included in the past first compressed data. It is assumed that the corresponding information elements included (that is, the dictionary match information (Flag # 0) and the dictionary address (Adrs # 0)) match. In this case, the encoding unit 733 generates second compressed data by replacing the dictionary match information and the dictionary address in the current first compressed data with a skip flag. For example, a processor included in an external device (such as a host) connected to the compression device according to the present embodiment may handle data in units of the size of powers of 2 bytes such as 2 bytes, 4 bytes, and 8 bytes. 10 may have periodicity depending on the size. In such a case, the same dictionary matching information and dictionary address may appear successively in the first compressed data. Therefore, the selection unit 140 can achieve a high compression rate by selecting and outputting the second compressed data.

  As described above, the compression apparatus according to the third embodiment replaces the information element that matches the previous first compressed data in the current first compressed data with the skip flag to replace the second compressed data. Is generated. Therefore, according to this compression apparatus, when the same information element appears continuously in the first compressed data, a high compression ratio can be achieved by selecting and outputting the second compressed data.

  In order to implement the second compression unit included in the compression apparatus according to the present embodiment, in addition to the encoding unit that performs encoding (that is, compression), the storage unit (that is, the previous first compression) Hardware for storing part or all of data) and a comparison unit (that is, each information element included in the current first compressed data is compared with a corresponding information element included in the previous first compressed data) Hardware) is required. However, the influence (cost, power consumption, circuit area, etc.) due to the addition of the storage unit and the comparison unit is limited.

(Fourth embodiment)
The compression device according to the fourth embodiment is different from the compression device of FIG. 1 or 6 in that the second compression unit 830 illustrated in FIG. 14 is provided. The second compression unit 830 receives the input data string 10 from an external device (for example, a host) not shown. The second compression unit 830 obtains the second compressed data by compressing the input data sequence 10 based on the mode value of the specific size in the previous input data sequence corresponding to the compressed data. The second compression unit 830 outputs the second compressed data to the selection unit 140. Second compression unit 830 includes a storage unit 831, a comparison unit 832, a mode value calculation unit 833, and an encoding unit 834.

  The storage unit 831 stores the previous input data string. The previous input data string stored in the storage unit 831 is read by the mode value calculation unit 833. The mode value calculation unit 833 reads the previous input data string from the storage unit 831. The mode value calculation unit 833 calculates the mode value by counting the appearance frequency of each data pattern of the specific size based on the partial data sequence obtained by dividing the previous input data sequence by the specific size. The mode value calculation unit 833 notifies the mode value to the comparison unit 832.

  The comparison unit 832 receives the input data string 10 and is notified of the mode value by the mode value calculation unit 833. The comparison unit 832 compares each partial data string obtained by dividing the input data string 10 with the specific size with the mode value. Specifically, the comparison unit 832 determines whether each of the partial data strings obtained by dividing the input data string 10 with a specific size matches the mode value. The comparison unit 832 notifies the comparison unit 834 of the comparison result.

  The encoding unit 834 generates the second compressed data by compressing the input data string 10 according to the comparison result notified by the comparison unit 832. The second compressed data includes mode value match information indicating a match / mismatch between each partial data string obtained by dividing the input data string 10 by the specific size and the mode value, and the mode value among the partial data strings. And a code word of a partial data string that does not match (hereinafter referred to as mode value mismatch data). That is, the second compressed data does not include the code word of the partial data string that matches the mode value among the partial data strings.

  The compression apparatus according to this embodiment performs data compression processing illustrated in FIG. In the example of FIG. 15, the data size of the input data string 10 is 16 bytes, and the specific size is 1 byte. This compression apparatus tries both the first method based on dictionary matching and the second method based on mode matching, and selects one compressed data with a small data size, thereby stably increasing the compression rate. Can be achieved. For some types of data, the mode value in a given block is likely to match the mode value in the next block, so when handling such data, high compression is achieved by the second method based on mode value matching. Rate can be achieved.

  In the example of FIG. 15, an information element (Codec) indicating a codec (compression method) is included in both the first compressed data and the second compressed data. When the codec is either the first method or the second method, the information element can be expressed by 1 bit. The decompression device can determine whether the compressed data is compressed by the first method or the second method based on the information element. In addition, the first compressed data includes the aforementioned dictionary match information (Flag), dictionary address (Adrs), and dictionary mismatch data (Data). On the other hand, the second compressed data includes mode value match information (Match) and mode value mismatch data (Data).

  The mode value match information expresses, in 1 bit (1/0), whether or not each of 16 partial data strings obtained by dividing the 16-byte input data string 10 by 1 byte matches the 1-byte mode value. Therefore, the data size of the mode value match information is 2 bytes (16 bits). The mode value mismatch data is prepared to transmit a partial data sequence that does not match the mode value among the 16 partial data sequences, and each has a data size of 1 byte. In the example of FIG. 15, since the two partial data strings did not match the mode value, the data size of the mode value mismatch data is 2 bytes in total. Therefore, according to the example of FIG. 15, the data size of the second compressed data is about 4 bytes in total.

  The specific size is not limited to 1 byte. The second compression unit 830 may calculate a mode value for a plurality of types of specific sizes, and may select a mode value that minimizes the data size of the second compressed data.

  For example, the compression apparatus according to the present embodiment may perform data compression processing as shown in FIG. In the example of FIG. 16, the data size of the input data string 10 is 8 bytes, and the specific size is 2 bytes. The input data string 10 is given to the compression device in the order of “00120000”, “00140000”, and “00160000”. The compression apparatus tries both “00120000”, “00140000”, and “00160000” by using both the first method based on dictionary matching and the second method based on mode matching, and compressing one of the smaller data sizes. By selecting data, a high compression ratio can be achieved stably.

  In the example of FIG. 16, the dictionary mismatch data and the mode value mismatch data are the same for the compressed data of “00140000” and “00160000”. Therefore, the compression apparatus can select compressed data having a small data size by comparing the data size of the dictionary match information and dictionary address with the data size of the mode value match information.

  In the example of FIG. 15, an information element (Codec) indicating a codec (compression method) is used for both the first compressed data and the second compressed data. For example, as shown in FIG. Further, the dictionary matching information (Flag) may be extended so that the information element can be expressed.

In the example of FIG. 17, an information element (Flag) indicating a codec is included in both the first compressed data and the second compressed data. Since this information element (Flag) corresponds to an extended version of dictionary matching information, when the codec is the first method, it corresponds to the division pattern applied to the input data string 10 and the division pattern. The match / mismatch between each partial data string and the dictionary registration data is further shown. In other words, this information element (Flag) can take a total of N + 1 types of values including N (for example, 26) types of values corresponding to the dictionary matching information and a value indicating that the codec is the second system. This information element (Flag) can be expressed by ceil (log ₂ (N + 1)) bits. Note that ceil (x) is a ceiling function that returns the smallest integer equal to or greater than x.

  The decompression device can determine whether the compressed data is compressed by the first method or the second method based on the information element (Flag). In addition, the first compressed data includes the aforementioned dictionary address (Adrs) and dictionary mismatch data (Data). On the other hand, the second compressed data includes mode value match information (Match) and mode value mismatch data (Data).

  As described above, the compression apparatus according to the fourth embodiment generates the second compressed data based on the comparison result between the mode value of the specific size in the previous input data string and the current input data string. . The second compressed data does not include the code word of the partial data string that matches the mode value among the partial data strings obtained by dividing the input data string by the specific size. Therefore, according to this compression apparatus, when the mode value in the input data string continuously matches, a high compression ratio can be achieved by selecting and outputting the second compressed data.

  In order to implement the second compression unit included in the compression device according to the present embodiment, in addition to the encoding unit that performs encoding (that is, compression), the storage unit (that is, the previous input data string is changed). Hardware for saving), mode value calculation unit (ie, hardware for calculating mode value) and comparison unit (ie, mode value for comparing the mode value with the current first input data string) Hardware). However, the influence (cost, power consumption, circuit area, etc.) due to the addition of the storage unit, the mode value calculation unit, and the comparison unit is limited.

  The codecs described in the above embodiments may be used in combination as necessary. For example, a combination of the first to fourth embodiments, a method based on dictionary matching, a method for replacing a part of compressed data based on dictionary matching with a skip flag, and a method based on mode matching A method for minimizing the size may be selectable.

  At least a part of the processing of each of the above embodiments can also be realized by using a general-purpose computer as basic hardware. A program for realizing the above processing may be provided by being stored in a computer-readable recording medium. The program is stored in the recording medium as an installable file or an executable file. Examples of the recording medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. The recording medium may be any recording medium as long as it can store the program and can be read by the computer. The program for realizing the above processing may be stored on a computer (server) connected to a network such as the Internet and downloaded to the computer (client) via the network.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 10 ... Input data sequence 11, 21 ... Compressed data 12 ... Original data sequence 22 ... Decompression data 110, 320 ... Dictionary memory 120 ... First compression unit 130, 730, 830 ... second compression unit 140,350 ... selection unit 310 ... determination unit 330 ... first decompression unit 340 ... second decompression unit 500 ... storage 650 ... division Units 731, 831 ... Storage unit 732, 832 ... Comparison unit 733, 834 ... Coding unit 833 ... Mode value calculation unit 901 ... SSD controller 902 ... Compression device 903 ...・ Extension device 904: Input / output I / F
905 ... NAND flash memory

Claims (13)

A dictionary memory that stores a dictionary table that associates dictionary addresses with dictionary registration data of a predetermined size;
By compressing the input data string based on a comparison result between each of the partial data strings obtained by dividing the input data string by the predetermined size and the dictionary registration data, at least one of the partial data strings is the dictionary registration data. A first compression unit that generates first compressed data that includes a dictionary address associated with at least one of the dictionary registration data,
A comparison between the first compressed data and the past first compressed data, and a second that does not include the dictionary address by compressing the input data string based on at least one of the past input data strings A second compression unit for generating the compressed data of
And a selection unit that obtains compressed data of the input data string by selecting one of the first compressed data and the second compressed data having a smaller data size.   The compression apparatus according to claim 1, further comprising a dividing unit that obtains the input data string by dividing the original data string at a fixed size. The second compression unit includes
A storage unit for storing the previous input data string;
A calculation unit that calculates a mode value by counting the appearance frequency of each data pattern of the specific size based on a partial data sequence obtained by dividing the previous input data sequence by a specific size;
A comparison unit that compares each of the partial data strings obtained by dividing the input data string with the specific size with the mode value;
By encoding the input data string based on the comparison result between the input data string and the mode value, the partial data string obtained by dividing the input data string by the specific size matches the mode value. A coding unit that generates the second compressed data including the mode value matching information indicating the mismatch and the code word of the mode value mismatch data that does not match the mode value in the partial data string. Prepare
The compression apparatus according to claim 1. The second compression unit includes
A storage unit for storing a part or all of the information elements included in the previous first compressed data;
A comparison unit that compares the first compressed data with an information element stored in the storage unit;
Based on the comparison result between the first compressed data and the information element stored in the storage unit, information that matches the information element stored in the storage unit among the information elements included in the first compressed data An encoding unit that generates the second compressed data by replacing an element with a skip flag.
The compression apparatus according to claim 1. The first compressed data includes a division pattern applied to the input data string, dictionary matching information indicating matching / mismatching between each partial data string corresponding to the division pattern and dictionary registration data, and the partial data Each codeword of the sequence, and
The code word of the partial data string is a dictionary address associated with the dictionary registration data if the partial data string corresponds to dictionary matching data that matches any of the dictionary registration data. If it corresponds to dictionary mismatch data that does not match any of the dictionary registration data, it is the dictionary mismatch data.
The compression apparatus according to claim 4.   If the first compressed data and the previous first compressed data match in the dictionary match information and the dictionary address, the encoding unit includes the dictionary match information included in the first compressed data and The compression apparatus according to claim 5, wherein the second compressed data is generated by replacing the dictionary address with the skip flag.   The encoding unit is included in the first compressed data if the first compressed data and the previous first compressed data match in the dictionary match information, the dictionary address, and the dictionary mismatch data. The compression apparatus according to claim 5, wherein the second compressed data is generated by replacing the dictionary match information, the dictionary address, and the dictionary mismatch data with the skip flag.   2. The compression device according to claim 1, wherein each of the first compressed data and the second compressed data includes an information element indicating whether a codec is a first scheme or a second scheme.   The first compressed data and the second compressed data indicate whether the codec is the first method or the second method, and when the codec is the first method, the input data The compression apparatus according to claim 1, further comprising: a division pattern applied to the column, and an information element further indicating a match / mismatch between each partial data sequence corresponding to the split pattern and the dictionary registration data. A dictionary memory that stores a dictionary table that associates dictionary addresses with dictionary registration data of a predetermined size;
A code word of each partial data string obtained by dividing the pre-compression data corresponding to the compressed data according to a predetermined division pattern is extracted from the compressed data, and if the partial data string corresponds to one of the dictionary addresses, the dictionary address A first decompression unit that generates first decompressed data by treating the dictionary registration data associated with the data as the partial data string;
A second decompression unit that generates second decompressed data by decompressing the compressed data based on at least one of past compressed data and past decompressed data;
Selecting the first decompressed data if the codec of the compressed data is a first scheme, and selecting the second decompressed data if the codec of the compressed data is a second scheme, A decompression device comprising: a selection unit that obtains decompressed data of the compressed data. A storage, a compression device, and a decompression device;
The compression device includes:
A dictionary memory that stores a dictionary table that associates dictionary addresses with dictionary registration data of a predetermined size;
By compressing the input data string based on a comparison result between each of the partial data strings obtained by dividing the input data string by the predetermined size and the dictionary registration data, at least one of the partial data strings is the dictionary registration data. A first compression unit that generates first compressed data that includes a dictionary address associated with at least one of the dictionary registration data,
A comparison between the first compressed data and the past first compressed data, and a second that does not include the dictionary address by compressing the input data string based on at least one of the past input data strings A second compression unit for generating the compressed data of
A selection unit that obtains compressed data of the input data string by selecting one of the first compressed data and the second compressed data having a smaller data size; and
The compressed data of the input data string is read and written in the storage,
The extension device
A dictionary memory that stores a dictionary table that associates dictionary addresses with dictionary registration data of a predetermined size;
A code word of each partial data string obtained by dividing the pre-compression data corresponding to the compressed data read from the storage according to a predetermined division pattern is extracted from the compressed data, and the partial data string is set to one of the dictionary addresses. A first decompression unit that generates first decompressed data by handling dictionary registration data associated with the dictionary address as the partial data string, if corresponding,
Second decompression that generates second decompressed data by decompressing compressed data read from the storage based on at least one of compressed data read from the storage in the past and past decompressed data And
If the codec of the compressed data read from the storage is the first method, the first decompressed data is selected, and if the codec of the compressed data read from the storage is the second method A selection unit that obtains decompressed data of the compressed data read from the storage by selecting the second decompressed data;
Storage device. The storage is a NAND flash memory,
The compression device and the decompression device are incorporated in an SSD (Solid State Drive) controller.
The storage apparatus according to claim 11. The storage is a magnetic disk;
The compression device and the decompression device are incorporated in a hard disk controller.
The storage apparatus according to claim 11.

Images