JP5413153B2 - Data compression apparatus, data expansion apparatus, data compression program, and data expansion program - Google Patents

Description

  The present invention relates to a data compression device, a data expansion device, a data compression program, and a data expansion program.

  As a method for compressing text data (plain text format data, CSV (Comma Separated Values) format data, XML (eXtensible Markup Language) data, etc.), a dictionary-type encoding method is known. The dictionary-type coding method creates a dictionary by assigning a predetermined number (code) to each character or character string that appears in the data string to be compressed, and encodes the actual input character based on this dictionary. I do. The lexicographic encoding method includes static lexicographic encoding (BPE (Byte Pair Encoding), STVF (Suffix-Tree based VF coding), etc.) that requires dictionary storage, and dynamic dictionary that does not require dictionary storage. There is formula coding (LZ77, LZ78, etc.). For data that contains a lot of duplicate value strings, such as data output from a database, apply dynamic lexicographic encoding after applying static lexicographic encoding. It is known that the compression rate is improved by applying only coding. It is also known that static lexicographic coding is a method suitable for pattern search without decompressing compressed data.

  Further, block stream processing is known in which blocks obtained by dividing input data into appropriate sizes are sequentially processed (stream processing). Many static lexicographic coding methods perform coding in two passes, dictionary creation and coding, so sequential processing cannot be performed. However, by applying block stream processing, only a limited calculation area can be obtained. Sequential processing can be performed without using it. In addition, the block stream processing has an advantage that filtering using a lighter index can be performed as compared with stream processing (character stream processing and word stream processing) in smaller units.

  Therefore, a compression / decompression algorithm that improves the compression rate in block stream processing has been proposed (for example, Patent Document 1). Also, a technique for compressing data using a dictionary in word stream processing has been proposed (for example, Patent Document 2).

JP-A-9-214353 JP-A-11-168390

  However, when performing lexicographic encoding of input data using block stream processing, the techniques of Patent Documents 1 and 2 have the following problems. The technique of Patent Document 1 does not consider the reduction of the dictionary size in each block, so there is a possibility that a sufficient compression rate cannot be obtained. Since the technique of Patent Document 2 performs word stream processing, it loses an advantage specific to block stream processing. In addition, although it is possible to naturally extend the technique of Patent Document 2 to block stream processing, there is a possibility that a sufficient compression rate may not be obtained due to an increase in the dictionary of each block.

  This case has been made in view of the above circumstances, and provides a data compression device, a data decompression device, a data compression program, and a data decompression program that improve the compression rate of text data in block stream processing. Objective.

  In order to solve the above problems, a data compression device disclosed in the specification includes an input unit that receives input of text data, a division unit that divides the text data into a plurality of blocks based on a predetermined rule, a character string, and a code And a character string that is not registered in the reference dictionary among the character strings that appear in the processing target block, and the processing target block in the reference dictionary is based on the reference dictionary that is the dictionary data stored in association with Based on the created difference dictionary and the reference dictionary, a processing target dictionary that is dictionary data based on the difference dictionary generation unit that generates a difference dictionary that is dictionary data associated with a code associated with a character string that does not appear The processing target dictionary generation unit for generating the processing target and the generated processing target dictionary are referred to, and the character string appearing in the processing target block is replaced with a corresponding code. And in comprises a compression unit for compressing the target block, and the data of the processing target block in which the compression unit is compressed, and an output unit for outputting a differential dictionary containing the product.

In order to solve the above problem, a data decompression device disclosed in the specification is a processing unit that divides decompression target data into a plurality of blocks based on a predetermined rule, and is a processing target among the plurality of blocks. Based on an acquisition unit that acquires a difference dictionary that is dictionary data in which a character string and a code are stored in association with each other from a processing target block, a reference dictionary that is dictionary data, and the difference dictionary acquired by the acquisition unit , the process target dictionary generating unit for generating a processed dictionary is dictionary data, based on the generated processed dictionary, the target block by replacing the code appearing in the target block into the corresponding character string and a decoder for decoding, the difference dictionary, a character string not registered in the reference dictionary in the character string appearing in the target block before compression The dictionary data is associated with a code associated with a character string that does not appear in the processing target block before compression in the reference dictionary, and the processing target block refers to the processing target dictionary, and the compression target dictionary It is compressed by replacing the character string appearing in the processing target block with the corresponding code .

  In order to solve the above problems, a data compression program disclosed in the specification includes an input step of accepting text data input to a computer, and a dividing step of dividing the text data into a plurality of blocks based on a predetermined rule. Based on the reference dictionary that is the dictionary data in which the character string and the code are stored in association with each other, the character string that is not registered in the reference dictionary among the character strings that appear in the processing target block, and the reference dictionary Dictionary data based on a difference dictionary generation step of generating a difference dictionary that is dictionary data associated with a code associated with a character string that does not appear in the processing target block, and the created difference dictionary and the reference dictionary A processing target dictionary generation step for generating a processing target dictionary, and the generated processing target dictionary By outputting a compression step for compressing the processing target block by replacing the character string appearing in the block with a corresponding code, the data of the processing target block compressed by the compression unit, and the generated difference dictionary Steps are executed.

In order to solve the above-described problem, a data decompression program disclosed in the specification includes a computer dividing a decompression target compressed data into a plurality of blocks based on a predetermined rule, and a processing target among the plurality of blocks. An acquisition step of acquiring a difference dictionary that is dictionary data in which a character string and a code are stored in association with each other, a reference dictionary that is dictionary data, and the difference dictionary acquired in the acquisition step the basis, the process target dictionary generating step of generating a processed dictionary is dictionary data, based on the generated the processing object dictionary, the processing by replacing the code appearing in the target block into the corresponding character string a decoding step of decoding the target block is executed, the difference dictionary, appear in the target block before compression Is a dictionary data in which a character string that is not registered in the reference dictionary is associated with a code that is associated with a character string that does not appear in the processing target block before compression in the reference dictionary, The processing target block is compressed by referring to the processing target dictionary and replacing a character string appearing in the processing target block before compression with a corresponding code .

  According to the data compression device and the data compression program disclosed in the specification, the compression rate of text data can be improved in block stream processing.

  According to the data decompression apparatus and the data decompression program disclosed in the specification, compressed data in which the compression rate of text data is improved can be decompressed in block stream processing.

It is a figure which shows an example of the system configuration | structure containing the data compression apparatus which concerns on an Example. It is a figure which shows an example of the hardware constitutions of a data compression apparatus. It is a functional block diagram which shows an example of a means which implement | achieves the function which a data compression apparatus has. It is a figure explaining the outline | summary of a dictionary type encoding system. It is a flowchart which shows an example of the data compression process which a data compression apparatus performs. It is a flowchart which shows an example of a dictionary update process. It is a figure which shows an example of to-be-compressed data. It is a figure for demonstrating an example of the process which produces compressed data. It is a flowchart which shows an example of a difference dictionary creation process. It is a figure for demonstrating an example of the process which produces compressed data. It is a figure for demonstrating an example of the process which produces compressed data. It is a figure for demonstrating an example of the process which produces compressed data. It is a figure for demonstrating an example of the process which produces compressed data. It is a figure for demonstrating an example of the process which produces compressed data. It is a figure for demonstrating an example of the process which produces compressed data. It is a figure for demonstrating an example of the process which produces compressed data. It is a figure which shows an example of compressed data. It is a figure for demonstrating an example of the compression data produced by the comparative example 1, the comparative example 2, and an Example. It is a figure for demonstrating an example of the compression data produced by the comparative example 1, the comparative example 2, and an Example. It is a schematic diagram which shows an example of the size comparison of the compression data by the comparative example 1, the comparative example 2, and an Example. 1 is a diagram illustrating an example of a system configuration including a data decompression apparatus according to an embodiment. It is a functional block diagram which shows an example of a means which implement | achieves the function which a data expansion | extension apparatus has. It is a flowchart which shows an example of a data expansion | extension process. It is a figure for demonstrating an example of the process which expand | extracts compressed data. It is a flowchart which shows an example of a dictionary restoration process. It is a figure for demonstrating an example of the process which expand | extracts compressed data. It is a figure for demonstrating an example of the process which expand | extracts compressed data. It is a figure which shows an example of the system configuration | structure containing a data compression / decompression apparatus.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.
[Data compression device]

  The data compression apparatus of this case will be described. FIG. 1 is a diagram showing an example of a system configuration including a data compression apparatus of the present case. As shown in FIG. 1, the data compression device 100 is connected to a storage device 10, a sensor device 20, and a data processing device 30 via a network 40.

  The storage device 10 is composed of a hard disk drive or the like, and stores data to be compressed (hereinafter referred to as data to be compressed) and compressed data.

  The sensor device 20 is, for example, an entrance / exit management device installed at a company employee entrance. The sensor device 20 transmits data acquired by the sensor to the data compression device 100. For example, the sensor device 20 acquires information on an ID card possessed by an employee, and transmits the employee number, name, and work location to the data compression device 100 as compressed data. The sensor device 20 may be a sensor network including a sensor and an information processing device that processes data acquired by the sensor.

  The data processing device 30 is composed of a personal computer or the like, and performs predetermined data processing such as computation on input data. The data processing device 30 transmits data subjected to predetermined processing to the data compression device 100 as compressed data. In addition, the data processing device 30 receives data compressed by the data compression device 100 from the data compression device 100.

  The network 40 is configured by, for example, a local area network (LAN), a wide area network (WAN), or the like. The network 40 transmits data transmitted by the sensor device 20, the data processing device 30, and the data compression device 100 to a transmission destination.

  The data compression device 100 receives data to be compressed from the storage device 10, the sensor device 20, and the data processing device 30. The compressed data is text data having a structure such as CSV format or XML format. The data compression apparatus 100 divides the data to be compressed into a plurality of blocks, executes compression processing for each block, and generates compressed data. The data compression device 100 stores the compressed data in the storage device 10. Alternatively, the data compression device 100 transmits the compressed data to the data processing device 30.

  Next, the hardware configuration of the data compression apparatus 100 will be described with reference to FIG. As shown in FIG. 2, the data compression apparatus 100 includes an input / output unit 101, a ROM (Read Only Memory) 102, a central processing unit (CPU) 103, and a RAM (Random Access Memory) 104.

  The input / output unit 101 receives data to be compressed from the storage device 10, the sensor device 20, and the data processing device 30. The input / output unit 101 outputs the compressed data to the storage device 10 and the data processing device 30. The ROM 102 stores a program for compressing data to be compressed. The CPU 103 reads and executes a program stored in the ROM 102. The RAM 104 stores temporary data used when executing the program. Further, the functions of the data compression apparatus 100 shown in FIG. 3 are realized by the calculation performed by the CPU 103 of the program stored in the ROM 102.

  Next, an example of the function of the data compression apparatus 100 will be described with reference to FIG. FIG. 3 is a functional block diagram showing an example of means for realizing the functions of the data compression apparatus 100.

  The data compression apparatus 100 includes a data acquisition unit 110, a dictionary creation unit 111, a difference dictionary creation unit 112, an encoding unit 113, and an output unit 114.

  The data acquisition unit 110 has a function as an input unit that receives compressed data (text data) transmitted from the storage device 10, the sensor device 20, the data processing device 30, and the like, and a plurality of compressed data based on a predetermined rule. It has a function as a dividing unit for dividing into blocks. The data acquisition unit 110 outputs the divided compressed data to the dictionary creation unit 111.

  The dictionary creation unit 111 (processing target dictionary generation unit) encodes data of a block to be compressed (hereinafter referred to as a processing target block) among a plurality of blocks obtained by dividing the compressed data based on a predetermined rule. A dictionary required at this time is created based on the data input from the data acquisition unit 110. The dictionary has a code item and a value item, and the dictionary creation unit 111 registers a character string that appears in the data of the processing target block in the value item, and codes a predetermined code associated with the character string. Register in the item. For example, if the input data is the data shown in FIG. 4A and the item to be encoded is “district”, the dictionary creation unit 111 associates a character string appearing in the district column with a predetermined code. In addition, the dictionary shown in FIG.

  The encoding unit 113 (compression unit) encodes data for each block using the dictionary created by the dictionary creation unit 111 and outputs the data to the output unit 114. For example, the encoding unit 113 encodes the data to be compressed by replacing the character string of “district” in FIG. 4A with a code using the dictionary shown in FIG. The encoded data is as shown in FIG.

  The difference dictionary creating unit 112 (difference dictionary creating unit) creates a difference dictionary and outputs it to the output unit 114. The difference dictionary creating unit 112 creates a difference dictionary based on the dictionary used for encoding the reference block different from the processing target block and the encoding target character string included in the processing target block. Similar to the dictionary used for encoding, the difference dictionary has a code item and a value item. The difference dictionary creation unit 112 registers character strings that are not registered in the dictionary used for encoding the reference block among the character strings that appear in the processing target block, in the value item of the difference dictionary. In addition, the difference dictionary creation unit 112 associates a character string registered in the difference dictionary with a character string that is registered in the reference block dictionary but does not appear in the processing target block. Assign a code. For example, it is assumed that the reference block dictionary is FIG. 4 (4) and the encoding target character string included in the processing target block is FIG. 4 (5). In this case, the difference dictionary creation unit 112 registers the character string “New York” that appears in the processing target block but is not registered in the reference block dictionary in the difference dictionary. Further, the difference dictionary creating unit 112 converts the character string “3” assigned to the character string “Las Vegas” that does not appear in the processing target block but is registered in the reference block into the character string “ Assign to “New York”. As a result, the difference dictionary is as shown in FIG.

  The output unit 114 outputs the difference dictionary and the encoded data to the storage device 10 or the data processing device 30 for each block.

  Next, an example of processing executed by the data compression apparatus 100 will be described with reference to FIGS. 5 and 6. FIG. 5 is a flowchart illustrating an example of the compression process executed by the data compression apparatus 100. In this embodiment, a front block in two consecutive blocks is described as a reference block, and a rear block is described as a processing target block.

  The dictionary creation unit 111 initializes the dictionary Dic0 of the reference block with an empty table (step S10). This is because the reference block does not exist when the first block of the compressed data is the processing target block.

  Next, the dictionary creation unit 111 initializes a dictionary (processing target dictionary) Dic1 used for encoding the processing target block and a variable M for counting the number of records read (step S12). By the initialization, the dictionary Dic1 becomes an empty table, and the value of M becomes 0.

  The data acquisition unit 110 determines whether there is a record to be processed in the compressed data (step S14). If there is a record to be processed (step S14 / YES), the data acquisition unit 110 acquires the record and adds 1 to M (step S16).

  Next, the dictionary creation unit 111 executes a dictionary update process in order to create a dictionary for the processing target block (step S18). Here, the dictionary update processing in step S18 will be described with reference to FIG.

  FIG. 6 is a flowchart illustrating an example of a dictionary update process for creating a dictionary of processing target blocks.

  The dictionary creation unit 111 determines whether the character string included in the acquired record exists in the entry of the dictionary Dic1 (step S50). When the character string included in the acquired record does not exist in the entry of the dictionary Dic1 (step S50 / NO), the dictionary creation unit 111 newly registers an entry in which the character string and the code are associated with each other in the dictionary Dic1 (step S52). ), The update process is terminated. On the other hand, when the character string included in the acquired record exists in the entry of the dictionary Dic1, the dictionary creation unit 111 ends the update process.

  Returning to FIG. 5, the description of an example of the compression process will be continued. The data acquisition unit 110 determines whether the value of M is smaller than a predetermined value MB (step S20). Here, MB defines the number of records included in one block. The value of MB may be held in the data to be compressed, or may be determined in advance by the user. Further, the value of MB may be the same throughout all blocks, or may be different for each block.

  When the value of M is smaller than the value of MB, it means that all the records of the processing target block have not been read yet. Therefore, when the value of M is smaller than the value of MB (step S20 / YES), the data acquisition unit 110 returns to step S14 and continues the process.

  When the value of M is equal to the value of MB, it means that all records of the processing target block have been read. Therefore, when the data acquisition unit 110 determines that the value of M is equal to the value of MB (step S20 / NO), the difference dictionary creation unit 112 executes a difference dictionary creation process (step S22). Details of the difference dictionary creation processing will be described later.

  The output unit 114 outputs the difference dictionary Δ created by the difference dictionary creation unit 112 in the difference dictionary creation process (step S24). Next, the dictionary creation unit 111 merges the reference block dictionary Dic0 and the difference dictionary Δ to construct a dictionary for encoding, and sets it as a new dictionary Dic1 (step S26). Specifically, when there is an overlapping code between the dictionary Dic0 and the difference dictionary Δ, the character string of the dictionary Dic0 associated with the overlapping code is replaced with the character string of the difference dictionary Δ. If a code not registered in the dictionary Dic0 is registered in the difference dictionary Δ, an entry of the difference dictionary Δ is added to the dictionary Dic0. That is, the dictionary Dic1 used by the encoding unit 113 has the same code as the dictionary of the reference block in the character string defined in the dictionary used in the reference block among the character strings appearing in the processing target block. It becomes the assigned dictionary. The dictionary Dic1 is a dictionary in which codes associated with character strings that do not appear in the processing target block in the reference block dictionary are assigned to character strings that are not registered in the reference block dictionary.

  The encoding unit 113 encodes the character string appearing in the processing target block using the dictionary Dic1 updated in step S26 (step S28). The output unit 114 outputs the data encoded by the encoding unit 113 (step S30). The compressed data of the block to be processed is created by the processing of step S24 and step S30.

  The dictionary creation unit 111 initializes the dictionary Dic0 for the compression process of the next block (step S32), and returns to the process of step S12. In this flowchart, of two consecutive blocks, the front block is the reference block and the rear block is the processing target block. Therefore, since the block processed this time becomes the reference block for the next block to be processed, the dictionary Dic0 is initialized with the dictionary Dic1.

  If there is no record to be processed (step S14 / NO), the data acquisition unit 110 determines whether the value of M is 0 (step S34).

  The case where the value of M is 0 is a case where the next block does not exist. Accordingly, when the value of M is 0 (step S34 / YES), the data acquisition unit 110 ends the data compression process.

  The case where the value of M is not 0 is a case where all reading of data existing in the last processing target block is completed. Therefore, when the data acquisition unit 110 determines that the value of M is not 0 (step S34 / NO), the difference dictionary creation unit 112 is step S22, the output unit 114 is step S24 and step S30, and the encoding unit 113 is step. S28, the dictionary creation unit 111 executes the process of step S26. Since the process of step S22-step S30 is the same as the process of each step mentioned above, description is abbreviate | omitted. With the above processing, the compressed data is compressed including the difference dictionary and the encoded data for each block.

  Next, data compression by the above-described compression processing using specific data will be described with reference to FIGS. 7 to 17 and details of the difference dictionary creation processing will be described. FIG. 7 is an example of compressed data used in this description. In this embodiment, it is assumed that compressed data is generated by encoding a character string input to a district item. In addition, it is assumed that data is compressed by block stream processing with 3 records (that is, MB = 3) as one block.

  The data compression apparatus 100 performs the initialization process of steps S10 and S12 in FIG. Next, since there is a record to be processed (step S14 / YES), the data acquisition unit 110 acquires the record in the first row shown in FIG. 8A (step S16).

  Next, the dictionary creation unit 111 executes a process for updating the dictionary Dic1 (step S18). Since the dictionary Dic1 is an empty table at this time, there is no entry having “San Francisco” as a value item in the dictionary Dic1 (step S50 / NO). Accordingly, the dictionary creation unit 111 newly registers an entry in which the code “1” is assigned to San Francisco in the dictionary Dic1 (step S52), and ends the process. The updated dictionary Dic1 is as shown in FIG.

  Since M (= 1) <MB (= 3) (step S20 / YES) and the next record exists (step S14 / YES), the data acquisition unit 110 displays the second line shown in FIG. 8 (3). Is acquired (step S16).

  In the dictionary update process of FIG. C. Does not exist in the dictionary Dic1 (step S50 / NO), the dictionary creation unit 111 uses the Washington D.1. C. Is newly registered in the dictionary Dic1 (step S52), and the process ends. The updated dictionary Dic1 is as shown in FIG.

  Since M (= 2) <MB (= 3) (step S20 / YES) and the next record exists (step S14 / YES), the data acquisition unit 110 displays the third line shown in FIG. 8 (5). Enter the record.

  In the dictionary update process of FIG. C. Already exists in the dictionary Dic1 (step S50 / YES), the dictionary creation unit 111 does not newly register an entry in the dictionary Dic1, and ends the process.

  Since M (= 3) matches MB (= 3) (step S20 / NO), the difference dictionary creation unit 112 executes difference dictionary creation processing (step S22).

  Here, the difference dictionary creation process will be described using a specific example. FIG. 9 is a flowchart illustrating an example of the difference dictionary creation process.

  The difference dictionary creating unit 112 initializes the difference dictionary Δ with the reference block dictionary Dic0 (step S60). In the first block, the dictionary Dic0 is an empty set, so the difference dictionary Δ is also an empty set.

  Next, the difference dictionary creation unit 112 obtains a difference set Diff0 between the set of value items of the dictionary Dic0 of the reference block and the set of value items of the dictionary Dic1 that has been subjected to the update process (step S62). In FIG. 9, a set of value items in the dictionary Dic0 is represented by Dic0. val, a set of value items of the dictionary Dic1 is stored in the dictionary Dic1. It is described as val.

  Next, the difference dictionary creation unit 112 obtains a difference set Diff1 between the set of value items in the dictionary Dic1 and the set of value items in the dictionary Dic1 (step S64). Further, the difference dictionary creation unit 112 obtains a product set NoDiff of the set of value items in the dictionary Dic1 and the set of value items in the dictionary Dic0 (step S66).

  In the specific example shown in FIG. 8, a set Dic0. val is an empty set, and a set of value items Dic1. val = {San Francisco, Washington D. C. }. In this case, there is no value item that exists in the dictionary Dic0 but does not exist in the dictionary Dic1. Therefore, when the process of step S62 is executed, the difference set Diff0 becomes an empty set. The value items that are not present in the dictionary Dic0 but are present in the dictionary Dic1 are San Francisco and Washington D.C. C. It is. Therefore, when the process of step S64 is executed, the difference set Diff1 = {San Francisco, Washington D.D. C. }. Furthermore, since there is no common value item between the dictionary Dic0 and the dictionary Dic1, when the process of step S66 is executed, the product set NoDiff becomes an empty set.

  Next, the difference dictionary creation unit 112 determines whether Diff1 is an empty set (step S68). When Diff1 is not an empty set (Step S68 / NO), the difference dictionary creating unit 112 determines whether Diff0 is an empty set (Step S70).

  When Diff0 is an empty set (step S70 / YES), the difference dictionary creation unit 112 newly registers all elements of Diff1 in the difference dictionary Δ (step S76). Then, the difference dictionary creation unit 112 removes all entries having values that match the elements of NoDiff from the difference dictionary Δ (step S78), and outputs the difference dictionary Δ to the dictionary creation unit 111 and the output unit 114 (step S80). ).

  In the specific example, Diff1 is not an empty set (step S68 / NO), and Diff0 is an empty set (step S70 / YES). Therefore, the differential dictionary creation unit 112 executes the process of step S76. The difference dictionary Δ after the process of step S76 is as shown in FIG. In the specific example, since NoDiff is an empty set, there is no entry to be removed from the difference dictionary Δ in step S78. Therefore, the difference dictionary creation unit 112 executes the process of step S80 and outputs the difference dictionary Δ shown in FIG. 10 (1) to the dictionary creation unit 111 and the output unit 114. Other steps in the difference dictionary creation process of FIG. 9 will be described later.

  When the difference dictionary creation process ends, the output unit 114 outputs the difference dictionary Δ as part of the compressed data (step S24). Next, the dictionary creation unit 111 updates the dictionary Dic1 based on the dictionary Dic0 and the difference dictionary Δ (step S26). Since the dictionary Dic0 is an empty set this time, the entry of the difference dictionary Δ is added to the dictionary Dic0 and becomes the dictionary Dic1 used for encoding (FIG. 10 (2)).

  Using the dictionary created by the dictionary creation unit 111 shown in FIG. 10 (2), the encoding unit 113 encodes records from the first line to the third line as shown in FIG. 10 (3). .

  The block B1 encoded as described above serves as a reference block for the next block to be subjected to compression processing. Accordingly, the dictionary creation unit 111 initializes the dictionary Dic0 with the dictionary Dic1 of the block B1 (step S32). The initialized dictionary Dic0 is as shown in FIG.

  Next, the data acquisition unit 110 acquires the record in the first row of the block B2 shown in FIG. 11 (2) (step S16). In starting the processing of a new block, the dictionary Dic1 is initialized to be an empty table in step S12, and no entry having the value of San Francisco exists in the dictionary Dic1 (step S50 / NO). Accordingly, the dictionary update process of FIG. 6 newly registers an entry assigned with the code “1” to San Francisco shown in FIG. 11 (3) in the dictionary Dic1 (step S52).

  Similarly, the data acquisition unit 110 acquires the record in the second row of the block B2 shown in FIG. 11 (4) (step S16). In the dictionary Dic1, Washington D.C. C. Since there is no entry having the value of (Step S50 / NO), as shown in FIG. C. Is newly registered in the dictionary Dic1 (step S52).

  Next, the data acquisition unit 110 acquires the record in the third row of the block B2 shown in FIG. 11 (6) (step S16). Since there is no entry having the value New York in the dictionary Dic1 (step S50 / NO), as shown in FIG. 11 (7), an entry assigned the code “3” to the New York is new to the dictionary Dic1. Registration is performed (step S52).

  Here, since the value of M is 3 and a predetermined number of records has been read (step S20 / NO), the difference dictionary creation unit 112 performs a difference dictionary creation process (step S22).

  In step S22, the difference dictionary creation unit 112 executes the processing of steps S60 to S66 of FIG. 9 described above. As a result, the initialized difference dictionary Δ is as shown in FIG. 12 (1), where Diff0 is an empty set, Diff1 = {New York}, NoDiff = {San Francisco, Washington D.D. C. }.

  Since Diff1 is not an empty set (step S68 / NO) and Diff0 is an empty set (step S70 / YES), the element of Diff1 is newly registered in the difference dictionary Δ (step S76). As a result, in the specific example, the difference dictionary Δ after the process of step S76 is as shown in FIG.

  Next, the differential dictionary creation unit 112 executes the process of step S78. In the specific example, an entry having a value of NoDiff elements (San Francisco and Washington DC) exists in the current difference dictionary Δ. Therefore, the difference dictionary creating unit 112 deletes the entry having (San Francisco and Washington DC) as values from the difference dictionary Δ (step S78), and the difference dictionary Δ is stored in the output unit 114 and the dictionary creating unit 111. Output (step S80). The output difference dictionary Δ is as shown in FIG.

  Next, the dictionary creation unit 111 updates the dictionary Dic1 based on the dictionary Dic0 shown in FIG. 13 (1) and the difference dictionary Δ shown in FIG. 13 (2) (step S26). In this case, since the code “3” included in the difference dictionary Δ is a code that is not registered in the dictionary Dic0, a dictionary Dic1 is obtained by adding the entry of the difference dictionary Δ to the dictionary Dic0. As a result, the dictionary Dic1 is as shown in FIG.

  The encoding unit 113 encodes the data of the block B2 using the dictionary of FIG. 13 (3) (step S28). The encoded data is as shown in FIG.

  The block B2 serves as a reference block for the next block to be subjected to compression processing. Accordingly, the dictionary creation unit 111 initializes the dictionary Dic0 with the dictionary Dic1 of the block B2 (step S32). The initialized dictionary Dic0 is as shown in FIG.

  Next, the data acquisition unit 110 acquires the record in the first row of the block B3 shown in FIG. 14 (2) (step S16). When starting the processing of a new block, since the dictionary Dic1 is initialized to be an empty table, there is no entry having a Chicago value in the dictionary Dic1 (step S50 / NO). Therefore, by the dictionary update process, an entry assigned with code “1” to Chicago shown in FIG. 14 (3) is newly registered in the dictionary Dic1 (step S52).

  The data acquisition unit 110 acquires the record in the second row of the block B3 shown in FIG. 13 (4) (step S16). Since there is no entry having the value of San Francisco in the dictionary Dic1 (step S50 / NO), as shown in FIG. 14 (5), an entry assigned the code “2” to the San Francisco is newly registered in the dictionary Dic1. (Step S52).

  Next, the data acquisition unit 110 acquires the record in the third row of the block B3 shown in FIG. 14 (6) (step S16). Since the dictionary Dic1 already has an entry having the value of San Francisco (step S50 / YES), the dictionary Dic1 is not changed and remains as shown in FIG. 14 (5).

  Here, since the value of M is 3 and a predetermined number of records has been read (step S20 / NO), the difference dictionary creation unit 112 performs a difference dictionary creation process (step S22).

  In step S22, the difference dictionary creation unit 112 executes the processing of steps S60 to S66 of FIG. 9 described above. As a result, in the specific example, the initialized difference dictionary Δ is as shown in FIG. Also, Diff0 = {Washington D. C. , New York}, Diff1 = {Chicago}, NoDiff = {San Francisco}. In the specific example, Diff1 is not an empty set (step S70 / NO), and Diff0 is not an empty set (step S70 / NO).

  In the flowchart of FIG. 9, when Diff0 is not an empty set (step S70 / NO), the difference dictionary creation unit 112 acquires the element d0 of the set Diff0 and the element d1 of the set Diff1, and sets d0 in the difference dictionary Δ to d1. Replace. Then, the difference dictionary creation unit 112 deletes the element d0 from Diff0 and the element d1 from Diff1 (step S72). Thereafter, the differential dictionary creation unit 112 returns to step S68 and continues the processing.

  In a specific example, in the difference dictionary Δ, Washington D. which is an element of Diff0. C. Is replaced with Chicago, which is an element of Diff1. Also, from Diff0 to Washington D.C. C. Are deleted, and Chicago is deleted from Diff1. As a result, the difference dictionary Δ after execution of step S72 is as shown in FIG. 15 (2), and Diff0 = {New York} and Diff1 is an empty set. Diff1 is an empty set means that the number of character strings that exist in the reference block dictionary Dic0 but do not appear in the processing target block is the number of character strings that do not exist in the dictionary Dic0 but appear in the processing target block. It is exceeding.

  Therefore, in the flowchart of FIG. 9, when Diff1 is an empty set (step S68 / YES), the difference dictionary creation unit 112 replaces all values that match the elements of Diff0 with NULL in the difference dictionary Δ (step S74). . Next, the difference dictionary creation unit 112 removes all entries having values that match the elements of Nodiff from the difference dictionary Δ (step S78), and outputs the difference dictionary Δ to the output unit 114 and the dictionary creation unit 111 (step S80). ).

  In the specific example, New York, which is an element of Diff0, is replaced with NULL in the difference dictionary Δ (step S74). As a result of this processing, the difference dictionary Δ after execution of step S74 is as shown in FIG. In step S78, the difference dictionary creating unit 112 removes an entry having a NoDiff element (San Francisco) value from the current difference dictionary Δ. As a result, the difference dictionary Δ output in step S80 is as shown in FIG.

  Next, the dictionary creation unit 111 updates the dictionary Dic1 used for encoding the block B3 based on the dictionary Dic0 shown in FIG. 16 (1) and the difference dictionary Δ shown in FIG. 16 (2) (step S26). ). In the dictionary Dic0 and the difference dictionary Δ, the codes “2” and “3” overlap. Therefore, in the dictionary Dic0, the character strings to which the codes “2” and “3” are assigned are the character strings of the difference dictionary Δ. Overwrite and update the dictionary Dic1. As a result, the dictionary Dic1 is as shown in FIG.

  The encoding unit 113 encodes the data of the block B3 using the dictionary Dic1 shown in FIG. 16 (3) (step S28). The encoded data is as shown in FIG.

  In this way, the compressed data shown in FIG. 7 is output as compressed data having the difference dictionary Δ and the encoded data for each block, as shown in FIG.

  As is apparent from the above description, the data compression apparatus according to the present embodiment includes a character string that is not registered in the reference dictionary among character strings that appear in the processing target block, and a processing target block in the reference dictionary. A difference dictionary, which is dictionary data associated with a code associated with a character string that does not appear in is generated. As a result, when compressing text data in block stream processing, compressed data including a difference dictionary and encoded data is created for each block, and a code assigned to a character string in the difference dictionary is reused to reduce the compression rate. Can be improved.

  Here, the compression ratios of the data to be compressed according to the first and second comparative examples and the present embodiment will be compared using FIGS. FIG. 18 (1) shows the compressed data used in this description. In this description, four records are processed as one block.

  First, creation of compressed data according to Comparative Example 1 will be described. FIG. 18B is a data example when the data to be compressed is compressed according to the first comparative example. In Comparative Example 1, for each block, a dictionary in which character strings appearing in the block are associated with codes is registered to create compressed data. Also, each time the block changes, the code is renumbered from 1.

  Specifically, description will be made with reference to the data in FIG. In the block B1 of the data to be compressed shown in FIG. C. , Los Angeles, and New Orleans. Therefore, in Comparative Example 1, a dictionary in which the above four character strings and codes are associated with each other is registered as a dictionary of the block B1, and as shown in FIG. 18 (2), the block B1 including the dictionary and the encoded data is registered. Create compressed data.

  Next, in Comparative Example 1, in the block B2, the Washington D.D. C. , New York, Chicago, and Los Angels, and a dictionary in which codes are associated with each other, are registered as a dictionary of the block B2. In block B3, Washington D. that appears in the block. C. , New York, Chicago, and Los Angels, and a dictionary in which codes are associated with each other, are registered as a dictionary of the block B3. Then, the compressed data is encoded using a dictionary registered for each block. As a result, the compressed data of Comparative Example 1 is as shown in FIG.

  Next, creation of compressed data according to Comparative Example 2 will be described. FIG. 19A is a data example when the data to be compressed is compressed according to the second comparative example. In Comparative Example 2, a difference dictionary similar to that of the present embodiment is created for each block, but a newly assigned code is assigned to a character string registered in the difference dictionary. That is, in Comparative Example 2, the code associated with the character string that appears in the reference block but does not appear in the processing target block is not reused.

  Comparative Example 2 will be specifically described. In Comparative Example 2, since the block B1 is the first processing target block, there is no reference block dictionary. Therefore, a difference dictionary having the same entry as that of the comparative example 1 is registered in the comparative example 2 for the block B1. Next, in block B2, comparative example 2 registers a difference dictionary for character strings that are not registered in the dictionary of block B1 but appear in block B2. That is, the character strings of New York and Chicago are registered in the difference dictionary. Here, in Comparative Example 2, a newly assigned code is assigned to the character string registered in the difference dictionary. Since the codes up to “4” are used in the block B1, the comparative example 2 assigns new codes “5” and “6” to New York and Chicago, respectively. In block B3, a difference dictionary is registered for a character string that does not exist in the dictionary used for encoding block B2 but appears in block B3. That is, the character strings of Las Vegas and Mexico City are registered in the difference dictionary. When the difference dictionary for the block B2 is created, since the codes are already numbered up to “6”, new codes “7” and “8” are assigned to Las Vegas and Mexico City, respectively. As described above, the second comparative example creates compressed data including a difference dictionary and encoded data for each block, as shown in FIG.

  FIG. 19B is a data example when the compressed data of FIG. 18A is compressed according to the present embodiment. In this embodiment, since the block B1 is the first processing target block, there is no reference block dictionary. Therefore, for the block B1, a difference dictionary having the same entries as those in the comparative examples 1 and 2 is registered in the embodiment. Next, in block B2, in the embodiment, a difference dictionary is registered for a character string that is not registered in the dictionary of block B1 but appears in block B2. That is, the character strings of New York and Chicago are registered in the difference dictionary. In the embodiment, the code assigned to the character string registered in the dictionary of the block B1 but not appearing in the block B2 is reused in the dictionary of the block B1. In the example of FIG. 19 (2), codes “1” and “4” assigned in the dictionary of block B1 are reused for San Francisco and New Orleans that do not appear in block B2, and New York, and Assign to Chicago. Also, in block B3, codes “3” and “4” assigned in the dictionary encoding block B2 are reused for Los Angels and Chicago that do not appear in block B3, and Las Vegas and Mexico City are used. Assign to. As described above, in this embodiment, as shown in FIG. 19B, compressed data including the difference dictionary and the encoded data is created for each block.

  Next, the compression ratios of Comparative Examples 1 and 2 and this example are compared.

  In Comparative Example 1, as described above, for each block, a dictionary in which a character string appearing in the block is associated with a code is created. Therefore, as shown in FIG. 18 (2), duplicate values are included between the dictionaries registered for each block. For example, in the dictionary of block B1 and the dictionary of block B2, Washington D.C. C. And Los Angels overlap. In addition, in the dictionary of block B2 and the dictionary of block B3, Washington D.C. C. And New York overlap. For this reason, in the comparative example 1, the size of the whole dictionary becomes large, and the compression rate is deteriorated.

  In Comparative Example 2, the character string that appears in block B1 is not registered in the dictionary of block B2, and only the character string that first appears in block B2 is registered in the difference dictionary. Therefore, the dictionary size can be reduced as compared with Comparative Example 1.

  However, in Comparative Example 2, a newly assigned code is assigned to a character string that does not appear in the reference block but appears in the processing target block. Therefore, compared to the case of outputting a dictionary for each block as in Comparative Example 1, the number of codes may increase, and the code length of the codes may become long. Therefore, the difference in compression rate between the case where a code is newly assigned as in Comparative Example 2 and the case where the code is reused as in this embodiment will be described.

  It is assumed that the compressed data D is divided into N blocks (D = B1, B2, B3... BN). Then, in the case of the comparative example 2, that is, when the code is not reused, the total number of codes required is the maximum number of entries included in the compressed data D. On the other hand, when codes are reused as in this embodiment, the total number of necessary codes is the maximum number of entries included in the block Bi (i = 1 to N).

  Assuming that integers are used for encoding, in the case of Comparative Example 2, if the number of codes is up to 256, encoding can be performed with a code length of 1 byte, but if it is more than that, code length of 2 bytes is required for 65536 types. If it is longer than that, the code length is further increased. On the other hand, in the case of the present embodiment, the total number of necessary codes is the maximum number of entries included in the block Bi. Therefore, if the number of records included in one block is 256 or less, the code length exceeds 1 byte. There is no.

  FIG. 20 is a diagram schematically illustrating the file size of the compressed data when the data is compressed according to Comparative Example 1, Comparative Example 2, and the present embodiment. In FIG. 20, the hatched portion represents dictionary data, and the unhatched portion represents encoded data. As can be seen from FIG. 20, in Comparative Example 1 and Comparative Example 2, since the dictionary size is smaller in Comparative Example 2, the compressed data size is also smaller.

Further, in the comparative example 2 and the present example, since the possibility of the code length becoming longer can be reduced in the present example than by reusing the code, the compressed data is further compared with the comparative example 2. The size can be reduced. That is, in the present embodiment, by using the differential dictionary, the number of entries registered in the dictionary can be reduced, so that the file size of dictionary data can be reduced. Furthermore, by reusing the code, the code length can be shortened and the data size required for the code can be reduced, so that the compression rate can be further improved.
[Data decompression device]

  Next, a data decompression apparatus according to the present case will be described. FIG. 21 shows a configuration example of a data decompression system including a data decompression apparatus according to the present case.

  The data decompression device 200 is connected to the storage device 10 and the data processing device 30 via the network 40.

  The storage device 10 stores data to be decompressed (described as decompressed data). The data decompression device 200 stores decompressed data.

  The data processing device 30 transmits the decompressed data to the data decompression device 200. Further, the data decompression apparatus 200 receives the decompressed data.

  Since the hardware configuration of the data decompression device 200 is the same as the hardware configuration of the data compression device 100, description thereof will be omitted. However, in the case of the data decompression apparatus 200, the ROM 102 stores a program for decompressing data to be decompressed. Further, the functions of the data decompression apparatus 200 shown in FIG. 22 are realized by the calculation by the CPU 103 of the program stored in the ROM 102.

  Next, means for realizing the functions of the data decompression apparatus 200 will be described with reference to FIG. FIG. 22 is an example of a functional block diagram of the data decompression apparatus 200.

  As illustrated in FIG. 22, the data decompression device 200 includes a block acquisition unit 201, a data acquisition unit 202 and a difference dictionary acquisition unit 203 that realize a function as an input unit that receives input of a differential dictionary and compressed data, and a dictionary restoration Unit 204, decoding unit 205, and output unit 206.

  The block acquisition unit 201 functions as a division unit that divides data to be decompressed, which is acquired from the storage device 10 or received from the data processing device 30 and configured by a plurality of blocks, into a plurality of blocks based on a predetermined rule Have The block acquisition unit 201 acquires a block (processing target block) that is a target of the decompression process from the decompressed data. Each block constituting the decompressed data has a difference dictionary and encoded data.

  The data acquisition unit 202 acquires encoded data from the processing target block acquired by the block acquisition unit 201.

  The difference dictionary acquisition unit 203 acquires a difference dictionary from the processing target block.

  The dictionary restoration unit 204 (processing target dictionary generation unit) encodes the processing target block based on the difference dictionary acquired by the difference dictionary acquisition unit 203 and the dictionary used for decoding the reference block different from the processing target block. Restore the dictionary (processing dictionary) used when creating the data. In this embodiment, in two consecutive blocks, the front block is set as a reference block, and the back block is set as a processing target block.

  The decoding unit 205 decodes the encoded data acquired by the data acquisition unit 202 using the dictionary restored by the dictionary restoration unit 204.

  The output unit 206 stores the block data decoded by the decoding unit 205 in the storage device 10 or transmits the data to the data processing device 30.

  Next, decompression processing executed by the data decompression apparatus 200 will be described. FIG. 23 is a flowchart illustrating an example of decompression processing executed by the data decompression apparatus 200.

  First, the dictionary restoration unit 204 initializes the dictionary Dic of the reference block with an empty table (step S100). This is because there is no reference block when the first block of the decompressed data is a processing target block.

  The block acquisition unit 201 determines whether there is a block to be processed in the decompressed data (step S102). If there is a block (step S102 / YES), the block acquisition unit 201 acquires the block (step S104). If the block does not exist (step S102 / NO), the block acquisition unit 201 ends this process.

  When the block acquisition unit 201 acquires a block (step S104), the difference dictionary acquisition unit 203 acquires a difference dictionary Δ included in the acquired block (step S106).

  Next, the dictionary restoration unit 204 executes a restoration process of the dictionary Dic1 used for decoding the processing target block (step S108). Details of the dictionary restoration processing will be described later.

  The restoration unit 205 decodes the encoded data using the dictionary Dic1 restored in the process of step S108 (step S110).

  The output unit 206 outputs the decoded data (step S112). Next, the dictionary restoration unit 204 initializes the dictionary Dic using the dictionary Dic1 used for decoding the processing target block (step S114). This is because in the present embodiment, the reference block and the processing target block are two consecutive blocks, and thus the dictionary Dic1 used for decoding in step S110 is the dictionary Dic of the reference block for the next processing target block.

  Next, data expansion by the above-described expansion processing will be described using specific data, and details of the dictionary restoration processing will be described. Here, it is assumed that the compressed data shown in FIG. 17 is decompressed to obtain decoded data.

  First, the block acquisition unit 201 acquires a block B1 (step S104). The difference dictionary acquisition unit 203 acquires the difference dictionary Δ included in the block B1 illustrated in FIG. 24A (step S106). Next, the dictionary restoration unit 204 executes a dictionary restoration process (step S108).

  Here, the dictionary restoration process will be described with reference to a specific example. FIG. 25 is a flowchart showing an example of the dictionary restoration process. The dictionary restoration unit 204 first initializes the dictionary Dic1 and the set DiffID used for data decoding of the processing target block (step S120). The dictionary restoration unit 204 initializes the dictionary Dic1 with the dictionary Dic, and initializes DiffID with the value set of the code items of the difference dictionary Δ (indicated as Δ.ID in FIG. 25).

  In the specific example, since the block B1 is the first block, the dictionary Dic used for decoding the reference block is an empty table. Therefore, as a result of the initialization in step S120, the dictionary Dic1 becomes an empty table, and DiffID = {1, 2}.

  Next, the dictionary restoration unit 204 determines whether or not an element exists in DiffID (step S122). When there is an element in DiffID (step S122 / YES), the dictionary restoration unit 204 acquires the minimum element k in DiffID and deletes the element k from DiffID (step S124). Next, the dictionary restoration unit 204 determines whether or not k acquired in step S124 is equal to or smaller than the number of entries in the dictionary Dic1 (step S126). In FIG. 25, the number of entries in the dictionary Dic1 is represented by | Dic1 |.

  When the value of k is larger than the number of entries in the dictionary Dic1 (step S126 / NO), the dictionary restoring unit 204 adds an entry satisfying Dic1 [k] = Δ [k] to the end of the dictionary Dic1. Thereafter, the dictionary restoration unit 204 returns to step S122 and continues the processing. Here, Dic1 [k] represents a character string associated with the code “k” in the dictionary Dic1, and Δ [k] represents a character string associated with the code “k” in the difference dictionary Δ. Represent.

  In the specific example, since there is an element in DiffID, the dictionary restoration unit 204 acquires the minimum element k = 1 in DiffID, and deletes k = 1 from DiffID (step S124). As a result, DiffID = {2}.

  Since k (= 1) is larger than the number of entries (= 0) in the dictionary Dic1, the dictionary restoring unit 204 causes the character string associated with the code “1” in the dictionary Dic1 to have the code “1” in the difference dictionary Δ. "Is added to the end of the dictionary Dic1 (step S130). As a result, the dictionary Dic1 after the process of step S130 is as shown in FIG.

  Since there is still an element in DiffID (step S122 / YES), dictionary restoring unit 204 acquires k = 2 from DiffID and removes k = 2 from DiffID (step S124). As a result, DiffID becomes an empty set.

  Since k = 2 is larger than the number of entries (= 1) in the dictionary Dic1, the dictionary restoration unit 204 corresponds to the character string associated with the code “2” in the dictionary Dic1 and the code “2” in the difference dictionary Δ. An entry that becomes the attached character string is added to the end of the dictionary Dic1. The dictionary Dic1 after the process of step S130 is as shown in FIG. In the process of step S124, DiffID is an empty set.

  In the flowchart of FIG. 25, when there is no element in DiffID (step S122 / NO), the dictionary restoring unit 204 outputs the dictionary Dic1 (step S132), and the process is terminated. Other steps of the dictionary restoration process will be described later.

  In the specific example, since the DiffID is an empty set (Step S122 / NO), the dictionary restoring unit 204 outputs the dictionary Dic1 (Step S132). FIG. 24 (4) shows the restored dictionary Dic1.

  The decoding unit 205 decodes the encoded data included in the block B1 acquired by the data acquisition unit 202 using the dictionary Dic1 shown in FIG. 24 (4) (step S110). The decrypted data is as shown in FIG.

  Since the next block (block B2) exists (step S102 / YES), the block acquisition unit 201 acquires the block B2. The difference dictionary acquisition unit 203 acquires the difference dictionary Δ included in the block B2 shown in FIG. 26 (1) (step S106). Next, the dictionary restoration unit 204 executes a dictionary restoration process (step S108).

  First, the dictionary restoration unit 204 initializes the dictionary Dic1 with the dictionary Dic used for decoding the block B1 serving as the reference block (step S120). Also, DiffID is set to Δ. Initialization is performed with the ID (step S120). As a result, the initialized dictionary Dic1 becomes FIG. 26 (2), and DiffID = {3}.

  Next, since there is an element in DiffID (step S122 / YES), the dictionary restoration unit 204 acquires k = 3 from DiffID and removes k = 3 from DiffID (step S124). As a result, DiffID becomes an empty set. Since k = 3 is larger than the number of entries in the dictionary Dic1 (= 2) (step S126 / YES), the dictionary restoring unit 204 determines that the character string associated with the code “3” of the dictionary Dic1 is the difference dictionary Δ The entry that becomes the character string associated with the code “3” is added to the dictionary Dic1 (step S130). The dictionary Dic1 after the process of step S130 is as shown in FIG.

  Since the DiffID is an empty set (Step S122 / NO), the dictionary restoring unit 204 outputs the dictionary Dic1 shown in FIG. 26 (4) (Step S132).

  The decoding unit 205 decodes the encoded data included in the block B2 using the dictionary shown in FIG. 26 (4) (step S110). As a result, the decoded data is as shown in FIG.

  Since the next block exists (step S102 / YES), the block acquisition unit 201 reads the block B3 (step S104). The difference dictionary acquisition unit 203 acquires the difference dictionary Δ illustrated in FIG. 27A from the block B3 (step S106). The dictionary restoration unit 204 executes dictionary restoration processing (step S108).

  The dictionary restoration unit 204 initializes the dictionary Dic1 with the dictionary Dic used for decoding the block B2 serving as the reference block (step S120). Also, DiffID is set to Δ. Initialization is performed with the ID (step S120). As a result, the initialized dictionary Dic1 becomes FIG. 27 (2), and DiffID = {2, 3}.

  Since there is an element in DiffID (step S122 / YES), the dictionary restoring unit 204 acquires the minimum element k = 2 from DiffID and removes k = 2 from DiffID (step S124). As a result, DiffID = {3}. Here, in a specific example, k = 2 is equal to or less than the number of entries (= 3) in the dictionary Dic1.

  In the flowchart of FIG. 25, when the value of k is less than or equal to the number of entries in the dictionary Dic1 (step S126 / YES), the dictionary restoration unit 204 determines a character string associated with the code “k” in the dictionary Dic1. The character string associated with the code “k” in the difference dictionary Δ is overwritten (step S128). When the process of step S128 is completed, the dictionary restoration unit 204 returns to step S122 and continues the process.

  In the specific example, the dictionary restoration unit 204 executes the process of step S128 because k = 2 is equal to or less than the number of entries in the dictionary Dic1 (= 3). That is, the character string associated with the code “2” in the dictionary Dic1 is overwritten with the character string associated with the code “2” in the difference dictionary Δ. The dictionary Dic1 after the process of step S128 is as shown in FIG.

  Next, since there is still an element in DiffID (step S122 / YES), the dictionary restoration unit 204 acquires element k = 3 and removes k = 3 from DiffID (step S124). As a result, DiffID becomes an empty set.

  Since the acquired k (= 3) is equal to or less than the number of entries (= 3) in the dictionary Dic1 (step S126 / YES), the dictionary restoring unit 204 executes the process of step S128. That is, the character string associated with the code “3” in the dictionary Dic1 is overwritten with the character string associated with the code “3” in the difference dictionary Δ. As a result, the dictionary Dic1 after the processing in step S128 is as shown in FIG. 27 (4).

  Since DiffID has become an empty set (step S122 / NO), the dictionary restoration unit 204 outputs the dictionary Dic1 shown in FIG. 27 (5). The decoding unit 205 decodes the data included in the block B3 with the dictionary shown in FIG. 27 (5). As a result, the decoded data is as shown in FIG.

  Since there is no block to be processed next to block B3 (step S102 / NO), the data expansion device 200 ends the data expansion processing.

  As is clear from the above description, the data decompression apparatus 200 according to the present embodiment, when there is an overlapping code between the dictionary used for encoding the reference block and the difference dictionary of the processing target block, The character string of the reference block corresponding to the overlapping code is replaced with the character string of the difference dictionary. As a result, it is possible to decompress the compressed data created by the above-described data compression method and restore the original data.

  The present embodiment has been described in detail above. However, the present embodiment is not limited to the specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. It is.

  For example, in the present embodiment, in two consecutive blocks, the front block is used as a reference block and the rear block is used as a processing target block, and compression processing and decompression processing are sequentially performed. However, the reference block may be the first block of the compressed data, and the processing target block may be an arbitrary block Bi other than the first block. That is, the data compression apparatus 100 may always create a difference dictionary using the first block as a reference block in the compression processing of the processing target block. When two consecutive blocks are compressed using this embodiment, for example, in order to decompress the data of the block Bi, after restoring the dictionaries used for encoding the blocks B1 to Bi-1, respectively. It is necessary to restore the block Bi dictionary and decrypt the data. However, when the difference dictionary for the first block is created, the dictionary used for encoding the block Bi can be restored by using the difference dictionary for the first block and the difference dictionary for the block Bi. There is no need to sequentially restore the dictionary in block B1 to block Bi-1. Therefore, it is possible to shorten the time until the data of the designated block is decoded. Further, the compressed data in this embodiment is text data having a structure, but the same can be implemented when the compressed data is plain text format data having no structure. Furthermore, in this embodiment, a simple “value encoding method” is adopted as the static lexicographic encoding method, but the same can be implemented even if other static lexicographic encoding methods are used. Is possible.

  Note that the functions of the data compression device and the data decompression device can be realized by a computer. In that case, a program describing the processing contents of the functions that the data compression apparatus and the data expansion apparatus should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium.

  When the program is distributed, for example, it is sold in the form of a portable recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

  In this embodiment, the data compression device and the data decompression device are described as separate devices. However, as shown in FIG. 28, one information processing device is configured to function as a data compression device and a data decompression device. You may do it. Further, for example, a server computer connected to a communication network such as the Internet is at least one of the data compression device and the data decompression device of the present case, and the communication device such as a personal computer connected thereto has at least data compression and data decompression. One service may be provided from a server computer (ASP (Application Service Provider)).

  In this embodiment, the data compression device 100 or the data decompression device 200 transmits / receives data to / from the storage device 10, the sensor device 20, and the data processing device 30 via the network 40. However, the data compression device 100 or the data decompression device 200 may be configured to directly connect (locally connect) to the storage device 10, the sensor device 20, and the data processing device 30 to transmit and receive data. . In this embodiment, only the district items are encoded, but it goes without saying that other items can also be encoded.

DESCRIPTION OF SYMBOLS 100 ... Data compression apparatus 110 ... Data acquisition part 111 ... Dictionary creation part 112 ... Difference dictionary creation part 113 ... Encoding part 114 ... Output part 200 ... Data decompression apparatus 201 ... Block acquisition part 202 ... Data acquisition part 203 ... Difference dictionary acquisition Unit 204 ... Dictionary restoration unit 205 ... Decoding unit

Claims (8)

An input unit that accepts input of text data;
A dividing unit that divides the text data into a plurality of blocks based on a predetermined rule;
Based on a reference dictionary that is dictionary data in which character strings and codes are stored in association with each other, among character strings that appear in a processing target block, character strings that are not registered in the reference dictionary and the reference dictionary A difference dictionary generation unit that generates a difference dictionary that is dictionary data associated with a code associated with a character string that does not appear in the processing target block;
Based on the created difference dictionary and the reference dictionary, a processing target dictionary generation unit that generates a processing target dictionary that is dictionary data;
A compression unit that compresses the processing target block by referring to the generated processing target dictionary and replacing a character string that appears in the processing target block with a corresponding code;
An output unit that outputs the processing target block data compressed by the compression unit and the generated difference dictionary;
A data compression apparatus comprising:   The data compression apparatus according to claim 1, wherein the reference dictionary is a processing target dictionary generated by the processing target dictionary generation unit for the previous processing target block.   The data compression apparatus according to claim 1, wherein the reference dictionary is a processing target dictionary generated by the processing target dictionary generation unit for the first processing target block. A dividing unit that divides data to be decompressed into a plurality of blocks based on a predetermined rule;
An acquisition unit that acquires a difference dictionary, which is dictionary data in which a character string and a code are stored in association with each other, from a processing target block to be processed among the plurality of blocks;
A processing target dictionary generation unit that generates a processing target dictionary that is dictionary data based on a reference dictionary that is dictionary data and the difference dictionary acquired by the acquisition unit ;
Based on the generated processing target dictionary, a decoding unit that decodes the processing target block by replacing a code appearing in the processing target block with a corresponding character string ;
Equipped with a,
The difference dictionary is associated with character strings that are not registered in the reference dictionary among character strings that appear in the processing target block before compression and character strings that do not appear in the processing target block before compression in the reference dictionary. Dictionary data in association with a given code,
The data decompression device , wherein the processing target block is compressed by referring to the processing target dictionary and replacing a character string appearing in the processing target block before compression with a corresponding code .   5. The data decompression apparatus according to claim 4, wherein the reference dictionary is a processing target dictionary generated by the processing target dictionary generation unit for the previous processing target block.   5. The data expansion device according to claim 4, wherein the reference dictionary is a processing target dictionary generated by the processing target dictionary generation unit for the first processing target block. On the computer,
An input step for accepting input of text data;
A division step of dividing the text data into a plurality of blocks based on a predetermined rule;
Based on a reference dictionary that is dictionary data in which character strings and codes are stored in association with each other, among character strings that appear in a processing target block, character strings that are not registered in the reference dictionary and the reference dictionary A difference dictionary generation step of generating a difference dictionary that is dictionary data associated with a code associated with a character string that does not appear in the processing target block;
Based on the created difference dictionary and the reference dictionary, a processing target dictionary generation step for generating a processing target dictionary that is dictionary data;
A compression step of compressing the processing target block by referring to the generated processing target dictionary and replacing a character string appearing in the processing target block with a corresponding code;
An output step for outputting the processing target block data compressed by the compression step and the generated difference dictionary;
A data compression program characterized in that is executed. On the computer,
A dividing step of dividing the compressed data to be decompressed into a plurality of blocks based on a predetermined rule;
An acquisition step of acquiring a difference dictionary that is dictionary data in which a character string and a code are stored in association with each other from a processing target block to be processed among the plurality of blocks;
A processing target dictionary generating step for generating a processing target dictionary that is dictionary data based on a reference dictionary that is dictionary data and the difference dictionary acquired in the acquiring step ;
A decoding step of decoding the processing target block by replacing a code appearing in the processing target block with a corresponding character string based on the generated processing target dictionary;
Was executed,
The difference dictionary is associated with character strings that are not registered in the reference dictionary among character strings that appear in the processing target block before compression and character strings that do not appear in the processing target block before compression in the reference dictionary. Dictionary data in association with a given code,
The data decompression program , wherein the processing target block is compressed by referring to the processing target dictionary and replacing a character string appearing in the processing target block before compression with a corresponding code .

Images