JP2007088962A - Ordinal data compressing method, ordinal data decompressing method, ordinal data processing program, ordinal data compressing apparatus, ordinal data decompressing apparatus, and ordinal data processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To compress ordial data such as time sequential data with a high compression ratio. <P>SOLUTION: An ordial data compressing method for compressing the ordial data of which the order is specified. A computer carries out: a compression pre-processing step of segmenting a plurality of pieces of continuous ordial data from the ordial data of which the order is specified, dividing each of the plurality of ordial data into a plurality of data elements, and creating a data element string by rearranging the data elements so as to continue data elements of the same bit position in the ordial data for each data element; a data compression step of compressing the data element string created by the compression pre-processing step; and a data storage step of storing, in a storage means, the data element string compressed by the data compression step. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an order data compression method, an order data decompression method, an order data processing program, an order data compression device, an order data decompression device, and an order data processing system.

  With the development of Internet technology and sensor technology, a time-series sensor data information collection system that collects time-series data generated from a sensor having a communication function is being realized. For example, the moving body position management system senses a large number of traveling vehicles and manages a log of vehicle movement. The seismic monitoring system collects information from a large number of scattered seismometers and records how it shakes when an earthquake occurs.

  Since the sensed time series data flows in one after another, it is required to store the data on the disk without loss. In the time-series sensor data information collection system, sensors are added to everything, and the sensors are operated for a long period of time, so the amount of time-series data generated from the sensors becomes enormous. Therefore, it is desirable to compress the time series data before storing it on the disk.

Conventional lossless compression methods include Huffman coding (Non-Patent Document 1) and block sorting (Non-Patent Document 2). Huffman coding is a method of generating a Huffman tree according to the appearance frequency of data to be compressed at the time of compression and allocating codes to data according to the Huffman tree. Similarly, when decompressing, a Huffman tree is generated and the compressed data is analyzed. In the block sorting, data to be compressed is rotated to create a plurality of data strings and then sorted. The sorted data string becomes the data to be compressed.
Huffman, DA, “A method for the construction of minimum redundancy codes”, In Proc. IRE, 1952 M. Burrows and DJ Wheeler, “A block-sorting lossless data compression algorithm”, ISRC Research Report, 1994

  However, simply using the conventional lossless compression method as it is cannot sufficiently increase the compression rate of the time series data. For example, although the appearance frequency of data is counted in Huffman coding, since time series data is a discrete value having a wide range of values that can be taken, a statistical result in which various types of values are generated at low frequencies, respectively. As a result, the compression rate becomes low.

  In view of the above, the main object of the present invention is to solve the above-described problems and realize compression of data in which the order of time-series data and the like is defined at a high compression rate.

  In order to solve the above-mentioned problem, the present invention provides a sequence data compression method for compressing sequence data in which a sequence is defined, wherein a computer continuously includes a plurality of the sequence data from the sequence data in which the sequence is defined. Is divided into a plurality of data elements for each of the plurality of order data, and the data elements are rearranged so that the data elements having the same bit position in the order data are consecutive for each of the data elements. A pre-compression processing procedure for creating a column, a data compression procedure for compressing the data element sequence created by the pre-compression processing procedure, and data for storing the data element sequence compressed by the data compression procedure in storage means And a storing procedure.

  Thereby, the order data can be compressed at a high compression rate by creating a data element sequence in which adjacent values are often the same.

  The present invention is characterized in that the data compression procedure compresses the data element sequence by run-length encoding.

  As a result, by compressing data element sequences in which adjacent values are often the same, the run becomes large, and compression can be performed at a high compression rate.

  According to the present invention, the data storage procedure stores the data element sequence in a sequential file according to the order of the data element sequence.

  Thereby, high-speed sequential access is possible.

  The present invention is an order data decompression method for decompressing order data in which an order is defined, wherein the computer compresses the data element sequence in which the data elements having the same bit position in the order data are consecutive. A data reading procedure for reading a column from a storage means, a data decompression procedure for decompressing the data element sequence read by the data reading procedure, and the data element sequence constituting the sequence data decompressed by the data decompression procedure. And a post-decompression processing procedure that is integrated into the sequence data.

  As a result, the sequence data compressed at a high compression rate can be decompressed by reading a data element sequence in which adjacent values are often the same.

  The present invention is characterized in that the data decompression procedure decompresses the data element sequence by run-length encoding.

  As a result, the data element sequences in which adjacent values are often the same are compressed with a large run, so that data compressed at a high compression rate can be decompressed.

  The present invention is characterized in that the data reading sequence reads the data element sequence from a sequential file in which the data element sequence is stored according to the order of the data element sequence.

  Thereby, high-speed sequential access is possible.

  The present invention is an order data processing program for causing a computer to execute the order data compression method or the order data decompression method.

  Thereby, the order data can be compressed at a high compression rate by creating a data element sequence in which adjacent values are often the same.

  The present invention is an order data compression device for compressing order data in which the order is defined, and a plurality of the order data are extracted from the order data in which the order is defined, and each of the order data A pre-compression processing unit that divides the data elements into a plurality of data elements and rearranges the data elements so that the data elements having the same bit position in the order data are consecutive for each of the data elements; A data compression unit that compresses the data element sequence created by the pre-compression processing unit, and a data storage unit that stores the data element sequence compressed by the data compression unit in a storage unit. .

  Thereby, the order data can be compressed at a high compression rate by creating a data element sequence in which adjacent values are often the same.

  The present invention is an order data decompressing device for decompressing order data in which the order is defined, and stores the compressed data element sequence for the data element sequence in which data elements having the same bit position in the order data are continuous. A data reading unit read from the means, a data decompression unit for decompressing the data element sequence read by the data reading unit, and the data element sequence constituting the order data decompressed by the data decompression unit as the sequence data And a post-decompression processing unit to be integrated into the system.

  As a result, the sequence data compressed at a high compression rate can be decompressed by reading a data element sequence in which adjacent values are often the same.

  The present invention is an order data processing system including the order data compression device and the order data decompression device.

  Thereby, the order data can be compressed at a high compression rate by creating a data element sequence in which adjacent values are often the same.

  According to the present invention, order data can be compressed at a high compression rate. Further, since the amount of data can be reduced, the I / O (Input / Output) cost for accessing the disk can be reduced, and the disk capacity required for storage can be reduced.

  The best mode of the present invention will be described below.

  FIG. 1A is a configuration diagram showing the sequential data processing apparatus 1. The sequential data processing device 1 is configured as a computer including at least a memory as a storage unit used when performing arithmetic processing and an arithmetic processing device that performs the arithmetic processing. The memory is constituted by a RAM (Random Access Memory) or the like. Arithmetic processing is realized by an arithmetic processing unit configured by a CPU (Central Processing Unit) executing a program on a memory. Hereinafter, description will be made based on time-series data as an example of order data, but the description can be applied to arbitrary order data.

  The sequential data processing device 1 includes a pre-compression processing unit 10, a data compression unit 12, a data storage unit 14, a data decompression unit 16, and a post-decompression processing unit 18. Note that the sequential data processing apparatus 1 may have a stand-alone configuration as shown in FIG. 1A or a network system as shown in FIG. In FIG. 1B, the sequential data processing device 1 is distributed to the sequential data compression device 2 that is responsible for compression and the sequential data decompression device 3 that is responsible for decompression, and is connected to each other via a network. The compressed data is communicated.

  The pre-compression processing unit 10 divides and rearranges the data values of the time series data that flows in one after another. The data compression unit 12 compresses the time series data after the pre-compression processing unit 10 has rearranged. The data storage unit 14 is configured by a storage unit such as a memory or a hard disk, and stores time-series data compressed by the data compression unit 12. The data decompression unit 16 decompresses the time series data stored in the data storage unit 14. The post-decompression processing unit 18 rearranges and integrates the time-series data after decompression by the data decompression unit 16.

  An arrow connected from the pre-compression processing unit 10 to the post-decompression processing unit 18 means that a parameter related to data processing is notified in this direction. Details of this parameter will be described later. The parameter notification method may be notified directly or indirectly by adding the compressed data as an attribute thereof. Furthermore, the order data compression apparatus 2 and the order data decompression apparatus 3 use the same preset parameter values, thereby omitting the parameter notification process from the order data compression apparatus 2 to the order data decompression apparatus 3. can do.

  FIG. 2A is a configuration diagram illustrating the pre-compression processing unit 10. The data temporary storage unit 30 is configured by storage means such as a memory, and stores time-series data flowing in one after another. The data dividing unit 32 divides the data value of the time series data. The data rearrangement unit 34 rearranges the divided data values by using a memory as a work area.

  FIG. 2B is a configuration diagram showing the post-decompression processing unit 18. The data rearrangement unit 34 rearranges the divided data values based on the amount of time series data stored in the data temporary storage unit 30 of the pre-compression processing unit 10. The data integration unit 36 integrates the rearranged data values.

  FIG. 3 is a flowchart showing compression processing in the time-series data lossless compression method according to this embodiment. This compression processing is characterized in that a high compression ratio is realized by preprocessing for dividing time-series data along the time axis before executing the compression algorithm. As a result, the amount of data to be stored can be reduced and the disk access processing can be reduced, which speeds up the processing and reduces the cost of the disk medium.

  First, the data temporary storage unit 30 of the pre-compression processing unit 10 divides the time series data in a continuous order and cuts out the time series data every N hours (S11). Each time series data is expressed as a set of a plurality of bytes inside the computer. Hereinafter, as an example of a data value, a 4-byte data value will be described assuming a little-endian format in which data is stored in order from the least significant byte. Also good. Of the 4-byte length, the 4th byte stores a sign part indicating +/- of the value and a mantissa part indicating the number of digits of the value.

  Next, the data division unit 32 of the pre-compression processing unit 10 divides one time series data into M data strings for each time series data cut out by the data temporary storage unit 30 (S12). For example, as a result of dividing 4-byte time-series data into four, a data string in which four 1-byte data are arranged is generated. FIG. 4A is an explanatory diagram showing an internal representation of time series data in a computer. Each of the N values of the time-series data is divided into four, thereby forming a matrix having N row elements and M (4) column elements.

  Further, the data rearrangement unit 34 of the pre-compression processing unit 10 rearranges the data divided in byte units by the data division unit 32 (S13). FIG. 5 is a flowchart for specifically explaining the data rearrangement process (S13).

  The data rearrangement unit 34 uses the loop variable i (S131) taking values from 1 to N and the loop variable j (S132) taking values from 1 to M to the matrix shown in FIG. A transposed matrix operation for exchanging rows and columns is performed (S133), and a matrix shown in FIG. 4B is generated. And the data rearrangement part 34 produces | generates the one order sequence shown in FIG.4 (c) by connecting each row | line | column to the column direction about the matrix produced | generated by S133 (S134). Thereby, data of the same position (same byte) constituting the time series data are arranged adjacent to each other.

Then, the data compression unit 12 compresses the data (byte string) rearranged by the data rearrangement unit 34 (S14). For the encoding method, for example, run-length encoding is used. This is because run-length encoding can be executed in one pass and is suitable for high-speed processing. In addition, the data compression part 12 may utilize various encoding systems shown in the following table independently, or may use a some encoding system together, for example.

  Note that if the data cut out in S11 is compressed as it is in S14, the adjacent values are not similar, so a high compression rate cannot be expected. However, by performing pre-processing (S12, S13) on the data cut out in S11, the adjacent values often become the same, so a high compression rate can be expected. This effect is particularly prominent in time-series data measured by a sensor or the like because the data value does not change abruptly but changes gradually (gradually). Since the sign part and the mantissa part change infrequently, adjacent values are often the same. In addition, for each data constituting the time series data, data with a large number of digits is less changed than data with a small number of digits.

  Further, the data storage unit 14 stores the data compressed by the data compression unit 12 (S15). The storage is preferably performed in the form of a sequential file for accessing the data in order from the top for each compressed data order. This is to enable high-speed sequential access.

  The compression process has been described above. The parameters relating to the arrows connected from the pre-compression processing unit 10 to the post-decompression processing unit 18 in FIG. 1 are N time for extracting time-series data in S11 and M for the number of divisions of one time-series data in S12. is there. If each of these parameters is set to a large value, a higher compression rate can be expected than a small value. On the other hand, when it is desired to shorten the processing time from the input of time series data to the output of compressed time series data in real time processing or the like, the parameter value may be reduced. Therefore, when the processing time is limited, a higher compression ratio can be realized by increasing the parameter value within the limited processing time.

  FIG. 6 is a flowchart showing the decompression process in the time-series data lossless compression method according to this embodiment. This decompression process is obtained by reversing the processing order as compared with the compression process of FIG.

  The data decompression unit 16 reads the time-series data compressed from the data storage unit 14 (S21), and performs decompression by executing a decompression algorithm corresponding to the compression algorithm of S14 (S22). The data rearrangement unit 34 rearranges the data decompressed in S22 by the same method as S13 (S23). The data integration unit 36 combines the M data strings into one time series data (S24). The decompressed post-processing unit 18 outputs time series data collected every N hours (S25).

It is a block diagram which shows the order data processing apparatus regarding one Embodiment of this invention. It is a block diagram which shows the pre-compression process part and post-decompression process part regarding one Embodiment of this invention. It is a flowchart which shows the compression process regarding one Embodiment of this invention. It is explanatory drawing which shows the internal expression in the computer of the time series data regarding one Embodiment of this invention. It is a flowchart which shows the rearrangement process regarding one Embodiment of this invention. It is a flowchart which shows the decompression | decompression process regarding one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Order data processing apparatus 2 Order data compression apparatus 3 Order data decompression apparatus 10 Pre-compression processing part 12 Data compression part 14 Data storage part 16 Data decompression part 18 Decompression | decompression processing part

Claims (10)

An order data compression method for compressing order data in which an order is defined,
Computer
A plurality of consecutive order data is cut out from the order data in which the order is defined, divided into a plurality of data elements for each of the plurality of order data, and a bit position in the order data for each of the data elements. A pre-compression processing procedure for creating a data element sequence by rearranging the data elements so that the same data elements are continuous;
A data compression procedure for compressing the data element sequence created by the pre-compression processing procedure;
A data storage procedure for storing the data element sequence compressed by the data compression procedure in a storage means;
A sequential data compression method comprising:   2. The sequential data compression method according to claim 1, wherein the data compression procedure compresses the data element sequence by run-length encoding.   3. The sequential data compression method according to claim 1, wherein the data storage procedure stores the data element sequence in a sequential file according to the order of the data element sequence. An order data decompression method for decompressing order data in which an order is defined,
Computer
A data reading procedure for reading the compressed data element sequence from the storage means for the data element sequence in which the data elements having the same bit position in the sequence data are continuous;
A data decompression procedure for decompressing the data element sequence read by the data read procedure;
A post-decompression processing procedure for integrating the data element sequence constituting the sequence data, which is decompressed by the data decompression procedure, into the sequence data;
An order data decompression method comprising:   5. The order data decompression method according to claim 4, wherein the data decompression procedure decompresses the data element sequence by run-length encoding.   6. The ordered data decompression method according to claim 4, wherein the data reading procedure reads the data element sequence from a sequential file in which the data element sequence is stored according to the order of the data element sequence. .   The sequential data for causing a computer to execute the sequential data compression method according to any one of claims 1 to 3 or the sequential data decompression method according to any one of claims 4 to 6. Processing program. An order data compression device for compressing order data in which an order is defined,
A plurality of consecutive order data is cut out from the order data in which the order is defined, divided into a plurality of data elements for each of the plurality of order data, and a bit position in the order data for each of the data elements. A pre-compression processing unit that rearranges the data elements so that the same data elements are continuous and creates a data element sequence;
A data compression unit for compressing the data element sequence created by the pre-compression processing unit;
A data storage unit that stores the data element sequence compressed by the data compression unit in a storage unit;
A sequential data compression apparatus comprising: An order data decompressing device for decompressing order data in which an order is defined,
A data reading unit that reads the compressed data element sequence from the storage unit for the data element sequence in which the data elements having the same bit position in the sequence data are continuous;
A data decompression unit for decompressing the data element sequence read by the data reading unit;
A post-decompression processing unit that uncompresses the data element sequence that constitutes the order data and that is decompressed by the data decompression unit;
An order data decompressing device characterized by executing   An order data processing system comprising: the order data compression apparatus according to claim 8; and the order data decompression apparatus according to claim 9.

Images