JP2000252832A - Data compressing device and recording medium with data compressing program is recorded therein - Google Patents

Abstract

PROBLEM TO BE SOLVED: To compress physical quantities as survey and statistic results with high compressibility by compressing data except images measured or calculated by sample positions obtained from outside, while removing the redundancy of data with the spatial correlate of a data array corresponding to the sample positions. SOLUTION: An MPU 12 opens a data file to be compressed, as a data compressing program is started, decides the kind of data from the extension or header information, and refers to a corresponding table on the basis of the data kind to automatically determine a compression system and compression parameters. When the MPU 12 can discriminate the delimiters of data and data lines according to the data kind and contents, sample space reproduction is possible as is, so that a multidimensional array corresponding to the sample positions is prepared in a memory 17. Then when the MPU 12 discriminates that the data are in text format and in floating-point format, the difference between the predicted value of following data and the actual data is calculated, on the basis of previously compressed data on the condition such that a compression system which was automatically determined is a reversible compression system, and then entropy encoding is carried out to compress the data amount.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data compression device for efficiently compressing physical quantities such as survey results and statistical results. Further, the present invention relates to a machine-readable recording medium on which a data compression program for executing such processing is recorded.

[0002]

2. Description of the Related Art Conventionally, terrain data (altitude, geology, geographic information, etc.), statistical data and weather data (barometric pressure, temperature,
Discrete digital data such as humidity is stored in a recording medium in an uncompressed state. For example, the Geographical Survey Institute in Japan records elevation data measured at intervals of 50 m or 250 m on a recording medium in an uncompressed text format.

In addition, the Japan Meteorological Service Support Center records the average value observed by the Japan Meteorological Agency together with the position information of the sample position (observation point) on a recording medium in an uncompressed text format. Conventionally, when compressing such data, a compression method (so-called compression method such as ZIP, TAR, LHA, etc.) for eliminating code redundancy of a data file has been generally used.

[0004]

Incidentally, the above-mentioned data generally has a large number of observation points and a large amount of data. In particular, in recent years, there are many opportunities to process global-scale data, and more data volumes must be processed. Since the data amount is large as described above, the following problem occurs. (1) To record data, a large-capacity recording medium is required. (2) When transmitting data, the transmission time becomes longer. (3) When processing data, a large-capacity memory is required, so a large-scale computer system is required. (4) Since the data file is large, it is difficult to simultaneously store it on a memory or a recording medium, and it is not possible to efficiently manage various types of data.

Therefore, in the inventions according to claims 1 to 8,
In order to solve the above problems, it is an object to provide a data compression device capable of compressing a physical quantity such as a survey result or a statistical result at a high compression rate. Another object of the present invention is to provide a recording medium on which a program for realizing the data compression apparatus on a computer is recorded.

[0006]

According to a first aspect of the present invention, there is provided a data acquisition unit for acquiring data (excluding image data) measured or calculated for each of a plurality of sample positions from outside. And a data compression unit that eliminates data redundancy based on the spatial correlation in the “data array in which data is arranged according to the sample position” and compresses data. .

In general, in a conventional compression method, a serialized data string in a data file is divided into appropriate lengths, and then compression encoding (Huffman encoding, run-length encoding, etc.) is performed. For this reason, in the serialized data string, the redundancy between the data cannot be determined for the data separated to some extent, and the redundancy is not sufficiently eliminated.

On the other hand, in the present invention, the space of the data array is conceptually configured by arranging the measured or calculated data in association with the sample position. For example, if the sample position is two-dimensional, the space of the data array is a two-dimensional space. In general, if the sample position is n-dimensional, the space of the data array is an n-dimensional space (n ≧ 3).

In the data array space configured as described above, there is strong correlation between neighboring data, such as similar values, continuous values, and periodic fluctuations of values. The correlation (that is, redundancy) at this time occurs in any direction in the data array space, unlike the conventional case where the serialization occurs only in one direction. Such spatial correlations occurring in multiple directions naturally include redundancy between data separated on a serialized data sequence, which could not be eliminated conventionally.

[0010] The data compression section of the present invention eliminates such spatial correlation, so that a higher compression ratio can be achieved more efficiently than in the past. Note that a time axis may be added as one of the dimensions of the sample position described above. In such a case, it is possible to arrange the time-series data collectively in a single data array space. As a result, it is possible to eliminate the redundancy in the time axis direction as a spatial correlation in the data array space.

In the present invention, if the data to be compressed is in text format, it is preferable to convert the data from text format to binary format before eliminating spatial redundancy. In this case, the binary format has higher data correlation than the text format, so that a high compression rate can be achieved more efficiently. further,
According to the present invention, when data to be compressed is in a floating-point format, before eliminating spatial redundancy, data is converted from a floating-point format to a fixed-point format in units of data blocks to be compressed, Alternatively, it is preferable that the value of the exponent part is uniformly set in units of data blocks to be compressed. By such conversion, the data digits are aligned and the correlation appears strongly, so that a high compression rate can be achieved more efficiently.

According to a second aspect of the present invention, in the data compression device according to the first aspect, the data compression unit is configured to perform a lossless compression method or a lossy compression method based on a predetermined correspondence. One of the methods is determined from the type of data, and the data is compressed using the determined compression method.

According to the present invention, a plurality of types of data can be compressed. In this case, depending on the type of data, there are data that needs to be completely stored and data that can be deleted to some extent. According to the second aspect of the present invention, the specification of compression is predetermined in accordance with the type of data. The data compression unit automatically determines a compression method (lossless compression or lossy compression) to be used from the type of data to be compressed based on the correspondence.

Therefore, if the data is of the type expected in advance, it is possible to easily determine an appropriate compression method in accordance with the data and perform appropriate data compression. If the data is of an unexpected type, the product of the present invention may ask the user about the compression specification, and may additionally record the result of the inquiry in the correspondence. With such a configuration, it is possible to flexibly and naturally expand the types of data that can be handled.

According to a third aspect of the present invention, in the data compression apparatus according to the first or second aspect, the data compression section includes “a plurality of pieces of compressed data having a common sample space”. And is stored in a single file or as a plurality of associated files.

Conventionally, data files are often individually managed because of the large size of the data files. For this reason, for example, in the case where the entire data is collectively updated or transmitted, each data file must be individually processed, and data management is very complicated. However, according to the third aspect of the present invention, a plurality of compressed data having a common sample space are stored in one of the following formats. (1) Collectively store in one file. (2) Store as a plurality of associated files. By storing in such a format, it becomes easy to manage and operate the data common to these sample spaces collectively.

According to a fourth aspect of the present invention, in the data compression device according to any one of the first to third aspects, the data compression unit divides the data array into blocks, A compression storage unit that calculates a DC component and an AC component for each block, compresses and stores the DC component and the AC component, reads out compressed data from the compression storage unit, and reads each compressed block from the compressed data. Average data extracting means for extracting a DC component and providing the data to data processing.

In general, for example, spreadsheet software used for data processing can process only up to 256 columns of data. Also, in applications such as data analysis and graph display, it is often preferable that the data sampling interval be appropriately wide in order to grasp the overall tendency. Under such circumstances, for data with a large number of samples, a process of appropriately reducing the number of data samples (hereinafter referred to as “subtraction process”) occurs at a high frequency prior to data processing.

For this reason, conventionally, data reduction processing is often performed after decoding (decompressing) a compressed file. In this case, a huge amount of decrypted data must be handled in the memory, and depending on the data reference order, there is a problem that the processing speed is slow, such as frequent swap processing with the hard disk. I was

However, in the fourth aspect of the present invention, a DC component is directly extracted from the compressed data. Since the DC component has already been reduced to the number of compressed blocks, the above-described reduction processing can be omitted or reduced.
In this case, it is not necessary to expand a huge amount of data on the memory, and it is possible to avoid a problem such as a slow processing speed. In addition, since the DC component is an average value or an intermediate value of blocks, aliasing noise generated when the number of samples is reduced is not included. Therefore, such a DC component becomes data particularly suitable for grasping the overall tendency using data analysis and graph display.

(Claim 5) According to a fifth aspect of the present invention, in the data compression apparatus according to any one of the first to fourth aspects, the data compression section converts a data array into a frequency space by orthogonal transform. Up conversion is performed, and data is compressed by removing redundancy based on the bias of the converted data toward low frequency components.

Conventionally, in the field of handling terrain data, statistical data, and the like, individual values are often processed as they are, and almost nobody considered processing such as conversion to frequency space. However, the present inventor has made it possible to perform orthogonal transformation from the data array space to the frequency space by arranging these data in the data array space. In this case, the continuity of data in the data array space appears as a bias toward low frequency components in the frequency space. Therefore, by eliminating the redundancy based on this bias, it is possible to execute data compression rationally and efficiently.

(Claim 6) According to a sixth aspect of the present invention, in the data compression apparatus according to any one of the first to fifth aspects, the sample position is a geographical position, and the data is , Terrain data, national census data, or weather data.

According to a seventh aspect of the present invention, in the data compression apparatus according to the sixth aspect, the terrain data is at least one of elevation data, geological data, or geographic data indicating geographic information. It is characterized by being.

(Claim 8) The invention according to claim 8 is the data compression device according to claim 6, wherein the national census data is at least one of population data, traffic data and other social statistics data. It is characterized by.

(Claim 9) The recording medium according to claim 9 has a computer for causing the computer to function as the "data acquisition unit and data compression unit according to any one of claims 1 to 8". A data compression program is recorded. By running this data compression program,
On a computer, a “data compression device according to any one of claims 1 to 8” is realized.

[0027]

Embodiments of the present invention will be described below with reference to the drawings. This embodiment is defined by claims 1 to
This is an embodiment corresponding to the invention described in Item 9.

(Configuration of Embodiment) FIG. 1 is a diagram showing the overall configuration of a data compression apparatus using a computer 11. In FIG. 1, an MP
A U (microprocessor) 12 is provided. This MP
The input device 13 including a keyboard and a mouse, a hard disk 16, a memory 17, an image processing board 18, and an interface board 20 are connected to the U12. A monitor 19 is connected to an image output terminal of the image processing board 18. On the other hand, interface board 2
0 is connected to an external device 21 such as an external recording medium.

On the other hand, a CD-ROM drive device 22 is connected to the MPU 12. The CD-ROM drive 22 has a CD-ROM 23 in which a data compression program and an installation program thereof are recorded. With the installation program in the CD-ROM 23, the MPU 12 expands the data compression program in the CD-ROM 23 and stores it in the hard disk 16 in an executable state.

(Regarding Correspondence Between the Present Invention and the Embodiment) Regarding the correspondence between the invention described in claim 1 and the embodiment, the data acquisition unit is configured to execute the “
And a function of reading data from the CD-ROM drive device 22 or the like, and the data compression unit of the MPU 12 arranges data in a data array space, eliminates redundancy in the data array space, and performs data compression. Function to perform ".

Regarding the correspondence between the invention described in claim 2 and the embodiment, the data compression unit corresponds to the “function of determining the type of data and automatically determining the data compression method” of the MPU 12. The correspondence between the invention described in claim 3 and the embodiment corresponds to the “function of recording data common to a sample space as one file or a related file” of the MPU 12.

As for the correspondence between the invention described in claim 4 and the embodiment, the compression storage means is configured to divide the data array of the MPU 12 into a DC component and an AC component in block units,
The average data extracting means corresponds to the “function of extracting a DC component from a compressed file and providing the data processing” of the MPU 12. Regarding the correspondence between the invention described in claim 5 and the embodiment, the data compression unit determines whether the data array of the MPU 12
T-conversion and compression function ". Regarding the correspondence between the invention described in claim 9 and the embodiment, the machine-readable recording medium corresponds to the CD-ROM 23, the hard disk 16, or the memory 17.

(Explanation of Operation of Compression Processing) FIG. 2 is a diagram showing a compression processing routine in this embodiment. Hereinafter, the operation of the compression process will be described with reference to the step numbers shown in FIG. First, when a data compression program is started by a user, the MPU 12 opens a data file to be compressed (step S1). Next, MPU
12 determines the type of data from the extension of the data file or the header information (step S2).

The MPU 12 refers to the correspondence table based on the determined data type. In the correspondence table, data types and corresponding “compression specifications such as compression method and compression parameter” are registered in advance. If the data type has not been registered, the MPU 12
9 to display an input dialog and ask the user to input a new compression specification. This new compression specification is additionally recorded in the correspondence table in association with the type of data. On the other hand, if the data type has been registered, the compression specification is read from the correspondence table, and the compression method and compression parameter are automatically determined (steps S3 and S4).

Next, the MPU 12 determines from the type of data and the contents of the data whether or not data or a data line break can be identified (step S5). For example, data that can be delimited include comma-separated / new-line delimited text data and fixed-length data in which the number of vertical and horizontal samples is stored in a file header or the like.

With respect to such data, the reproduction of the sample space can be performed as it is.
Move the operation to 7. On the other hand, if the data or the data line cannot be separated (NO in step S5), M
The PU 12 displays an input dialog on the monitor 19, and requests information such as (1) the number of dimensions of the sample space and (2) the number of samples in each axial direction on the sample space (step S6).

After such processing, the MPU 12 prepares a multidimensional array corresponding to each sample position on the memory 17 (step S7). Next, the MPU 12 determines whether the data is in a text format based on the type of the data and the content of the data (Step S8). If the data is in binary format (NO in step S8), the MPU 12 shifts the operation to step S10.

On the other hand, if the data is in a text format (YES in step S8), the MPU 12 converts the text data into binary data (step S9). next,
The MPU 12 determines whether the data is in a floating-point format based on the type of the data and the content of the data (Step S1).
0). If the data is not in the floating-point format (NO in step S10), the MPU 12 proceeds to step S1
The operation is shifted to 2.

On the other hand, if the data is in the floating-point format (YES in step S10), the MPU 12 aligns the exponent part of the data for each compression block unit described later (step S11). After such processing, the MPU
12 stores the data in a multidimensional array according to the sample position (step S12).

Next, the MPU 12 executes the above step S3
Data compression is started in accordance with the compression method determined in (1) (step S13). For example, when the reversible compression method is selected as the compression method, the MPU 12 calculates a predicted value of data to be compressed later based on the data compressed earlier. Subsequently, the MPU 12 calculates a difference (prediction error) between the predicted value and actual data (Step S14).

The thus obtained prediction error is entropy-coded to compress the data amount (step S15).
On the other hand, when the irreversible compression method is selected as the compression method, the MPU 12 divides the multidimensional array into approximately 8 × 8 compression blocks, and performs DCT (discrete cosine) transform on a block basis (step S16). ). In the case of exponential data, DCT is performed on the mantissa.

The MPU 12 quantizes the DCT coefficients thus obtained by using the quantization table determined by the compression specification (Step S17). The MPU 12 entropy-encodes the DC component of the quantized DCT coefficient by taking the difference from the DC component of the immediately preceding block (step S18). In the case of index data, MPU12
Performs the same compression processing on the exponent part of the data as the DC component.

The MPU 12 performs a zigzag scan on the AC component of the quantized DCT coefficient in the order of the spatial frequency.
Entropy coding is performed (step S19). After completing the above compression processing, the MPU 12 stores the compressed data in a new compressed file, and closes all the files (Step S20). Through such a series of operations, compression of the data file is completed.

(Storage of Compressed Data Common to Sample Space) Next, several examples of the storage form of compressed data will be described with reference to FIG. FIG. 3A is a diagram schematically illustrating a plurality of types of compressed data (here, elevation data, geological data, and land use data) having a common sample space. FIG. 3B is a diagram showing an interleaved storage format. In the case of this storage mode, the MPU 12 stores a plurality of types of compressed data alternately in one file. In such a storage form, a plurality of types of data can be collectively read for each sample position.
Therefore, the storage mode is particularly suitable for analysis such as performing correlation processing on a plurality of types of data.

FIG. 3C shows a non-interleaved storage format. In the case of this storage mode, MPU1
2 separately stores a plurality of types of compressed data in independent areas within one file. In such a storage form, it becomes easy to individually read a plurality of types of data. Therefore, the storage mode is particularly suitable for the purpose of individually processing and analyzing a plurality of types of data.

FIG. 3D is a diagram showing a storage form using an association file. In the case of this storage mode, MPU1
2 stores a plurality of types of compressed data in individual files. At this time, the MPU 12 records the association information indicating that the sample space is common in the header part (or another information file) of each file. In such a storage form, flexible data management is possible, such as managing individual files individually or managing them by associating them.

(Regarding DC Component Extraction Operation) FIG.
FIG. 4 is a diagram showing a DC component extraction routine by the MPU 12. Next, an operation for extracting a DC component from a compressed file will be described with reference to FIG. First, when the user instructs a DC component extraction operation, the MPU 12
Opens the compressed file (step S3
1).

The MPU 12 selectively reads only the DC component from the compressed file (step S32).
Next, the MPU 12 decodes the DC component that has been compression-encoded (Step S33). Then, MPU12
Converts the decoded DC component into a comma-separated text format to create a text file (step S34). By a series of DC component extraction operations as described above, a comma-delimited text file suitable for processing with spreadsheet software or the like can be quickly created.

(Example of Compression Process Result) FIG.
(A) is a graph showing data before compression. This data expresses the altitude data at the two-dimensional position of Sounkyo in Hokkaido as a 4-byte integer.
The data capacity is about 5.6 kbytes. On the other hand, FIG.
(B) is a graph obtained by temporarily compressing the altitude data according to the procedure of the irreversible compression method, decompressing the altitude data, and displaying the data in a graph. The data capacity of the compressed file is as small as about 3.4 kbytes. FIG. 6 is a graph showing the compression error of the elevation data. In this case, the compression error is generally limited to 15 m or less, depending on the setting of the quantization table and the scale factor.

(Effect of the present embodiment) By the operation described above, in the present embodiment, it is possible to efficiently eliminate the spatial redundancy in the data array space and achieve a high compression ratio. . Further, in the present embodiment, since an appropriate compression method according to the data is automatically determined, appropriate data compression can be performed on various data.

Further, in this embodiment, a plurality of compressed data having a common sample space are stored in a storage form as shown in FIG. Therefore, it becomes easy to collectively manage and operate data common to the sample space. In the present embodiment, by directly extracting the DC component from the compressed data, input data suitable for spreadsheet software can be quickly created. Furthermore, in the present embodiment, since data compression is performed on the frequency space using orthogonal transformation, attention is paid to continuity, periodicity, and the like of data.
Data compression can be performed reasonably and efficiently. By the way, the following are particularly mentioned as compression targets suitable for the present embodiment.

◎ Terrain data (especially elevation data, geological data, geographic information) ◎ Census data (especially population data, traffic data and other social statistical data) ◎ Meteorological data These data are all data samples The number is particularly large, and the data array space is likely to have redundancy. Therefore, it is particularly suitable as a data compression target of the present embodiment.

(Supplementary Items of the Embodiment) Note that, not limited to the types of data described above, in general, physical quantity data using a two-dimensional space, a three-dimensional space, or an n-dimensional space as a sample space may be set as a compression target. More specifically, data having a sample space of the ocean, the human body, the solid sample surface, or the like may be the compression target.

The types of individual data include electric field, magnetic field, temperature, humidity, weight, elastic modulus, moisture%, abnormal cell amount, blood amount, fat amount (for each position of the human body), protein mass, and ocean current. Data such as the velocity of the sea current, the velocity vector of the ocean current, the amount of plankton, and the composition of the seawater may be used as the compression target.
Further, as described in the section of [Means for Solving the Problem], a time axis may be added to the dimension of the data array space. An example of such an example is a process of compressing monthly temperature data measured for each two-dimensional measurement point. In this case, the data array space is composed of two spatial axes and one time axis, and is a three-dimensional space. In this data array space, since a strong correlation is generated in the directions of these three axes, a high compression ratio can be obtained by applying the data compression of the present invention.

Further, in the above-described embodiment, it is possible to easily compress the various physical quantity data as described above by an appropriate compression method and appropriately manage them collectively. Therefore, the present embodiment is very effective in providing a research environment for analyzing the correlation between these physical quantities. In addition,
In the embodiment described above, the case where the DCT transform is performed has been described, but the present invention is not limited to this. Other orthogonal transform may be performed instead of the DCT transform.

Further, in the above-described embodiment, data in which the sample intervals are equal are described. However, the present invention is not limited to this. In general, if the data correlation on the data array space can be used, the effect of the present invention can be obtained.
The sample intervals need not necessarily be equal. For example,
The data arrangement space may be configured by arranging unequally-spaced town-unit data in accordance with the approximate positional relationship of the town. Even in such a case, if the data correlation exists in the data array space, the effect of the present invention can be sufficiently obtained.

[0057]

According to the first aspect of the present invention, since the spatial redundancy in the data array space is eliminated, a high compression ratio can be achieved.

(Claim 2) In the invention according to claim 2,
If the data is of the type expected in advance, it is possible to automatically determine an appropriate compression method according to the data without any hassle, and to appropriately perform appropriate data compression.

(Claim 3) In the invention according to claim 3,
A plurality of compressed data having a common sample space is stored in one of the following formats.

(1) Collectively store in one file (2) Store as associated multiple files By storing in such a format, data common to these sample spaces can be managed and operated collectively. Becomes easier.

(Claim 4) In the invention according to claim 4,
Since the DC component is directly extracted from the compressed data, it is not particularly necessary to perform the data reduction process again. Therefore, the data processing time can be shortened accordingly.

(Claim 5) In the invention according to claim 5,
Performing orthogonal transform from the data array space to the frequency space facilitates rational and efficient data compression.

(Claims 6, 7, and 8) In the invention described in claim 6, terrain data, national census data, or weather data is compressed. According to the seventh aspect of the present invention, altitude data, geological data, or geographic information is compressed.

In the invention according to claim 8, the population data,
Compress traffic data and other social statistics data. Each of these data generally has an enormous number of data samples, and redundancy tends to occur in the data array space. Therefore, it is particularly suitable as a data compression target of the present invention. Therefore, the present invention is a particularly important technique when processing such a large amount of data with a small computer.

(Claim 9) By using the recording medium according to claim 9, a "data compression device according to any one of claims 1 to 8" is realized on a computer. Becomes possible.

[Brief description of the drawings]

FIG. 1 is a diagram showing an overall configuration of a data compression device using a computer 11;

FIG. 2 is a flowchart illustrating a compression process according to the embodiment.

FIG. 3 is a diagram illustrating a storage form of data common to a sample space.

FIG. 4 is a diagram showing a DC component extraction routine.

FIG. 5 is a diagram illustrating an example of a compression processing result.

FIG. 6 is a diagram illustrating an example of a compression error.

[Explanation of symbols]

 11 Computer 12 MPU 13 Input Device 16 Hard Disk 17 Memory 18 Image Processing Board 19 Monitor 20 Interface Board 21 External Equipment 22 CD-ROM Drive Device 23 CD-ROM

Claims (9)

[Claims] A data acquisition unit for acquiring data measured or calculated for each of a plurality of sample positions (excluding image data) from outside, and a data array in which the data is arranged according to the sample positions. A data compression device, comprising: a data compression unit that eliminates redundancy of the data based on spatial correlation and compresses the data. 2. The data compression device according to claim 1, wherein the data compression unit determines one of a lossless compression method and a lossy compression method from the type of the data based on a predetermined correspondence relationship. A data compression device that compresses the data by using the determined compression method. 3. The data compression device according to claim 1, wherein the data compression unit collects a plurality of pieces of compressed data having a common sample space into one file or as a plurality of associated files. A data compression device characterized by storing. 4. The data compression device according to claim 1, wherein the data compression unit divides the data array into blocks, and converts a DC component and an AC component for each block. Calculated, compressed storage means for compressing and storing the DC component and the AC component, reading compressed data from the compression storage means, extracting a DC component for each block from the compressed data, And a mean data extracting means for processing. 5. The data compression device according to claim 1, wherein the data compression unit transforms the data array into a frequency space by orthogonal transformation, and reduces a data after conversion. A data compression apparatus, wherein the data compression is performed by eliminating the redundancy based on a bias toward a band frequency component. 6. The data compression apparatus according to claim 1, wherein the sample position is a geographical position, and the data is terrain data, national census data, or weather data. A data compression device characterized by at least one of the following. 7. The data compression device according to claim 6, wherein the terrain data is at least one of elevation data, geological data, and geographic data indicating geographic information. 8. The data compression device according to claim 6, wherein the national census data is at least one of population data, traffic volume data, and other social statistics data. 9. A machine-readable recording medium recording a data compression program for causing a computer to function as the “data acquisition unit and data compression unit according to any one of claims 1 to 8”.