JPH06236302A - File compressing method - Google Patents

Abstract

PURPOSE:To attain random access and to efficiently compress a file. CONSTITUTION:A reference record wherein a value predicted to be the highest appearance frequency in an object file is set is previously generated in a data part except a record key, and data parts L1 and L2 whose values malues match each other when a record to be compressed and the reference record are compared with each other are compressed into compressed codes to generate a compression result record whose record length is shorter than its original length while a compression identification code and the compressed code are added to data of differences from the reference record.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a file compression method capable of random access in a VSAM data set or the like by compressing data items having the same value in a plurality of records.

[0002]

2. Description of the Related Art The main conventional file compression methods are as follows. (1) A file compression method for sequential data sets that outputs the difference from the previous record as a compressed record by using the similarity of the records in the file. (2) A random-accessible file compression method that compresses the same characters and repeated patterns in a record. (3) A file compression method that combines the above-mentioned two methods of compressing the same character and repeated pattern included in the difference from the previous record.

[0003]

As described above, in the conventional file compression methods (1) and (3), only sequential access from the beginning of the file is possible, and in principle random access is impossible. In (2), there is a problem that the compression efficiency is generally not very good despite the large amount of CPU usage.

The present invention has been made to solve such a problem, and a reference record in which data predicted to have the same value is set in a plurality of records to be compressed is created, and a reference record is created. An object of the present invention is to provide a file compression method that outputs a difference from a record to be compressed as a compressed record, has high compression efficiency, and allows random access.

[0005]

A file compression method according to the present invention is a file compression method for compressing a data portion having the same value in a plurality of records. , A reference record in which data that is expected to have the same value in multiple records is created in advance, the compression target record is compared with the reference record, and the value matches the value of the reference record. The data part of is compressed.

Further, the file compression method according to the present invention is
It is characterized by compressing the reference record.

Further, the file compression method according to the present invention is characterized in that a code indicating compression is added to the compressed record.

[0008]

With the file compression method according to the present invention, a reference record in which data having a high frequency of appearance is set in a compression target file is created in advance, and the difference between the compression target record and the reference record is output as a compression result record. , Random access to VSAM data set etc. is possible.

[0009]

Embodiments VCP (VSAM COMPRESSION PACKAGE) for realizing the file compression method of the present invention (hereinafter referred to as the present invention method)
Will be described in detail with reference to the drawings. FIG. 1 is a record format diagram showing an outline of a compression method in VCP. VCP
Then, create a reference record in which values that are expected to match among multiple records are set in advance for items after the record key item that is the compression prohibited area, and the compression target record is compared with the reference record. That has the same value as the reference record (indicated by hatching in the figure)
Compress.

When the number of BYTEs in the matching portion is 3 BYTE or more and 252 BYTE or less at the time of compression, the identification code X'FD 'and the compression code consisting of 2 BYTE of the record length (L2) of the matching portion are replaced, and 252 BYTE or more. If it is 32760 BYTE or less, it is replaced with a compression code consisting of the identification code X'FE 'and 3 BYTE of the record length (L1) of the matching portion. When the number of consecutive matched portions is 2 BYTE or less, the number of BYTEs is increased by compression and the compression effect cannot be obtained. Therefore, the replacement with the compression code is not performed.

In VCP, X'F is used as an identification character of the compression code.
Although D'and X'FE 'are used, it is possible that X'FE' and X'FD ', which means other than the compression code, exist in the record to be compressed, so X'FD with the meaning other than the compression code is used. ', X'
When FE 'appears, two X'FD' or X'FE 'are output in succession to enable discrimination from the compressed code.

If the record length of the compression result exceeds the original record length, the compression process is bypassed and the original record is output as it is. Therefore, it is possible to mix uncompressed records and compressed records in the same data set. Therefore, the compressed record has X'FE FF FF 'immediately after the compression prohibited area.
And a compressed identification area of 5 BYTE consisting of the record length (RL).

FIG. 2 is a format diagram of a VSAM data set created by VCP. In this embodiment, in order to compress / decompress the record at the time of access, a reference record having ALL LOW-VALUE (X'0000 ... ') as a key is held in the VSAM data set, and the data record stores this reference record. Recorded as a VCP-compressed variable length record. VC
In P, the reference record is created in the target VSAM file when it is loaded by the provided utility, and the access routine provided by VCP uses this reference record to decompress / compress the data. You don't have to give a reference record from.

In addition, in the VCP access routine, the general VS
Since AM (non-VCP VSAM) handling is also possible, general
For the purpose of distinguishing the VSAM ALL LOW-VALUE key record from the VCP format reference record, the reference record itself is also VCP-compressed with all space (X'40 ') data. Assuming that the reference record cannot be compressed, the area immediately after the compression prohibited area must always be X'FE F when loading.
It is considered to be F FF '.

The method for holding the reference record is as described above.
In addition to registering as a data with a special key such as ALL LOW-VALUE in the compression target file, the reference record is
It is also possible to create a load module having a name determined for each compression target file and manage the library. Even if the reference record is retained as a load module, the reference record is loaded into the memory space when the compression target file is accessed for the first time, and is loaded into the memory space in the subsequent accesses, as in the case where it is registered in the compression target file. Since the reference record is used, it is not necessary to perform unnecessary access such as module loading each time compression is performed.

The concrete processing procedure of the VCP will be described below with reference to the flowcharts of FIGS. 3 to 11. (A) Creation of compressed VSAM (see FIGS. 3 to 5) First, a sample record serving as a reference record is compressed. Open the sample file in which the VSAM file to be compressed and the sample record are registered (S1, S2),
Read the sample file (S3). Obtain key length, key position, and maximum record length information from the VSAM file (S
4) Calculate the key end position from the key length and key position (S
5), X'0 from the beginning of the sample record to the key end position
Set 0 '(S6). Next, after the key end of the sample record, a compression process as described later is performed using all the records of space (X'40 ') as samples (S7).

As a result of the compression, the [length to the end of key + compression identification code length + length of sample record compressed data] is set as the WRITE record length (S8), and the WRITE record length is set.
Compare with VSAM maximum record length (S9). If the WRITE record length is less than or equal to the VSAM maximum record length, [SAMPLE RECORD HEAD-KEY END + COMPRESSION ID CODE + SAMPLE RECORD COMPRESSED DATA] is taken as the VSAM record (S10).

If the WRITE record length exceeds the VSAM maximum record length as a result of the comparison in step S9, a compression identification code is provided immediately after the end of the sample record key (S11), the sample record is the VSAM record (S12), and the sample record length. As the WRITE record length (S13). The compressed or uncompressed VSAM record obtained as described above, that is, the sample record is written to the VSAM file (S14), and the sample file is closed (S15).

Next, the compression target data is compressed based on the sample record. Open the input data file (S1
6) Read the input data file (S17). After the key end of the input record, the sample record is used as a sample for compression processing as described later (S18). Set [length to end of key + length of compressed identification code + length of input record compressed data] as WRITE record length (S19), WRITE
The record length is compared with the VSAM maximum record length (S20).

When the WRITE record length is equal to or less than the VSAM maximum record length, [SAMPLE RECORD HEAD-KEY END + COMPRESSION IDENTIFICATION + SAMPLE RECORD COMPRESSED DATA] is set as a VSAM record (S21). On the other hand, as a result of the comparison in step S21, W
If RITE record length exceeds VSAM maximum record length, input record is VSAM record (S22), input record length is W
RITE record length (S23). The compressed or non-compressed VSAM record obtained as described above is written (S24), the process returns to step S17, and the same processing is repeated for all data records. When the input data file reaches the end of file, the VSAM file and the input data file are closed (S25, 26).

(B) Compression processing (see FIGS. 6 and 7) Variables I, J and K are set to "1" (S41), and variable I is set.
The value of is saved in the register SV-I (S42). IBYTE of the input record to be compressed (sample record in case of sample record compression) and J of sample record (all space records in case of sample record compression)
Compare with BYTE eye (S43). If the comparison results match, the values of variables I and J are each incremented by "1" (S44), and the value of variable I and the input record length are compared (S4
5), if the value of variable I is less than or equal to the input record length and processing for all record lengths has not ended,
Return to S43.

When the value of the variable I exceeds the input record length, or when the IBYTEth of the input record does not match the JBYTEth of the sample record, (I-SVI) is set to the record length of the matching portion of the compression code. Variable L (binary 2 BYTE)
Set to (S46). Compare the value of variable L with "2" (S4
7) If it exceeds “2”, compare it with “252” (S4
8) If the value of variable L is "3" or more and "252" or less,
X'FD 'is KBYTE of output record, lower 1 BYTE of variable L
To the (K + 1) BYTE of the output record (S49,
50), the value of the variable K is incremented by "2" and the process proceeds to step S62 (S51).

When the value of the variable L exceeds "252" as a result of the comparison in step S48, X'FE 'is output as KB of the record.
YTE eye, upper 1 BYTE of variable L is output record (K + 1) BYT
The Eth and lower 1 BYTE are output to the (K + 2) BYTE of the output record (S52, 53, 54), the variable K is incremented by "3", and the process proceeds to step S62 (S55).

On the other hand, as a result of the comparison in step S47, the variable L
If the value of is less than “2”, that is, if it does not match the sample record or if the matching part is less than 2 BYTE and the compression effect is not obtained, the value of the variable L is compared with “0” (S
56) If the variable L is “0”, that is, if it does not match the sample record, the process proceeds to step S62, and it is determined whether the IBYTE entry of the input record is X'FD 'or X'FE'. After the judgment (S64), the IBYTE eye of the input record is output to the KBYTE eye of the output record (S67), and the values of variables I, J, and K are incremented by "1" (S68, 69), and step S62.
Return to. The check with X'FD 'and X'FE' is unnecessary here because it is premised that the sample record does not include X'FD 'and X'FE'. Therefore, when it matches the sample record, it cannot be X'FD 'or X'FE'.

As a result of the comparison in step S56, if the value of the variable L is not "0", it is compared with "1" (S57), and if the value of the variable L is "1", (I-1) of the input record. Output the BYTE eye to the KBYTE eye of the output record (S60) and set the value of variable K to "1".
Only the increment is performed (S61), and the process proceeds to step S62.

If the value of the variable L is "2" as a result of the comparison in steps S56 and 57, the (I-2) BYTE eye of the input record is output to the KBYTE eye of the output record (S58) and the value of the variable K is output. Is incremented by "1" (S59), and the (I-1) BYTE eye of the input record is output to the KBYTE eye of the output record (S6
0), the value of the variable K is incremented by "1" (S61),
Control goes to step S62.

Next, the value of the variable I and the input record length are compared (S62), and if the value of the variable I is less than or equal to the input record length, the IBYTE's eye of the input record and the JBYTE's eye of the sample record are compared ( S63). If the comparison result does not match, the IBYTE'th entry of the input record is compared with X'FE 'and X'FD' (S
64). If the IBYTE index of the input record is either X'FE 'or X'FD', the IBYTE index of the input record is output to the KBYTE index of the output record to distinguish it from the compression code (S
65) After incrementing the value of the variable K by "1" (S6
6), input record IBYTE (second X'FD 'or X'F
E ') is output to the KBYTE of the output record (S67), and the variable
Increments the I, J, and K values by "1" (S6
8,69).

If the result of the comparison in step S64 is that the IBYTE entry of the input record is neither X'FE 'nor X'FD', the IBYTE entry of the input record is output to the KBYTE entry of the output record (S67), and the variable The values of I, J and K are incremented by "1" respectively (S68, 69) and the process returns to step S62. Step
When the value of the variable I exceeds the input record length as a result of the comparison in S62, the value of the variable K is set as the output record length (S70), and the compression process for one data record is completed. Also,
As a result of the comparison in step S63, if the IBYTE's eye of the input record and the JBYTE's eye of the sample record match, the process returns to step S42.

(C) Compressed VSAM access routine (see FIGS. 8 to 10) The initial setting SW indicating the preparation completion of the sample record is “1”
(S81), if "1" (ready), move to step S93. If not "1", set "1" in the initial setting SW (S82), VSAM The file is opened (S83), key length, key position, and maximum record length information is obtained from the VSAM file (S84), and the end position of the key is calculated from the key length and key position (S85).

X'0 from the beginning of the record to the key end position
Set 0 '(S86) and randomly read VSAM file
Search for sample records with X'00 'as the key (S
87). If there is no record with the key of X'00 'in the VSAM file, "0" is set in the compression processing SW (S89).

If there is a record having X'00 'as a key in the VSAM file, it is judged whether or not the compression identification code (X'FE FF FF') is immediately after the key end position (S88). In the case of the compressed identification code, after the key end of the record, the record of all spaces (X'40 ') is used as a sample for restoration processing as described later (S90), and the restoration result is used as a sample record (S91), and compressed. Set "1" to the processing switch (S92)
. On the other hand, if the result of determination in step S88 is that the compression identification code is not immediately after the key end position, "0" is set in the compression processing SW.
Set (S89).

It is determined whether or not the compression processing SW is "1".
(S93), if it is "0" (other than compressed file), it is normal VS
AM access is performed (S109), and if it is "1" (compressed file), it is determined whether it is a reference processing or an update processing (S94, 95). For reference processing, access VSAM,
Search the VSAM file using the method requested for access (S9
6) If the reference record does not exist in the VSAM file, end the reference.

If the record to be referenced exists in the VSAM file, it is judged whether the code immediately after the key is the compression identification code (S97).
Restoration processing, which will be described later, is performed using the sample record as a sample after the key end of the AM record (S98), the restoration result record is set as the reference result record (S99), and the reference is terminated. If the result of determination in step S97 is that the key immediately after the key is not the compressed identification code, the decompression process is bypassed and the VSAM record is used as is as the reference result record (S100), and the reference is terminated.

On the other hand, in the case of the update processing, the above-described compression processing is performed by using the sample record as a sample after the key end of the input record newly registered in the VSAM file (S101). The [length up to the end of key + compression identification code length + compressed input record data length] is set as the WRITE record length (S102), and the WRITE record length is compared with the VSAM maximum record length (S103).

If the WRITE record length is less than or equal to the VSAM maximum record length, the [input record start-key end + compression identification code + input record compressed data] is set as a VSAM record and the VSAM file is accessed (S104). If the WRITE record length exceeds the VSAM maximum record length as a result of comparison in step S103, the input record that is not compressed is used as the VSAM record (S105), the input record length is used as the WRITE record length (S106), and the VSAM file is accessed. Yes (S
107).

(D) Restoration processing (see FIG. 11) Variables I, J, and K are set to "1" (S121), and the IBYTE entry of the input record (sample record in the case of restoration of the sample record) is X '. It is determined whether it is FD 'or X'FE' (S122). IBYTE of input record is X'FD 'or X'
If it is neither FE ', the IBYTE of the input record is output to the KBYTE of the output record (S123), and the values of variables I, J, and K are incremented by "1" (S124, 125).
, And proceeds to step S138.

If the result of determination in step S122 is that the IBYTE index of the input record is either X'FD 'or X'FE', the IBYTE index of the input record is compared with the (I + 1) BYTE index (S12
6) If they match (in the case of data other than the compression code), the value of the variable I is incremented by "1" (S127) and the IBYTE eye of the input record is output to the KBYTE eye of the output record (S
123), the values of the variables I, J, and K are each incremented by "1" (S124, 125), and the process proceeds to step S138.

As a result of the comparison in step S126, if the IBYTE eye of the input record and the (I + 1) BYTE eye of the input record do not match (compression code), it is determined whether the IBYTE eye of the input record is X'FE '. It is determined (S128). In the case of X'FE ', the (I + 1) BYTE of the input record is set to the upper 1 BYTE of the variable L (S129), and the (I + 2) BYTE of the input record is set to the lower 1 BYTE of the variable L (S130). , Increment the value of variable I by “3” (S
131).

As a result of the comparison in step S128, when the IBYTE eye of the input record is X'FD ', the variable L is temporarily set to "0" and cleared (S132), and the (I + 1) BYTE eye of the input record is cleared. Is set to the lower 1 BYTE of the variable L (S133), and the value of the variable I is incremented by "2" (S134).

After setting the record length of the compression code in the variable L, LBYTE from the JBYTE'th position of the sample record (records of all spaces when restoring the sample record)
The minute data is output to the LBYTE from the KBYTE position of the output record (S135), and the values of the variables J and K are incremented by "L" (S136, 137). The above restoration process is repeated until the value of the variable I exceeds the input record length (S138), and when the value of the variable I exceeds the input record length, the value of the variable K is set as the output record length (S139), one record. The restoration process of is ended.

In the present embodiment, the case where the difference data from the reference record is output as it is as the compression result has been described, but it is possible to further compress the same character or the repeated pattern in the difference data.

[0042]

As described above, according to the method of the present invention, since compressed records can be restored from any record in a file, random access is possible, and in batch processing, items related to extraction of target data are If it is uncompressed, only the target data can be extracted from a large amount of data and then only the target data can be restored.
It has the excellent effect of significantly reducing the U occupation time.

Further, since the method of the present invention does not require the ordering of the records in the file, for example, by using the first data as a sample and compressing the parts other than the sort key of the data to be sorted after the second data, sorting is performed. Since the amount of work data is reduced, there is an excellent effect that the sorting time is greatly reduced due to the reduction of I / O and paging.

[Brief description of drawings]

FIG. 1 is a record format diagram showing an outline of a method of the present invention.

FIG. 2 is a format diagram of a VSAM data set in the method of the present invention.

FIG. 3 is a flowchart for creating a compressed VSAM in the method of the present invention.

FIG. 4 is a flowchart for creating a compressed VSAM in the method of the present invention.

FIG. 5 is a flowchart for creating a compressed VSAM in the method of the present invention.

FIG. 6 is a flowchart of a compression process in the method of the present invention.

FIG. 7 is a flowchart of compression processing in the method of the present invention.

FIG. 8 is a flow chart of a compressed VSAM access routine in the method of the present invention.

FIG. 9 is a flow chart of a compressed VSAM access routine in the method of the present invention.

FIG. 10 is a flow chart of a compressed VSAM access routine in the method of the present invention.

FIG. 11 is a flowchart of a restoration process in the method of the present invention.

Claims (3)

[Claims] 1. In a file compression method for compressing a data portion having the same value in a plurality of records, it is predicted that the plurality of records have the same value in the data portion excluding the key item of the record to be compressed. A file compression characterized in that a reference record in which data is set is created in advance, the compression target record is compared with the reference record, and the data part of the compression target record whose value matches the value of the reference record is compressed. Method. 2. The file compression method according to claim 1, wherein the reference record is compressed. 3. The file compression method according to claim 1, wherein a code indicating compression is added to the compressed record.