JP2639776B2 - File compression method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a file compression method capable of compressing data items having the same value in a plurality of records and enabling random access in a VSAM data set or the like.

[0002]

2. Description of the Related Art The main ones of conventional file compression methods are as follows. (1) A file compression method for sequential datasets that outputs the difference from the previous record as a compressed record using the similarity of the records in the file. (2) A random access file compression method that compresses the same character and repeated pattern in a record. (3) A file compression method that combines the above two methods for compressing the same character and repeated pattern included in the difference from the previous record.

[0003]

As described above, in the conventional file compression methods (1) and (3), only sequential access from the beginning of a file is possible, and random access is impossible in principle. (2) has a problem that the compression efficiency is generally not so good in spite of the large amount of CPU usage.

The present invention has been made to solve such a problem. A reference record in which data predicted to have the same value in a plurality of compression target records is created, and the reference record and the reference record are created. An object of the present invention is to provide a file compression method capable of performing random access with high compression efficiency by outputting a difference from a record to be compressed as a compressed record.

[0005]

A file compression method according to the present invention is a file compression method for compressing a data portion having the same value in a plurality of records. , A reference record in which data that is expected to have the same value in a plurality of records is created in advance, and the record to be compressed is compared with the record to be compressed, and the value matches the value of the reference record. The data portion is compressed.

Further, a file compression method according to the present invention
It is characterized in that the reference record is compressed.

Further, a file compression method according to the present invention is characterized in that a code indicating compression is added to a compressed record.

[0008]

According to the file compression method of the present invention, a reference record in which data having a high frequency of appearance is set in a compression target file is created in advance, and the difference between the compression target record and the reference record is output as a compression result record. , Random access to VSAM data sets and the like.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS VCP (VSAMCOMPRESSION PACKAGE) for realizing the file compression method of the present invention (hereinafter referred to as the method of the present invention)
Will be described in detail with reference to the drawings. FIG. 1 is a record format diagram showing an outline of a compression method in VCP. VCP
In this section, a reference record is created in advance with a value that is expected to match between multiple records in the items following the record key item that is a compression prohibited area, and the record to be compressed is compared with the reference record. Where the value matches the reference record (shown by hatching in the figure)
Compress.

In the compression, if the number of BYTEs in the matching portion is 3 or more bytes and 252 BYTE or less, it is replaced with a compression code consisting of the identification code X'FD 'and the record length (L2) of the matching portion of 2 BYTE. If it is less than 32760 BYTE, it is replaced with a compression code consisting of the identification code X'FE 'and the 3BYTE of the record length (L1) of the matching part. If the number of consecutive matching parts is 2 BYTE or less, the number of BYTEs increases by compression, and the compression effect cannot be obtained. Therefore, replacement with a compressed code is not performed.

In VCP, X'F
Although D 'and X'FE' are used, there may be X'FE 'and X'FD' which means other than the compression code in the record to be compressed. ', X'
When FE 'appears, two X'FD's or X'FE's are continuously output to enable discrimination from a compressed code.

Further, when the record length of the compression result exceeds the original record length, the original record is output as it is, bypassing the compression processing. Therefore, it is possible to mix non-compressed records and compressed records in the same data set. X'FE FF FF 'immediately after the compression prohibited area
And a 5-byte compression identification area including a record length (RL).

FIG. 2 is a format diagram of a VSAM data set created by the VCP. In this embodiment, a reference record using ALL LOW-VALUE (X'0000...) As a key is stored in the VSAM data set for record compression / decompression at the time of access. It is recorded as a variable-length record compressed using VCP. VC
In P, the reference record is created in the target VSAM file when loaded by the provided utility, and in the access routine provided by VCP, the data is restored / compressed using this reference record. You do not need to provide a reference record from.

In the VCP access routine, a general VS
AM (non-VCP VSAM) handling is also possible,
In order to distinguish between the VSAM ALL LOW-VALUE key record and the VCP format reference record, the reference records themselves are all VCP-compressed between space (X'40 ') data. Note that, assuming that the reference record cannot be compressed, X'FE F
F FF 'is considered.

The method of holding the reference record is as described above.
In addition to registering data with a special key such as ALL LOW-VALUE in the file to be compressed,
A method of creating a load module having a name determined for each compression target file and managing the library is also possible. Even if the reference record is stored as a load module, the reference record is loaded into the memory space at the first access to the file to be compressed, and is loaded into the memory space in subsequent accesses, as in the case where the reference record is registered in the file to be compressed. Since a reference record is used, it is not necessary to perform useless access such as loading of a module every time compression is performed.

Hereinafter, the specific processing procedure of the VCP will be described with reference to the flowcharts of FIGS. (A) Creating a compressed VSAM (see FIGS. 3 to 5) First, a sample record serving as a reference record is compressed. Open the VSAM file to be compressed and the sample file in which the sample record is registered (S1, S2),
Read the sample file (S3). Obtain key length, key position, and maximum record length information from the VSAM file (S
4) Calculate the key end position from the key length and key position (S
5), X'0 from the beginning of the sample record to the key end position
Set '0' (S6). Next, after the key end of the sample record, compression processing as described below is performed using all space (X'40 ') records as samples (S7).

As a result of the compression, [length to key end + length of compression identification code + length of compressed sample record data] is set as the WRITE record length (S8), and
Compare with VSAM maximum record length (S9). If the WRITE record length is equal to or less than the VSAM maximum record length, [sample record head to key end + compression identification code + sample record compressed data] is set as a VSAM record (S10).

If the result of the comparison in step S9 indicates that the WRITE record length exceeds the VSAM maximum record length, a compression identification code is provided immediately after the end of the sample record key (S11), the sample record is stored in the VSAM record (S12), and the sample record length is set. Is the WRITE record length (S13). The compressed or uncompressed VSAM record obtained as described above, that is, a sample record is written in the VSAM file (S14), and the sample file is closed (S15).

Next, the data to be compressed is compressed based on the sample records. Open the input data file (S1
6) Then, the input data file is read (S17). A compression process as described later is performed using a sample record as a sample after the key end of the input record (S18). [Length to key end + compression identification code length + input record compression data length] is set as the WRITE record length (S19), and WRITE
The record length is compared with the VSAM maximum record length (S20).

If the WRITE record length is equal to or less than the VSAM maximum record length, [start of sample record to key end + compression identification code + compressed sample record data] is set as a VSAM record (S21). On the other hand, as a result of the comparison in step S21, W
If the RITE record length exceeds the VSAM maximum record length, set the input record to VSAM record (S22) and set the input record length to W
The RITE record length is set (S23). The compressed or uncompressed VSAM record obtained as described above is written (S24), and the process returns to step S17 to repeat the same process for all data records. When the input data file reaches the end of file, the VSAM file and the input data file are closed (S25, S26).

(B) Compression processing (see FIGS. 6 and 7) "1" is set to each of the variables I, J and K (S41), and the variable I
Is saved in the register SV-I (S42). IBYTE of the input record to be compressed (sample record in the case of sample record compression) and J of the sample record (all records in the case of sample record compression)
A comparison is made with the BYTE order (S43). If the comparison results match, the values of the variables I and J are each incremented by "1" (S44), and the value of the variable I is compared with the input record length (S4).
5) If the value of variable I is less than or equal to the input record length and the processing for all record lengths has not been completed,
Return to S43.

If the value of the variable I exceeds the input record length, or if the IBYTE of the input record does not match the JBYTE of the sample record, (I-SVI) is replaced by the record length of the part where the compression code matches. Variable L (binary 2 BYTE)
(S46). The value of the variable L is compared with “2” (S4
7) If “2” is exceeded, compare with “252” (S4
8) If the value of the variable L is equal to or more than “3” and equal to or less than “252”,
X'FD 'is the KBYTE of the output record, the lower 1 BYTE of the variable L
Are output to the (K + 1) BYTE of the output record, respectively (S49,
50) Then, the value of the variable K is incremented by "2", and the routine goes to step S62 (S51).

As a result of the comparison in step S48, if the value of the variable L exceeds "252", X'FE 'is set to KB of the output record.
YK, top 1 BYTE of variable L is (K + 1) BYT of output record
The Eth and lower 1 BYTE are output to the (K + 2) BYTE of the output record (S52, 53, 54), the variable K is incremented by "3", and the routine goes to step S62 (S55).

On the other hand, as a result of the comparison in step S47, the variable L
Is less than or equal to “2”, that is, when the value does not match the sample record or when the matching portion is 2 BYTE or less and the compression effect cannot be obtained, the value of the variable L is compared with “0” (S
56) If the variable L is "0", that is, if the variable L does not match the sample record, the flow shifts to step S62 to determine whether or not the IBYTE of the input record is X'FD 'or X'FE'. After the determination (S64), the IBYTE of the input record is output to the KBYTE of the output record (S67), and the values of the variables I, J, and K are respectively incremented by "1" (S68, 69), and step S62 is performed.
Return to Here, the reason why the check for X'FD 'and X'FE' is unnecessary is because it is assumed that the sample record does not include X'FD 'and X'FE'. Therefore, when they match the sample record, they cannot be X'FD 'and X'FE'.

As a result of the comparison in step S56, if the value of the variable L is not "0", it is compared with "1" (S57). If the value of the variable L is "1", (I-1) of the input record is compared. The BYTE is output to the KBYTE of the output record (S60), and the value of the variable K is set to "1".
Then, the process proceeds to step S62.

As a result of the comparison in steps S56 and S57, if the value of the variable L is "2", the (I-2) BYTE of the input record is output to the KBYTE of the output record (S58), and the value of the variable K is Is incremented by “1” (S59), and the (I-1) BYTE of the input record is output to the KBYTE of the output record (S6).
0), the value of the variable K is increased by “1” (S61),
Move to step S62.

Next, the value of the variable I is compared with the input record length (S62). If the value of the variable I is equal to or less than the input record length, the IBYTE of the input record is compared with the JBYTE of the sample record (S62). S63). If the comparison result does not match, the IBYTE of the input record is compared with X'FE 'and X'FD' (S
64). If the IBYTE of the input record is either X'FE 'or X'FD', the IBYTE of the input record is output to the KBYTE of the output record to distinguish it from the compression code (S
65) After incrementing the value of the variable K by “1” (S6
6), IBYTE of the input record (the second X'FD 'or X'F
E ') is output to the KBYTE of the output record (S67), and the variable
The values of I, J, and K are each increased by “1” (S6
8,69).

If the result of the comparison in step S64 is that the IBYTE of the input record is not X'FE 'or X'FD', the IBYTE of the input record is output to the KBYTE of the output record (S67), and the variable The values of I, J, and K are each increased by “1” (S68, 69), and the process returns to step S62. Steps
As a result of the comparison in S62, if the value of the variable I exceeds the input record length, the value of the variable K is set as the output record length (S70), and the compression processing for one data record is terminated. Also,
As a result of the comparison in step S63, when the IBYTE of the input record matches the JBYTE of the sample record, the process returns to step S42.

(C) Compressed VSAM Access Routine (see FIGS. 8 to 10) Initial setting SW indicating completion of sample record preparation is “1”
Is determined (S81). If "1" (preparation completed), the process proceeds to step S93. If not "1", "1" is set to the initial setting SW (S82), and the VSAM The file is opened (S83), the key length, the key position, and the maximum record length information are obtained from the VSAM file (S84), and the end position of the key is calculated from the key length and the key position (S85).

X'0 from the beginning of the record to the key end position
Set to 0 '(S86) and read VSAM file randomly
To search for a sample record with X'00 'as a key (S
87). If there is no record using X'00 'as a key in the VSAM file, "0" is set to the compression processing SW (S89).

If there is a record having X'00 'as a key in the VSAM file, it is determined whether or not a compression identification code (X'FE FF FF') is located immediately after the key end position (S88). In the case of the compression identification code, after the key end of the record, the restoring process is performed as described later using all space records (X'40 ') as a sample (S90), and the decompression result is set as a sample record (S91), and "1" is set to the processing SW (S92)
. On the other hand, if the result of determination in step S88 is that the compression identification code is not immediately after the key end position, the compression processing SW is set to “0”.
Is set (S89).

It is determined whether or not the compression processing switch is "1".
(S93) If "0" (other than compressed file), normal VS
AM access is performed (S109), and if it is "1" (compressed file), it is determined whether the process is a reference process or an update process (S94, 95). For reference-related processing, access VSAM,
Search for VSAM file using requested access method (S9
6) If the record to be referenced does not exist in the VSAM file, terminate the reference.

If a record to be referred to exists in the VSAM file, it is determined whether or not the code immediately after the key is a compression identification code (S97).
Restoration processing described later is performed using the sample record as a sample after the key end of the AM record (S98), the restoration result record is set as a reference result record (S99), and the reference is terminated. If the result of the determination in step S97 is that the compression identification code is not immediately after the key, the decompression process is bypassed and the VSAM record is used as a reference result record (S100), and the reference is terminated.

On the other hand, in the case of update processing, the above-described compression processing is performed using a sample record as a sample after the key end of an input record to be newly registered in the VSAM file (S101). [Length to key end + compression identification code length + input record compression data length] is set as the WRITE record length (S102), and the WRITE record length is compared with the VSAM maximum record length (S103).

If the WRITE record length is equal to or less than the VSAM maximum record length, [input record head-key end + compression identification code + input record compression data] is set as the VSAM record, and the VSAM file is accessed (S104). If the WRITE record length exceeds the VSAM maximum record length as a result of the comparison in step S103, the input record that is not compressed is used as the VSAM record (S105), the input record length is used as the WRITE record length (S106), and the VSAM file is accessed. (S
107).

(D) Restoration processing (see FIG. 11) "1" is set to each of the variables I, J, and K (S121), and the IBYTE of the input record (the sample record in the case of restoring the sample record) is X '. It is determined whether it is FD 'or X'FE' (S122). The IBYTE of the input record is X'FD 'or X'
If it is not FE ', the IBYTE of the input record is output to the KBYTE of the output record (S123), and the values of the variables I, J, and K are each incremented by "1" (S124, 125).
Then, control goes to a step S138.

If the result of the determination in step S122 is that the IBYTE of the input record is either X'FD 'or X'FE', the IBYTE of the input record is compared with the (I + 1) BYTE (S12).
6) If they match (in the case of data other than the compression code), the value of the variable I is incremented by “1” (S127), and the IBYTE of the input record is output to the KBYTE of the output record (S127).
123), the values of the variables I, J, and K are each incremented by "1" (S124, 125), and the flow shifts to step S138.

As a result of the comparison in step S126, if the IBYTE of the input record does not match the (I + 1) BYTE (in the case of the compressed code), whether or not the IBYTE of the input record is X'FE ' It is determined (S128). In the case of X'FE ', the (I + 1) BYTE of the input record is set to the upper 1 BYTE of the variable L (S129), and the (I + 2) BYTE of the input record is set to the lower 1 BYTE of the variable L (S130). , The value of the variable I is increased by “3” (S
131).

As a result of the comparison in step S128, if the IBYTE of the input record is X'FD ', the variable L is temporarily set to "0" and cleared (S132), and the (I + 1) BYTE of the input record is cleared. Is set to the lower 1 BYTE of the variable L (S133), and the value of the variable I is increased by "2" (S134).

After setting the record length of the compression code to the variable L, the LBYTE from the JBYTE of the sample record (in the case of restoring the sample record, the record of all spaces)
The minute data is output to LBYTE from the KBYTE of the output record (S135), and the values of the variables J and K are incremented by "L" respectively (S136 and 137). The above restoration processing is repeated until the value of the variable I exceeds the input record length (S138), and when the value of the variable I exceeds the input record length, the value of the variable K is set as the output record length (S139). Is completed.

Although the present embodiment has been described with reference to the case where the difference data from the reference record is output as it is as the compression result, the same character and the repetition pattern in the difference data can be further compressed.

[0042]

As described above, according to the method of the present invention, a compressed record can be restored from any record in a file, so that random access is possible. If it is uncompressed, after extracting only the desired data from the large amount of data, only the desired data can be restored
It has an excellent effect of greatly reducing the U occupation time. Also, when compressing, it is only necessary to compare with the reference record.
High-speed compression is possible. It is compressed and recorded
Expansion of other data even if any of the data
Can be done without any problems.

Further, since the method of the present invention does not require the order of the records in the file, for example, the data other than the sort key of the data to be sorted is compressed by using the first data as a sample to sort the data. Since the amount of work data is reduced, an excellent effect of greatly reducing sort time due to reduction of I / O and paging is achieved.

[Brief description of the drawings]

FIG. 1 is a record format diagram showing an outline of a method of the present invention.

FIG. 2 is a format diagram of a VSAM data set in the method of the present invention.

FIG. 3 is a flowchart of creating a compressed VSAM in the method of the present invention.

FIG. 4 is a flowchart of creating a compressed VSAM in the method of the present invention.

FIG. 5 is a flowchart of creating a compressed VSAM in the method of the present invention.

FIG. 6 is a flowchart of a compression process in the method of the present invention.

FIG. 7 is a flowchart of a compression process in the method of the present invention.

FIG. 8 is a flowchart of a compressed VSAM access routine in the method of the present invention.

FIG. 9 is a flowchart of a compressed VSAM access routine in the method of the present invention.

FIG. 10 is a flowchart of a compressed VSAM access routine in the method of the present invention.

FIG. 11 is a flowchart of a restoration process in the method of the present invention.

Claims (3)

(57) [Claims] In a file compression method for compressing a data portion having the same value in a plurality of records, it is predicted that a plurality of records have the same value in a data portion excluding a key item of a record to be compressed. A file compression method in which a reference record with data set is created in advance, and the data portion of the compression target record whose value matches the reference record value is compared with the compression target record and the reference record. Method. 2. The file compression method according to claim 1, wherein the reference record is compressed. 3. The file compression method according to claim 1, wherein a code indicating compression is added to the compressed record.