JPH0772859B2 - Data compression method, data decompression method and device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data compression method and device, and a data decompression method and device, and is particularly used when transmitting and receiving (transmitting) data between computers using communication lines. Suitable for transmitting the newly created data file to another computer when the old data file is stored and one computer has updated the old data file to create a new data file Data compression method and device, and data decompression method and device.

[0002]

2. Description of the Related Art In communication between computers, various data compression techniques are used to improve communication efficiency. Basically, these data compression methods cannot perform efficient compression unless the data has a high degree of redundancy or regularity.

The following is an example of a conventional compression method.

When two or more same characters are consecutive, the two characters are arranged, and subsequently, data representing the number of consecutive same characters is added after the above two characters.
For example, when two same characters are consecutive like AA, the compressed data is AA0. The compressed data of AAA becomes AA1 and the compressed data of AAAA becomes AA2.

This method has a problem that the compressed data becomes longer in the case where many consecutive two characters such as AA are included. Also, in the case of three consecutive characters such as AAA, there is no compression effect.

In the conventional data compression method, if the data has a high degree of redundancy or the data has a strong regularity, a high compression rate can be obtained by taking advantage of its characteristics. However, when it comes to data such as executable files (programs) that have a strong randomness, there is a problem that the compression effect is very small, or the data length becomes rather long as described above. .

[0007]

SUMMARY OF THE INVENTION The present invention provides a data compression method and apparatus capable of highly efficient data compression even for a data file having a strong randomness as long as a reference file exists. The purpose is to

It is another object of the present invention to provide a data decompression method and device capable of restoring the data compressed by the above data compression method and device to the original data.

In the data compression method according to the present invention, the data file to be data-compressed and the corresponding reference file are partially compared, and a part of the reference file has an arbitrary data length having a high data matching rate. When a record with a high data matching rate is found in the above data file, the exclusive OR of the record found in the data file and the corresponding record in the reference file is found. And compress the data obtained by this exclusive OR operation,
A compressed data record is created by adding decompression auxiliary data to the compressed data obtained by the data compression process, and the record of the data file having a low matching rate with any part of the reference file is added to the record. It creates compressed data records by adding auxiliary data for restoration, and edits these compressed data records to create a compressed file.

The data compression method according to the present invention is premised on the existence of a reference file corresponding to a data file to be data-compressed. An example of the file is a program execution format file. For example, the data to be compressed
The file is the new version program, and the corresponding reference file is the old version program. Since the version upgrade of a program is usually performed by changing a part of the program, much of the old version program remains in the new version program. Therefore, even if the programs of the old and new versions are compared sequentially from the beginning, they do not necessarily match, but if the parts to be compared are shifted from the beginning, 100%
There are many parts that match at a rate close to.

The present invention utilizes the existence of a reference file that partially matches the data file to be data-compressed, such as old and new versions of the program.

According to the present invention, first, a data file to be subjected to data compression and a corresponding reference file are partially compared, and a part of the reference file and a record having an arbitrary data length having a high data matching rate are obtained. Search in data file.

As described above, the old and new versions of the program include parts that match at a rate close to 100%.
The other parts hardly match. In this way, a portion of the reference file that is very similar (or nearly identical) to the portion of the reference file may be found, and if it is otherwise almost unmatched, the matching rate is extremely high ( It is divided into 100%) and extremely low area (0%). Therefore, if a threshold value is set in the middle of it, it is extremely easy to determine whether the data matching rate is high. When the data matching rate is continuous from 0% to 100%, a threshold value (for example, 50%) is set appropriately, and it is determined whether the data matching rate is high or low using this threshold value. be able to. It is a relative concept that the data matching rate is high or low, but a threshold value may be set for each application for carrying out the present invention, and the threshold value may be used to discriminate between high and low.

Further, in the present invention, a record means a collection of continuous data of arbitrary length, and it is determined that the file portion found in the process of searching for a file portion having a high data matching rate, the matching rate is low. The file parts that have been created, and the collection of data (including auxiliary data for decompression) obtained by performing compression processing on them.

Then, according to the present invention, when a record having a high data matching rate is found, the exclusive OR of the found record of the data file and the corresponding record of the reference file. Calculate. The data string obtained by the exclusive OR operation of two records having a high data coincidence rate includes many consecutive word 0s, and is suitable for data compression.

The data obtained by this exclusive OR operation is data compressed. As the data compression method, the above-described conventional method may be used, or the data compression method according to the present invention described in detail later may be used. A compressed data record is created by adding decompression auxiliary data to the compressed data obtained by this data compression processing.

On the other hand, the record of the data file determined to have a low matching rate with any part of the reference file is not subjected to the compression process, but the auxiliary data for restoration is simply added to the record to obtain the compressed data record. To create. In this case, the data length is rather lengthened by the addition of the auxiliary data for restoration, but in this specification, the term "compressed data record" is also used for the data obtained by this process for the purpose of unifying the terms.

The creation of the compressed data record including the compression processing and the creation of the compressed data record not including the compression processing described above are performed for all the records constituting the data file. The compressed data record thus obtained is edited to create a compressed file. Editing here generally means that compressed data records are simply arranged in a certain order (for example, in the order of compressed data record numbers).
It means to line up. Also, the editing process will generally be performed in parallel with the compressed data record creation process.

As described above, according to the present invention, a record having a high data matching rate with a part of the reference file is searched in the data file, and the exclusive OR operation is performed between the records having a high matching rate. Therefore, even data with strong randomness can be converted into records with high redundancy. In this way, a record suitable for compression processing is obtained, so that highly efficient data compression is possible. According to the present invention, generally, compression to a fraction of a fraction is possible, and if conditions are good, compression to a size of 1/10 or less is possible. According to the result actually carried out by the inventor, the compression was possible up to 7% at the maximum.

When transmitting a data file, the communication time can be greatly reduced by transmitting the file compressed according to the compression method of the present invention.

In one embodiment of the present invention, it is first determined whether a data file or a portion thereof is capable of data compression by itself without the use of a reference file. Then, if possible, the data file or part of it is data compressed by itself. If it is not possible, the data compression method using the above-mentioned reference file is executed.

Whether or not the data file or a part thereof can be compressed by itself can be judged from the viewpoint of whether the data contained in the data file has high redundancy or whether it has regularity. . Even in this case, a certain standard for data compressibility should be set,
It is preferable to judge according to the standard.

Specifically, the above-mentioned restoration auxiliary data includes a record number, data indicating a position of a record having a high matching rate in a data file (offset), and data indicating a position of a record having a high matching rate in a reference file. (Offset), data indicating whether compression processing using a reference file is performed, data indicating whether data is compressed, data indicating data size, data relating to data compression processing, etc. may be included.

The present invention also provides a data decompression method for restoring a data file from a compressed file created by data compression as described above. In this data decompression method, the same reference file as that used in the data compression method is used. Decompression of the compressed file may be performed in the reverse order of the data compression processing described above.

The data decompression method according to the present invention determines whether or not decompression processing is required by referring to the decompression auxiliary data for each compressed data record of a compressed file and determining that decompression processing is necessary. Is decompressed from the compressed data record, and the exclusive-OR operation is performed between the decompressed record and the corresponding record in the reference file to create a data record. The data file is restored by editing the data record and the data record contained in the compressed data record that does not require decompression processing by referring to the auxiliary data for restoration.

Since the original data file is restored in this way, the restored data file can be used.

In one embodiment of the data decompression method according to the present invention, when the data file or a part thereof is data-compressed by itself, the compressed file or a part thereof is used without using a reference file. To decompress and restore the data file. This decompression method is used for data decompression without using the above-mentioned reference file.
It corresponds to the method of data compression in the file or a part thereof.

The present invention further provides a data transmission method.

In a system to which this data transmission method is applied, it is necessary that both the transmitting device that transmits the compressed data and the receiving device that receives the compressed data transmitted from this transmitting device hold the reference file. Assumption. The transmitter compresses the data according to the above-described data compression method, and transmits the compressed data obtained by this processing to the receiver. In the receiving device,
The received data is decompressed according to the decompression method described above to restore the original data.

The present invention further provides a data compression device and a data decompression device. These data compression and decompression devices correspond to the above-described data compression and decompression methods, respectively.

The data compression apparatus according to the present invention partially compares a data file to be data-compressed with a reference file corresponding to the data file, and compares a part of the reference file with an arbitrary data length having a high data matching rate. Means for searching the record in the data file, and when a record having a high data matching rate is found by the search by the searching means, the record found in the data file corresponds to that in the reference file. Data compression means for performing an exclusive OR operation with the record to be recorded and for compressing the data obtained by the exclusive OR operation,
And a compressed data record is created by adding decompression auxiliary data to the compressed data obtained by the data compression means, and the record of the data file having a low matching rate with any part of the reference file is the record. A compressed data record is created by adding decompression auxiliary data to, and editing means is provided for editing these compressed data records to create a compressed file.

The data decompression device according to the present invention requires a decompression process by the above-mentioned judging means for deciding whether or not the decompression process is necessary by referring to the decompression auxiliary data for each compressed data record of the compressed file. If it is determined that the compressed data record is decompressed, the data is decompressed and the exclusive-OR operation is performed between the decompressed record and the corresponding record in the reference file to create a data record. A data file by editing the decompressing means and the data record created by the data decompressing means and the data record included in the compressed data record that does not require decompression processing by referring to the auxiliary data for restoration. Is provided with an editing means for restoring.

[0033]

[0034]

[0035]

[0036]

[0037]

Further, the present invention partially compares the data file to be data-compressed and the corresponding reference file, and records a part of the reference file and a record having an arbitrary data length having a high data matching rate. It provides a way to find it in a data file.

A method of finding similar records according to the present invention is a method of finding a first record of a first predetermined length from a data file.
Of the first partial data of the first predetermined length from the reference file and shifting the position of extracting the first partial data of at least a predetermined byte in the reference file, By comparing the above-mentioned first partial data extracted from the reference files with each other, it was investigated whether or not there was the first partial data with a high matching rate, and the first partial data with a high matching rate was found. Sometimes, the shift amount for taking out the first partial data is fixed, and the second partial data having a second predetermined length shorter than the first predetermined length is provided in the vicinity of the fixed taking-out position. Extract from the file and the reference file respectively, and shift their extraction positions by a specified number of bytes in the reference file and the data file. , By comparing the second partial data taken out from each of the data files and the reference files to each other, the range of the high concordance rate records are those determined in the data file and the reference file.

In one embodiment of the method for finding similar records according to the present invention, one block data having a fixed data length is extracted from the data file, and the extracted one block data is stored in the reference file. When the above investigation processing is performed while shifting the extraction position of the first partial data, and the first partial data with a high matching rate is not found, the position of one block data to be extracted from the above data file is set to one block length. After shifting by a minute, the above-mentioned investigation process is repeated.

According to the present invention, the partial comparison between the data file and the reference file is performed in two steps. In the first stage, the first partial data having a relatively long data length is extracted from the data file and the reference file and roughly compared. In the comparison process of the first stage, if a portion with a matching rate equal to or higher than a predetermined value is found, the process proceeds to the second stage. In the second stage, the extraction position of the first partial data having a high coincidence rate found in the first stage is fixed, and the second partial data having a relatively short data length in the vicinity of the fixed position is converted into the data. A range of records of arbitrary length with a high coincidence rate by extracting from the file and the reference file and performing more detailed comparison processing while shifting the second partial data separately or together by predetermined bytes. In the data file and the reference file.

Thus, according to the present invention, a part of the data file and a part of the reference file are roughly compared in the first step, and after a similar part is found, a detailed comparison process in the second step is performed. By proceeding, similar records can be found in the data file and the reference file in a much shorter time than if the detailed comparison process were performed from the beginning. This makes it possible to put the data compression method described above into practical use.

In this way, when a record having a high data matching rate with a part of the reference file is found in the data file, the record found in the data file and the record corresponding to it in the reference file are excluded as described above. Logical OR is calculated. The result of the exclusive OR operation is a data string including many consecutive words 0.

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[Explanation of Examples]

(1) Overview of Overall Processing It is assumed that the transmitting device 10 transmits data to the receiving device 20 as shown in FIG. These sending and receiving devices 10, 20
Is a computer system, for example, and the data transmitted is a program.

Both the sending and receiving devices 10 and 20 have the same reference file FS. The reference file FS is, for example, an old version program executed by these devices 10 and 20.

A new version program is created by upgrading the old version program in the transmitter 10. Data file F that this new version program should be sent to the receiver 20
It is D. When the receiving device 20 receives the new version program, the new version program can replace the old version program.

Data, which is the new version program
In order to transmit the file FD to the receiving device 20, the transmitting device
In 10, the data compression processing of the data file FD is executed using the reference file FS, and the compressed file FC
Is created. The compressed file FC is transmitted to the receiving device 20 by wire (for example, digital line) or wirelessly.

Compressed file FC sent from the transmitter 10
When receiving the data, the receiving device 20 executes the data decompression process of the compressed file FC using the reference file FS stored therein, and restores the data file FD.

FIG. 2 shows a state of compression processing of the data file FD using the reference file FS executed by the transmission device 10.

Data file FD and reference file FS
And are partially compared, and a record with an arbitrary data length with a high data matching rate is searched for. In the example shown in FIG. 2, the records DR ₂ , DR ₄ , DR ₅ in the data file FD are
Are records SR ₂ and S of the reference file FS, respectively.
It is assumed that it is similar to R ₄ and SR ₅ , and the data matching rate is high (hereinafter, simply referred to as “similar”). The records DR ₂ and SR ₂ have the same data length. Similarly, the records DR ₄ and SR ₄ , and DR ₅ and SR ₅ have the same data length. The data lengths of the records DR ₂ , DR _4, and DR ₅ may be the same or different. Generally, the data lengths of the records DR ₂ , DR ₄ and DR ₅ will be different.

The reference file FS is an old version program as described above, and the data file FD is a new version program created by improving a part of the old version program. Therefore, the part of the reference file FS that has not been modified constitutes a part of the data file FD as it is. In this way, the data file FD and the reference file FS have many parts (records) that are almost identical to each other, so that the records DR ₂ and SR that are similar to each other are similar.
₂ , DR ₄ and SR ₄ , DR ₅ and SR _5, etc. can be extracted.

In the data file FD, the records DR ₁ , DR ₃ , DR _n, etc. cannot be found in the reference file FS in the portion having a high (similar) data matching rate. These newly added routines and completely rewritten program parts in the program version upgrade are recorded in these records D.
It would correspond to R ₁ , DR ₃ , DR _n, etc.

Subsequently, an exclusive OR (hereinafter referred to as XOR) is calculated between the record of the data file FD and the corresponding record of the reference file FS which are determined to be similar to each other, and the calculation result is the data. After being compressed, a compressed data record is created using this compressed data and a decompression auxiliary code to be described later, and written into the compressed file FC.

XOR is performed between the record DR ₂ of the data file FD and the record SR ₂ of the reference file FS similar to the record DR ₂ to create the intermediate data XRD ₂ .
Since the records DR ₂ and SR ₂ contain a lot of data that match each other, the intermediate data XRD ₂ that is the XOR operation result of them is data containing a lot of 0s. This intermediate data XRD ₂ has high redundancy and is suitable for data compression. A known method can be adopted as the data compression processing method, but the method described later is preferably used. The compressed data generated by the data compression processing is edited together with the decompression auxiliary code according to the format described later to become the compressed data record dr ₂ .

Similarly, the intermediate data XRD ₄ is created by the XOR operation of the records DR ₄ and SR _4, and the intermediate data XRD is made by the XOR operation of the records DR ₅ and SR _5.
RD ₅ is created. These intermediate data XRD ₄ , X
Each RD ₅ is compressed and compressed data record dr according to the compressed data record format.
₄ and dr ₅ .

Data records DR ₁ , DR for which similar parts could not be found in the reference file FS
_Since compression processing cannot be performed on ₃ , DR _n, etc., they are directly edited together with the decompression auxiliary code in accordance with the compressed data record format to become compressed data records dr ₁ , dr ₃ , dr _n, etc. These records dr ₁ , dr ₃ and dr _n are the original records D
Although R _1, DR _3, an amount corresponding data size of the restoration for the supplementary code than DR _n is greater, it will be referred to herein as the compressed data records for unification of terminology.

A compressed file FC is obtained by arranging the compressed data records dr ₁ , dr ₂ , ..., Dr _n created in this way in a fixed order (for example, the order of record NO. Described later). In the actual process, the compressed data record created every time the compressed data record is created will be arranged in the compressed file FC.

Compared to the original data file FD, the compressed file FC generally has a fraction of 1 to 1/10,
Data is compressed more than that. The compressed file FC has compressed data records dr ₁ , dr ₃ , dr _n.
, The records are not compressed, but the number is relatively small, and the records dr ₂ , d
Although the data is compressed like r ₄ and dr ₅ , the compression rate is high, so that a considerably high compression rate can be obtained even when viewed as a whole.

FIG. 3 shows a state of decompression processing of the compressed file FC using the reference file FS executed by the receiving device 20.

The decompression process is performed in the reverse order of the compression process described above. For the compressed data records dr ₂ , dr ₄ , dr _5, etc. compressed using the records in the reference file, first, the intermediate data XRD ₂ , X ₅ is decompressed.
Converted to RD ₄ and XRD ₅ . These intermediate data X
The XOR operation between RD _2, corresponding record SR ₂ of XRD _4, XRD ₅ reference file FS, SR _4, SR _5, data records DR _2, DR _4, DR ₅ is restored, respectively.

With respect to the compressed data records dr ₁ , dr ₃ and dr _n which have not been subjected to the compression processing, the decompressing auxiliary code is removed from them, so that the original data
Records DR ₁ , DR ₃ and DR _n are obtained.

Record D restored in this way
If R ₂ , DR ₄ , DR ₅ , DR ₁ , DR ₃ , and DR _n are arranged in the original order, the data file FD is finally restored.

(2) Retrieval Process for Similar Records A process for partially comparing the data file FD and the reference file FS and finding a record similar to a part of the reference file FS in the data file FD will be described.

FIG. 4 shows one block (eg, 10 blocks) from the beginning of the data file FD and the reference file FS.
(24 bytes).

As described above, the record DR ₂ of the data file FD and the record SR _{2 of the} reference file FS are similar. Record DR ₂ is data file F
The data length is OFFD ₂ from the beginning position A of D, and it starts at position B and ends at position C. The record SR ₂ starts from the beginning position D of the reference file FS at position E of length OFFS ₂ and continues to position F. As described above, the records DR ₂ and SR _{2 have} the same length, but the position B where the record DR ₂ starts and the position E where the record SR ₂ starts are different. The data length (OFFD ₂ or OFFS ₂ ) from the beginning of the file to the position where each record starts is called the offset of that record.

After all, the process of retrieving the similar records ends with the start positions B and E of the similar records DR ₂ and SR _2.
(Offsets OFFD ₂ and OFFS ₂ ) and end positions C and F are found.

In the example shown in FIG. 4, the similar records DR ₂ and SR ₂ found are shorter than one block length. Therefore, the entire record DR ₂ and the record SR _{2 are}
Is subjected to XOR operation. When the found similar records are longer than one block length, these similar records are divided by one block, XOR operation is performed for each divided block, and the compression processing of the XOR operation result is performed. , Compressed data record is created. That is, in this embodiment, all data is subjected to all processing in units of one block (or in a state where the data length is shorter than one block).

The similar record search process includes a rough first stage process and a more detailed second stage process.

First, the processing in the first stage will be described.

As shown in FIG. 5A, from one block of the data file FD and one block of the reference file FS, the first partial data fd ₁ , fd ₂ , fd of the first predetermined length are obtained.
₃ and fs ₁₀ , fs ₂₀ , fs ₃₀ are extracted. In this embodiment, the first partial data is extracted at three points, that is, the front end portion, the rear end portion, and the intermediate portion of each block. The extraction of the first partial data may be performed at one location, two locations, or four or more locations. Further, in this embodiment, the data length of the first partial data is 10 bytes.

Corresponding comrades of these first partial data are compared with each other. That is, the data fd ₁ and the data fs ₁₀ are compared with each other, and it is determined whether or not their matching rate is equal to or higher than a predetermined value. Similarly, data fd ₂ and fs
₂₀ is compared, data fd ₃ and fs ₃₀ are compared,
It is determined whether the matching rates are equal to or higher than a predetermined value. Whether or not the data match will generally be done word by word (1 byte).

In any of the comparisons, if the matching rate is less than or equal to the predetermined value, the first partial data fd ₁ , fd ₂ , fd ₃ in the data file FD is left as it is and extracted in the reference file FS. The first partial data to be shifted is shifted backward by 1 byte as shown in FIG. 5 (B). Partial data to be extracted next in the reference file FS are fs ₁₁ , fs ₂₁ , and fs ₃₁ . These partial data fd ₁ , fd ₂ , fd ₃ are compared with fs ₁₁ , fs ₂₁ , fs ₃₁ , respectively, and the data matching rate is determined.

The above processing is repeated while shifting the partial data to be taken out from the reference file FS by 1 byte.

As shown in FIG. 5C, the reference file FS
The partial data to be extracted from the reference file FS is shifted by m bytes from the beginning, and fd ₁ , fd ₂ , fd ₃ and fs
_If the matching rate of the partial data fd _{2 of the} data file and the partial data fs _2m of the reference file exceeds a predetermined value when comparing _{1 m} , fs _2m , and fs _3m , respectively, these partial data fd ₂ , Since it is considered that the data file FD and the reference file FS are similar in a certain range including fs _2m , the process proceeds to the second stage to determine the similar range.

When the concordance rate of the partial data fd ₂ and fs _2m exceeds a predetermined value, the first fetched from the reference file FS
Shift the partial data of 1 byte by 1 byte from the position of fs _2m , and the first partial data and data file FD
Of comparing the partial data fd _2, more when to locate partial data of high reference file FS most consistent rate and partial data fd ₂ preferred.

If the first partial data to be taken out from the reference file FS is sequentially shifted by 1 byte,
When the first partial data to be extracted reaches the end of the reference file FS, it is determined that the reference file FS does not have a portion similar to the one-block-length data previously extracted from the data file FD. , The data of the next one block length is taken out from the data file FD, and the same processing as above is repeated.

Next, the second stage processing will be described with reference to FIG.

In the second stage processing, the positions of the first partial data fd ₂ and fs _2m for which the matching rate is determined to be equal to or higher than the predetermined value (or the highest matching rate) in the first stage processing are fixed. To do. That is, the shift amount m of the reference file FS is fixed. And the first of the data file FD
The second partial data d ₁ having a shorter data length than the first partial data is extracted from the partial data fd ₂ of the above. In this embodiment, the second partial data d ₁ has a length of 1 byte (1 word)
Therefore, the second partial data will be referred to as a word hereinafter. See taken out word S _11, S ₁₂ or S ₁₃ and the like in the vicinity of the position corresponding to the word d ₁ from the file FS, sequentially compares with these and word d _1.

If the words d ₁ and S ₁₂ match,
Next, these left or right adjacent words are extracted from both files FD and FS to check whether they match. Words to be extracted from both files FD and FS are shifted one word at a time in one direction, and are sequentially compared, for example, d ₂ and S ₂₂ , d ₃ and S ₃₂ until a predetermined set of unmatched words appears.

If the words d ₁ and S ₁₁ also match, the shift amount of the reference file FS is fixed to (m-1), and words adjacent to the left or right of these are similarly extracted and matched. Find out if While shifting the words to be taken out one word at a time, until a predetermined set of unmatched words appears, for example, d ₂ and S ₂₁ , d ₃ and S _31.
It will be compared sequentially.

In this way, the widest range is searched from the range in which a large number of matching word sets (more than a predetermined ratio) appear, and the range is searched for in the data file FD, B to C, and in the reference file FS. Identify E to F. B and E are the offsets of the records found in this way.

If word d ₁ and words S ₁₁ , S ₁₂ , S
_{If 13} mag does not match, data file FD
, The word d ₄ adjacent to the word d ₁ and the word S ₁₂ near the corresponding position in the reference file FS,
Comparing the S _13, S ₁₄ or the like, so that the search for a set of matching word.

Referring again to FIG. 4, in this way the record DR ₂ of the data file FD and the reference file FS
If it is determined that the record SR ₂ is similar to the record SR _{2 of the}
Data of one block length following R ₂ is read, and the above-mentioned first and second steps are performed on the data of one block length.

By repeating such processing, a record similar to the part of the reference file FS is found in the entire area of the data file FD.

(3) Compressing Process An example of the process of compressing the record DR _i of the data file FD similar to the record SR _i of the reference file FS thus found by using the record SR _i will be described. .

When the records DR _i and SR _i are longer than one block length, as described above, X is calculated for each block.
An OR operation is performed and compression processing is performed for each block.

Intermediate data of one block length (or shorter than this) after the XOR operation is word 00 as described above.
It contains a lot of (hexadecimal expression, the same below). With respect to all the words forming such intermediate data, statistics of the appearance frequency are taken, and a predetermined number of types of words are selected in order from the one having the highest appearance frequency. In this embodiment, the appearance frequencies are 1 and 2.
No. 3 and No. 3 words are selected, and these are designated as H1, H2, and H3. H1, H2, and H3 are referred to as high-frequency words. In most cases H1 will be word 00.

Next, the number of types required for compression is searched for words that do not appear in the intermediate data of one block length. In this embodiment, it is assumed that nine kinds of words are required, which are represented by S1, S2, S3, ..., S9. These words are called replacement words.

Further, in this embodiment, continuous characters that frequently appear are also subject to data compression. For example, a continuous character of this kind is 0D0A which indicates a line feed and a carriage return which often appear in the case of text data.

The following replacement is performed for data compression.

When two words H1 are consecutive, these are replaced by the word S1.

When three consecutive words H1 are present, these are replaced by the word S2.

When four words H1 are consecutive, these are replaced by the word S3.

When five or more words H1 are consecutive, these are expressed in combination with the word S4 and a numeral representing the number of consecutive words (up to 255 consecutive words can be represented by 8 bits).

When two words H2 are consecutive, these are replaced by the word S5.

When three or more words H2 are consecutive, these are expressed by a combination of the numbers representing the number of consecutive words S6.

When two words H3 are consecutive, these are replaced by the word S7.

When three or more words H3 are consecutive, these are represented by a combination of the numbers representing the consecutive numbers with the word S8.

Replace consecutive characters 0D 0A with word S9.

These are summarized as follows.

H1 H1 → S1 H1 H1 H1 → S2 H1 H1 H1 H1 → S3 When H1 is 5 or more in succession → S4 + number H2 H2 → S5 H2 is 3 or more in succession → S6 + number H3 H3 → S7 H3 When three or more are consecutive → S8 + number 0D 0A → S9

The XOR operation results of similar records include many words 00 so that they are repeated many times, and the number of types of words that appear is not large. Therefore, the above-described compression method enables very efficient data compression.

FIG. 7 shows a state of compression processing according to the above compression method. In this figure, numbers other than high-frequency words and replacement words are expressed in hexadecimal. Words other than high-frequency words (eg 88 in FIG. 7)
Are arranged as they are.

In the compressed data, a column CC of replacement words and a column CP of high-frequency words precede the actual compressed data D1.
Cs are arranged in a fixed order.

Decompression processing of compressed data is performed by following the above compression processing procedure in reverse.

(4) Compressed Data Record Format FIG. 8 shows the compressed data record format. This compressed data record is also variable length.

The compressed data record is record No. R.
NO, offset OFFD from the beginning in the data file, offset OFFS from the beginning in the reference file, reference flag CF indicating whether compression processing is performed using the reference file, compression indicating whether compression processing is performed Flag CPF, size data SZ indicating the length of the following data (replacement word string CC, high-frequency word string CPC and actual compressed data D1) (the above is referred to as auxiliary data for restoration), replacement word string CC, high frequency It is composed of a word string CPC and actual compressed data D1. For records such as records DR ₁ , DR ₃ , and DR _n that are not compressed, the uncompressed data of these records is followed by decompression auxiliary data RNO, OFFD, OFFS, CF, CPF, SZ. By arranging (uncompressed data) D2, a compressed data record is created.

As will be described later, in this embodiment, the compression process is also performed without referring to the reference file FS.
If the data file FD (or one block thereof) itself has high redundancy or regularity, the compression process can be performed without using any reference file.
Also in this case, the compressed data obtained by compressing each block is edited according to the format shown in FIG.

The above-mentioned reference flag CF is used when the data file is compressed by itself without using the reference file as described above, and DR ₁ , DR ₃ , D.
It is reset when a similar record such as R _n cannot be found in the reference file, and is set when the record in the reference file is used for compression.

The compression flag is reset when similar records such as DR ₁ , DR ₃ , and DR _n cannot be found in the reference file and the compression processing is not performed, and other reference files are used. It is set if the compression process is performed regardless of whether or not.

(5) Process Flow FIG. 9 shows a flow of a data file compression process procedure in the transmission device 10.

After initial processing such as file opening (step 101), data of one block length is read from the data file FD (step 103), and the one block data itself does not use the reference file FS. It is determined whether or not the compression is possible (step 104).
As described above, this is determined based on whether the redundancy is high, the regularity is strong, or the like.

If the compression process is possible by itself, the reference flag CF is reset, the compression flag CPF is set (step 105), and the process advances to the compression process, and the above-mentioned compression process or data compression by a known method is performed. (Step 110). The compressed data is edited according to the above-mentioned compressed data record format, and is stored at a predetermined position in the compressed file (FC) (for example, record NO.
It is stored (at positions following the order) (step 112).

If the file itself is not compressible, the reference file FS is searched and it is checked whether a similar record exists in the reference file FS (step 106).

If there are similar records, the XOR operation is performed between the similar records in the data file FD and the reference file FS as described above, and intermediate data is obtained (step 108). Both the reference flag CF and the compression flag CPF are set (step 109), and data compression is performed by the above-described or known method (step 109).
110), according to the compressed data record format, is written in a predetermined position of the compressed file (FC) (step 112).

If no similar record is found, both the reference flag CF and the compression flag CPF are reset (step 111) and the data is written to the compressed file FC according to the compressed data record format without being compressed. (Step 112).

The above processing is repeatedly executed for each block length, and when the processing is completed for all the data in the data file FD (step 102), all the processing is completed by the ending processing such as file closing (step 113). Processing is complete.

FIG. 10 shows the procedure of the decompression process of the received compressed file in the receiving device 20.

After the initial processing (step 121), the data of one compressed data record is read from the compressed file FC (step 123), and it is judged by referring to the compression flag CPF whether or not the compression processing has been performed (step 121). Step 12
Four ).

If the compression processing has been performed, the data expansion processing is executed (step 125), and if not, this step 125 is skipped.

Next, by checking the state of the reference flag CF, it is checked whether or not the compression uses the reference file FS (step 126), and if so, the corresponding record is obtained by using the reference file offset OFFS. Is read (step 127) and the decompressed data and the record of the reference file FS are subjected to XOR operation (step 128). As a result, the original data is restored, and the data file offset OFFD is referred to and the data is written in the corresponding location of the data file FD (step 129).

When the data is not compressed, the actual data is as it is. When the data is compressed by itself without using the reference file, the data decompressed in step 125 is used as a clue for the data file offset OFFD. It is written to the original location of the data file FD (step 129).

The above processing is executed for each compressed data record of the compressed file FC, and when the processing is completed for all the compressed data (step 122), all the processing is completed through the ending processing (step 130).

[Brief description of drawings]

FIG. 1 is a block diagram illustrating a data file communication system.

FIG. 2 illustrates a data compression process using a reference file.

FIG. 3 illustrates a data decompression process using a reference file.

FIG. 4 shows one block data of a data file and a reference file.

FIG. 5 shows a first stage of similar record search processing.

FIG. 6 shows a second stage of similar record search processing.

FIG. 7 shows an example of original data and compressed data obtained by compressing the original data.

FIG. 8 shows a compressed data record format.

FIG. 9 is a flow chart showing a compression processing procedure of a data file in the transmission device.

FIG. 10 is a flowchart showing a procedure of decompressing a compressed file in the receiving device.

[Explanation of symbols]

 10 transmitter 20 receiver

Claims (13)

[Claims] 1. A data file to be subjected to data compression and a reference file corresponding to the data file are partially compared, and a part of the reference file and a record having an arbitrary data length having a high data matching rate are recorded in the data file. When a record having a high data matching rate is found in the file, an exclusive OR operation is performed between the record found in the data file and the corresponding record in the reference file, and The data obtained by the exclusive OR operation is data-compressed, and the auxiliary data for restoration is added to the compressed data obtained by the above-mentioned data compression processing to create a compressed data record, and any part of the reference file can be created. For records in the above data files that have a low match rate, add auxiliary data for restoration to those records. To create a reduced data record, to create a compressed file to edit these compressed data records, data compression method. 2. Determining whether the data file or part thereof can be data compressed by itself without using the reference file, and if possible, the data file or part thereof by itself. A data compression method, wherein data is compressed, and if not possible, the data compression method according to claim 1 is executed. 3. The auxiliary data for restoration is a record number, data indicating the position of a record with a high matching rate in a data file, data indicating the position of a record with a high matching rate in a reference file, and compression using a reference file. The data compression method according to claim 1 or 2, which includes data indicating whether processing is performed, data indicating whether data is compressed, data indicating a data size, and data related to data compression processing. 4. A compressed file is checked for each compressed data record by referring to its auxiliary data for decompression to determine whether decompression processing is required. If it is determined that decompression processing is required, the compressed data The record is decompressed, an exclusive OR operation is performed between the decompressed record and the corresponding record in the reference file to create a data record, and the data record created by the above process and the decompression Data contained in compressed data records that do not require processing
A data decompression method in which a data file is restored by editing records and auxiliary data for restoration. 5. A method of decompressing the compressed data record to restore the data file without using the reference file, when the data file or a part thereof is data-compressed by itself. Item 4. The data decompression method according to Item 4. 6. The auxiliary data for restoration is a record number, data indicating a position of a record having a high matching rate in a data file, data indicating a position of a record having a high matching rate in a reference file, and compression using a reference file. The data decompression method according to claim 4 or 5, which includes data indicating whether processing is performed, data indicating whether data is compressed, data indicating a data size, and data related to data compression processing. 7. The system according to claim 1, wherein the transmitting device transmitting the compressed data and the receiving device receiving the compressed data transmitted from the transmitting device both hold the reference file. A data transmission method, in which data is compressed according to the data compression method described in, and the compressed data obtained by this processing is transmitted from the transmission device to the reception device. 8. The system according to claim 1 or 2, wherein the transmitting device transmitting the compressed data and the receiving device receiving the compressed data transmitted from the transmitting device both hold the reference file. 6. The data compression method according to claim 4, wherein the compressed data obtained by this processing is transmitted from the transmitting device to the receiving device, and the received data is received by the receiving device.
A data transmission method that expands data according to the data expansion method described in. 9. A data file to be data-compressed and a corresponding reference file are partially compared, and a part of the reference file is replaced with a record having an arbitrary data length having a high data matching rate. When a record having a high data matching rate is found by the searching means in the file, the record found in the data file and the corresponding record in the reference file are exclusive. A data compression means for performing a logical sum operation and for compressing the data obtained by the exclusive logical sum operation, and a compressed data record by adding decompression auxiliary data to the compressed data obtained by the data compression means. To a record in the above data file that has a low match rate with any part of the above referenced file. Create a compressed data record by adding auxiliary data for restoration in the record Te, editing means for creating a compressed file by editing these compressed data records, data compression device provided with a. 10. The decompressing auxiliary data is a record number, data indicating a position of a record having a high matching rate in a data file, data indicating a position of a record having a high matching rate in a reference file, and compression using a reference file. 10. The data compression apparatus according to claim 9, including data indicating whether processing is performed, data indicating whether data is compressed, data indicating a data size, and data related to a data compression process. 11. A means for deciding whether decompression processing is required by referring to auxiliary data for decompression of a compressed file for each compressed data record, and in the case where the decompression processing judges that decompression processing is necessary. Is a data decompression means for decompressing the compressed data record and performing an exclusive OR between the data decompressed record and the corresponding record in the reference file to create a data record; Editing means for restoring the data file by editing the data record created by the decompressing means and the data record included in the compressed data record that does not require decompression processing by referring to the auxiliary data for restoration, Data decompression device equipped with. 12. The method according to claim 12,Data match with part of the reference file above
Records with a high rate of arbitrary data length are
The process of finding out in the above The first partial data of the first predetermined length from the data file
Take out the data,the aboveFrom the reference file, the first predetermined length
Extract the first partial data of
At least the position to take outthe aboveIn the reference file
While shifting by a predetermined byte,the abovedata file
andthe aboveThe above 1st extracted from each reference file
Comparing the partial data of each other with each other,
Minute data is examined, and when the first partial data with a high concordance rate is found,
The shift amount for extracting the partial data of No. 1 is fixed, and in the vicinity of the fixed extraction position, the first predetermined length
The second partial data of the second predetermined length which is much shorter than the above data
・ Extracted from the file and the reference file above
The extraction location of those files and the reference file and data above.
While shifting by a predetermined byte in the data file,
Above data file andthe aboveFrom reference file it
It is possible to compare the above-mentioned second partial data that has been extracted with each other.
The range of records with a high matching rate is
Files andthe aboveDetermine in reference fileprocessing
including，Data compression according to claim 1. Method. 13. The one-block data having a fixed data length is extracted from the data file, and the extracted one-block data is shifted while shifting the extraction position of the first partial data in the reference file. If the first partial data with a high concordance rate is not found,
13. The data compression method according to claim 12, wherein the position of one block of data to be extracted from the data file is shifted by one block length and the checking process is repeated.