JP5492103B2 - Backup apparatus, backup method, data compression method, backup program, and data compression program - Google Patents

Description

  The present invention relates to a technology such as a backup device, a backup method, and a data compression method that reduce the time required for data compression and decompression and reliability check.

  In recent years, the speed of data recording and transmission has become indispensable due to the increase in the amount of programs and operating system (OS) installed in computers and the increase in capacity of HDDs (Hard Disk Drives). ing. In addition to the increase in data volume, the importance of data has also increased, and the necessity and effectiveness of backup has permeated regardless of whether it is an individual or a company.

When a large amount of data such as backup data is recorded and transmitted to another storage medium, it is desirable to compress the data to a smaller amount in consideration of the capacity of the storage medium and the transmission efficiency. There are two types of data compression methods: lossless compression that completely returns to the original when data is restored (lossless compression) and lossy compression that does not return completely (lossy compression). It is necessary to use lossless compression for compression, except for data that does not change the meaning at all even if some data is missing or altered, such as video and audio.
As algorithms used for lossless compression, run-length encoding, Huffman encoding, and the like are widely known. In particular, Lempel-Ziv coding achieves a compression rate superior to run-length coding and Huffman coding in a normal data set.

Further, when compressing and decompressing data by lossless compression, it is necessary to perform a reliability check operation to confirm that the data before compression and the data after decompression match. As a reliability check method, a method of calculating a checksum, which is one of error detection methods, is known. In this method, data before compression is divided, and the total value is calculated by regarding the divided data as numerical values. It is possible to detect an error in the process from compression to decompression by checking whether the calculated value (checksum) matches the checksum calculated from the decompressed data after compression.
As algorithms for calculating the checksum, there are methods that use a hash function as a checksum in a broad sense, from a simple method of adding all data in units of bytes.
The hash function generates fixed-length data from a given value, but is a one-way function and cannot be inversely transformed.
Therefore, the original text cannot be reproduced from the calculated value, and it is extremely difficult to create different data with the same hash value. Yes. MD5 (Message Digest 5) and SHA-1 (Secure Hash Algorithm-1) are known as methods for calculating a checksum using a hash function.

In general, when backing up data on a storage medium using the above-described compression technology, in order to back up all data on the storage medium including the OS, not on a file basis, but on a sector basis, read from the beginning of the storage medium as needed. It is necessary to compress the read data. It is also necessary to create a checksum as a reliability check when restoring data. Therefore, the backup data is composed of two data, that is, the compressed data of the entire storage medium and the checksum.
At the time of data expansion, the compressed data is expanded and written from the beginning of the storage medium. The written data is read again, and a checksum is calculated and compared with a checksum created at the time of data compression, thereby checking whether there is any garbled data at the time of compression, decompression, or writing (for example, Patent Document 1). reference).

JP 2008-176420 A

  As described above, when data compression / decompression and reliability inspection are performed, an increase in processing time accompanying an increase in the capacity of a storage medium mounted on an electronic computer cannot be avoided. In particular, when the data of the entire storage medium is a processing target, the data must be compressed even in a storage area that is not used by the OS. In general, even if the OS is an unused storage area, an irregular list of data is stored in a storage area that has been used once. And no different.

  In addition, when backing up data on the entire storage medium, it is necessary to calculate checksums even for unused storage areas, and it takes time to calculate compression rates, compression and decompression, and checksums for large-capacity storage media. This is a problem.

  Therefore, the present invention has been made in view of the above problems, and provides a backup device, a backup method, a data compression method, a backup program, and a data compression program that reduce the time required for data compression and decompression and reliability inspection. The task is to do.

  In order to solve the above problems, a backup device according to the present invention is a backup device that compresses and backs up data in a storage area of a backup target storage device to be backed up. An unused storage area data clear unit that clears data and sets the data in the storage area as cleared data, and determines whether the cleared data is cleared data or not When the first clear data determination unit and the first clear data determination unit determine that the data is cleared, a clear data check code that is a check code of the cleared data calculated in advance is acquired, and the first When the clear data determination unit determines that the data is not cleared, the data obtained by subdividing the cleared data When the first check code generation unit that calculates the check code and generates the first check code information for performing the reliability check and the first clear data determination unit determine that the data is cleared, When the clear data compression data that is the compression result of the calculated cleared data is acquired and the first clear data determination unit determines that the data is not cleared, the compression result is obtained from the data obtained by subdividing the cleared data. A compressed data generating unit that generates compressed data that is calculated and compressed data, a compressed data expanding unit that expands the compressed data and generates expanded data, and data obtained by subdividing the expanded data, A second clear data determining unit for determining whether or not the data is cleared, and data obtained by clearing the second clear data determining unit. If it is determined, to obtain the clear data check code calculated in advance, if the second clear data determination unit determines that the data is not cleared, to calculate the check code from the data subdivided the expanded data, A second check code generation unit that generates second check code information for performing a reliability check, and a check code comparison unit that compares the first check code information with the second check code information. And

  According to the present invention, it is possible to reduce the time required for data compression and decompression and reliability inspection.

It is a block diagram which shows the structure of the backup device in embodiment of this invention. It is a figure which shows an example of the 1st checksum information in embodiment of this invention. It is a flowchart which shows the operation | movement of the unused area data clear process by the unused area data clear part of the backup device in embodiment of this invention. It is a flowchart which shows the operation | movement of the 1st checksum information generation process by the 1st clear data determination part and 1st checksum production | generation part of the backup device in embodiment of this invention. It is a flowchart which shows the operation | movement of the compression data generation process by the 1st clear data determination part of the backup apparatus in embodiment of this invention, and a compression data generation part. It is a flowchart which shows the operation | movement of the compression data expansion | deployment process by the compression data expansion | deployment part of the backup apparatus in embodiment of this invention, a 2nd clear data determination part, a 2nd checksum production | generation part, and a checksum comparison part.

Hereinafter, embodiments of the present invention will be described with reference to FIGS. In each figure, common portions are denoted by the same reference numerals, and redundant description is omitted.
As shown in FIG. 1, the backup device 1 compresses data in the backup target storage device 2 and stores it in the backup storage device 3 in order to back up data in the backup target storage device 2.
The backup device 1 decompresses the compressed data stored in the backup storage device 3, records it again in the backup target storage device 2, and performs a reliability check.
The storage area of the backup target storage device 2 includes a used area that stores data existing as a file and an unused area that is a so-called empty area. The backup device 1 backs up data in the entire storage area including the unused area of the backup target storage device 2.
The backup target storage device 2 is a storage medium that stores data to be backed up, and is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like. The backup target storage device 2 may be mounted on an electronic computer.
The backup storage device 3 is a storage medium for storing backup data, and may be external or internal to the backup device 1.

[Configuration of Backup Device 1]
As shown in FIG. 1, the backup device 1 includes an unused area data clear unit 11, a storage unit 12, a first clear data determination unit 13, a first checksum generation unit 14, and a compressed data generation unit 15. A compressed data expansion unit 16, a second clear data determination unit 17, a second checksum generation unit 18, and a checksum comparison unit 19.

The unused area data clear unit 11 clears the unused area data of the backup target storage device 2.
Here, the clearing of data in unused areas will be described. In general, data compression tends to have a high compression rate for a regular data enumeration and a low compression rate for a random data enumeration. Even in a storage area that the OS has determined to be unused, a list of irregular data remains stored in the storage area that has been used once. Therefore, the unused area can be said to be a storage area having a relatively low compression rate even if it is unused. Therefore, a regular list of data is stored in the unused area, and the compression rate of the data in the unused area is increased. In this way, storing a list of regular data in an unused area is called clearing the data in the unused area.

The method of clearing the data in the unused area is performed by creating a certain amount of binary file in which the contents are, for example, a list of 0 in the unused area. Create this file for all unused areas. After that, all these binary files are deleted and set as unused areas again. Then, a list of 0 is stored in this unused area.
As a result, since the regular data list is stored in the unused area of the backup target storage device 2 that is the compression target, the irregular data list before the data is cleared is compressed. Also, the effect of improving the compression rate can be obtained.
Note that the data in the entire storage area of the backup target storage device 2 including the unused area subjected to the data clearing process and the used area is referred to as cleared data.

The storage unit 12 stores a clear data checksum 121, which will be described later, and clear data compressed data 122.
The clear data checksum 121 is a checksum of a regular list of data. For example, the clear data checksum 121 is a checksum of a certain amount of zeros. Therefore, the clear data checksum 121 can be calculated in advance.
The clear data compression data 122 is a compression result of a regular list of data. Therefore, this clear data compressed data 122 can also be calculated in advance.

  The first clear data determination unit 13 acquires data obtained by subdividing the cleared data, and determines whether or not the data obtained by subdividing the cleared data is cleared data. The data obtained by subdividing the cleared data is data obtained by dividing the cleared data at an appropriate interval for calculating the checksum. A method for determining whether or not the data is cleared is performed by determining whether or not the data obtained by subdividing the cleared data is a list of regular data. For example, the first clear data determination unit 13 determines that the data obtained by subdividing the cleared data is cleared data when the data includes only 0 series. Therefore, even if the data in the cleared data is originally stored in the use area, the data that has only been listed as 0 due to the subdivision is the cleared data. It is determined that there is.

  When it is determined that the data obtained by subdividing the cleared data is the cleared data, the first checksum generation unit 14 acquires the previously calculated clear data checksum 121 and uses the cleared data as the cleared data. When it is determined that the subdivided data is not cleared data, a checksum for data obtained by subdividing the cleared data is calculated. There are various methods for calculating the checksum by calculation according to the conventional technique. For example, the checksum is calculated by considering the data obtained by subdividing the cleared data as a numerical value and calculating the total value.

  Thereby, although the determination process which determines whether the 1st clear data determination part 13 is the cleared data increases, the process which calculates a checksum can be reduced significantly. The process of calculating the checksum has a greater influence on the overall required time than the determination process. Therefore, it is possible to reduce the time required to calculate the checksum of the entire cleared data.

Further, the first checksum generation unit 14 adds the acquired clear data checksum 121 or the calculated checksum to the first checksum information 31 to generate the first checksum information 31.
Here, the first checksum information 31 will be described with reference to FIG.
As shown in FIG. 2, the first checksum information 31 includes a checksum calculation for each predetermined storage area section of the backup target storage device 2 for which the checksum has been calculated (that is, for each piece of data obtained by subdividing the cleared data). A start address 311 is described. The predetermined storage area section is, for example, every sector. The checksum usually has a different value like the checksum indicated by reference numeral 312. On the other hand, the checksum of the data determined to be cleared data has the same value (that is, the clear data checksum 121) as the checksum indicated by reference numeral 313.
In this specification and the like, a checksum is included in the concept of a check code.

  When it is determined that the data obtained by subdividing the cleared data is the cleared data, the compressed data generation unit 15 acquires the preliminarily calculated clear data compressed data 122 and subdivides the cleared data. If it is determined that the processed data is not cleared data, compressed data is calculated by compression processing using data obtained by subdividing the cleared data. There are various methods for calculating the compressed data by compression processing according to the prior art, and for example, it is performed by using a compression algorithm such as Lempel-Ziv encoding.

  As a result, for data determined to be cleared data, a compression result can be obtained without performing time-consuming compression processing, and the time required for data compression can be shortened.

  In addition, the compressed data generation unit 15 adds the acquired clear data compressed data 122 or the calculated compressed data to the compressed data 32 to generate the compressed data 32.

The compressed data expansion unit 16 acquires the compressed data 32 and expands the compressed data 32. The expanded data is called expanded data. Further, the compressed data expansion unit 16 records the expanded data in the backup target storage device 2.
The second clear data determination unit 17 acquires data obtained by subdividing the decompressed data, and determines whether the data obtained by subdividing the decompressed data is cleared data. Since the determination method is the same as that of the first clear data determination unit 13, the description thereof is omitted.

The second checksum generator 18 generates the second checksum information 21 in the same manner as the first checksum generator 14. That is, when it is determined that the data obtained by subdividing the decompressed data is cleared data, the clear data checksum 121 calculated in advance is acquired, and the data obtained by subdividing the decompressed data is cleared. If it is determined that it is not, a checksum for the data obtained by subdividing the expanded data is calculated by calculation.
Thereby, the time required to calculate the checksum of the entire decompressed data can be shortened.
Note that the calculation for calculating the checksum at this time is the same as the calculation for calculating the checksum during compression. Further, the decompressed data is subdivided into a segment in which the cleared data is subdivided when calculating the checksum at the time of compression.

  The checksum comparison unit 19 determines whether or not the first checksum and the second checksum match. If the first checksum and the second checksum match, the subdivided data in the section is restored without error. On the other hand, if the first checksum and the second checksum do not match, the backup device 1 outputs an error to an output device (not shown) because it is considered that there is a problem during data compression or decompression.

The backup device 1 can be realized by a general computer having a CPU (Central Processing Unit) and a memory (not shown). At this time, the backup device 1 operates by a backup program and a data compression program that cause the computer to function as each functional unit described above.
Moreover, the 1st clear data determination part 13 and the 2nd clear data determination part 17 are realizable with the same module. Further, the first checksum generation unit 14 and the second checksum generation unit 18 can be realized by the same module or the like.

[Operation of Backup Device 1]
Next, the operation of the backup device 1 will be described with reference to FIGS.
(Unused area data clear processing)
This process is a process of clearing data in an unused storage area of the backup target storage device 2.

As shown in the flowchart of FIG. 3, in step S <b> 101, the unused area data clear unit 11 creates a file for clearing data in an unused storage area of the backup target storage device 2. As described above, this file is, for example, a certain amount of binary files whose contents are a list of zeros.
In step S102, the unused area data clear unit 11 determines whether the file creation has succeeded.

If it is determined that the file has been successfully created (Yes in step S102), the process returns to step S101, and the unused area data clear unit 11 continues to create a file for clearing data.
On the other hand, if it is determined that the file creation has not been successful (No in step S102), the unused area data clearing unit 11 determines that the file has been created in all unused storage areas. The use area data clear unit 11 deletes all the files created to clear the data.
As described above, the unused area data clear unit 11 generates cleared data in the backup target storage device 2.

(First checksum information generation process)
This process is a process for generating the first checksum information 31. Further, the backup device 1 performs this processing after clearing the data in the unused area of the backup target storage device 2.

As shown in the flowchart of FIG. 4, in step S <b> 201, the first clear data determination unit 13 acquires data obtained by subdividing the cleared data.
In step S202, the first clear data determination unit 13 determines whether or not the data obtained by subdividing the cleared data is cleared data.

When it is determined that the data is cleared (step S202 / Yes), in step S203, the first checksum generation unit 14 acquires a pre-calculated clear data checksum 121.
On the other hand, if it is determined that the data is not cleared (No in step S202), in step S204, the first checksum generation unit 14 calculates a checksum from data obtained by subdividing the cleared data.

In step S205, the first checksum generation unit 14 adds the clear data checksum 121 acquired in step S203 or the checksum calculated in step S204 to the first checksum information 31, and the first checksum information 31 is added. Generate.
In step S206, the first checksum generation unit 14 determines whether or not the processing has been performed on the data in all the storage areas.

When it is determined that the processing is not performed on the data in all the storage areas (No at Step S206), the process returns to Step S201, and the backup device 1 continues the process.
On the other hand, when it determines with having implemented with respect to the data of all the storage areas (step S206 * Yes), the backup device 1 complete | finishes a 1st checksum information production | generation process.

(Compressed data generation processing)
This process is a process for generating compressed data for backup. Further, the backup device 1 can perform this process in parallel with the first checksum information generation process.

As shown in the flowchart of FIG. 5, in step S <b> 301, the first clear data determination unit 13 acquires data obtained by subdividing the cleared data.
In step S302, the first clear data determination unit 13 determines whether or not the data obtained by subdividing the cleared data is cleared data.

When it is determined that the data is cleared (Yes in step S302), in step S303, the compressed data generation unit 15 acquires the calculated clear data compressed data 122.
On the other hand, if it is determined that the data is not cleared (No in step S302), in step S304, the compressed data generation unit 15 calculates compressed data from the data obtained by subdividing the cleared data.

In step S <b> 305, the compressed data generation unit 15 adds the clear data compressed data 122 acquired in step S <b> 303 or the compressed data calculated in step S <b> 304 to the compressed data 32 to generate the compressed data 32.
In step S306, the compressed data generation unit 15 determines whether or not the processing has been performed on the data in all the storage areas.

When it is determined that the processing is not performed on the data in all the storage areas (No at Step S306), the process returns to Step S301, and the backup device 1 continues the process.
On the other hand, when it determines with having implemented with respect to the data of all the storage areas (step S306, Yes), the backup device 1 complete | finishes a compressed data production | generation process.

(Compressed data expansion processing)
This processing is processing for restoring data to the backup target storage device 2 using the compressed data for backup, and performing a reliability check to check whether the data before compression and the data after expansion match.

As shown in the flowchart of FIG. 6, in step S <b> 401, the compressed data expansion unit 16 acquires the compressed data 32.
In step S402, the compressed data expansion unit 16 expands the compressed data 32 and generates expanded data.
In step S403, the compressed data expansion unit 16 records the expanded data in the backup target storage device 2.

In step S404, the second clear data determination unit 17 and the second checksum generation unit 18 execute a second checksum information generation process. Thereby, the second checksum information 21 is generated. Since the second checksum information generation process is the same as the first checksum information generation process (see FIG. 4), the description thereof is omitted.
In step S405, the checksum comparison unit 19 determines whether or not the first checksum and the second checksum match.

If it is determined that the first checksum and the second checksum match (Yes in step S405), in step S406, the compressed data expansion unit 16 determines whether all the compressed data 32 has been expanded.
If it is determined that all the compressed data has not been expanded (No at Step S406), the process returns to Step S401, and the backup device 1 continues processing.
On the other hand, when it is determined that all the compressed data has been expanded (step S406: Yes), the backup device 1 ends the compressed data expansion process.

  If it is determined in step S405 that the first checksum and the second checksum do not match (No in step S405), the backup device 1 outputs an error to an output device (not shown) in step S407.

With the above operation, the effect of improving the compression ratio can be obtained. Further, no special algorithm is used, and the present invention can be implemented regardless of the compression algorithm.
Further, for data determined to be cleared at the time of compression, the checksum can be specified without performing the checksum calculation process, and the time required for checksum calculation can be shortened.
Further, for data determined to be cleared at the time of compression, the data of the compression result can be specified without performing data compression processing, and the time required for data compression can be shortened.
Further, for data determined to be cleared at the time of development, the checksum can be specified without performing checksum calculation processing, and the time required for checksum calculation can be reduced.

According to the present embodiment, by improving the compression ratio of the storage area, it is possible to reduce the processing time required for recording / transmission of compressed data and the capacity of the recording medium.
In addition, it is possible to obtain an effect of shortening the processing time in the work of compressing and decompressing data of the storage medium to be mounted on the electronic computer, and the reliability inspection, and to reduce the increase in the processing time due to the increase in the capacity of the storage medium. it can.

<Modification>
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to this, It can change in the range which does not deviate from the meaning of this invention.

In the embodiment of the present invention, the steps of the flowchart show an example of processing performed in time series according to the described order. However, the steps of the flowchart are not necessarily processed in time series, but are executed in parallel or individually. It also includes processing.
For example, step S403 and step S404 may be performed in parallel. That is, the backup device 1 may record the decompressed data in the backup target storage device 2 and execute the second checksum information generation process.

  In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 Backup apparatus 2 Backup object storage apparatus 3 Backup storage apparatus 11 Unused area | region data clear part 12 Memory | storage part 13 1st clear data determination part 14 1st checksum production | generation part (1st check code production | generation part)
DESCRIPTION OF SYMBOLS 15 Compressed data production | generation part 16 Compressed data expansion | deployment part 17 2nd clear data determination part 18 2nd checksum production | generation part (2nd check code production | generation part)
19 Checksum comparison part (check code comparison part)
21 Second checksum information (second check code information)
31 First checksum information (first check code information)
32 Compressed data 121 Clear data checksum (clear data check code)
122 Clear data compressed data 311 Checksum calculation start address 312 Normal checksum 313 Checksum of data determined to be cleared

Claims (8)

A backup device that compresses and backs up data in a storage area of a backup target storage device to be backed up,
An unused area data clear unit that clears unused area data of the storage area, and sets the storage area data as cleared data;
A first clear data determination unit that determines whether or not the data obtained by subdividing the cleared data is cleared data;
When the first clear data determination unit determines that the data is cleared, a clear data check code that is a check code of the cleared data calculated in advance is obtained, and the reliability is determined using the clear data check code. A first check code generation unit that generates first check code information for performing an inspection;
When the first clear data determination unit determines that the data is cleared, the clear data compressed data that is the compression result of the cleared data calculated in advance is acquired and compressed using the clear data compressed data. A compressed data generator that generates compressed data that is
A compressed data decompression unit that decompresses the compressed data and generates decompressed data;
A second clear data determination unit that determines whether or not the data obtained by subdividing the expanded data is cleared data;
When the second clear data determination unit determines that the data is cleared, the clear data check code calculated in advance is acquired, and a second check for performing a reliability check using the clear data check code A second check code generation unit for generating code information;
A check code comparison unit that compares the first check code information and the second check code information;
A backup device comprising: When the first check code generation unit determines that the first clear data determination unit is not cleared data, the first check code generation unit calculates a check code from data obtained by subdividing the cleared data, and the clear data check code or calculation The first check code information for performing the reliability check is generated using the check code,
When the compressed data generation unit determines that the first clear data determination unit is not cleared data, the compressed data generation unit calculates a compression result from data obtained by segmenting the cleared data, and the compressed data compression data or the compression result Is used to generate compressed data that is compressed data,
When the second check code generation unit determines that the second clear data determination unit is not cleared data, the second check code generation unit calculates a check code from data obtained by segmenting the expanded data, and calculates the clear data check code or the calculated The backup device according to claim 1, wherein second check code information for performing a reliability check is generated using the check code. The compressed data decompression unit records the decompressed data in the backup target storage device;
The second clear data determination unit acquires the decompressed data recorded in the backup target storage device, and determines whether the acquired and subdivided data is cleared data. The backup device according to claim 1, characterized in that: The backup device according to claim 1, wherein the check code is a checksum for digitizing and summing data. A backup method in a backup device that compresses and backs up data in a storage area of a backup target storage device to be backed up,
An unused area data clear unit of the backup device clears unused area data among the storage areas, and uses the storage area data as cleared data.
A first clear data determination unit that determines whether the first clear data determination unit of the backup device is data that has been cleared with respect to data obtained by subdividing the cleared data;
When the first check code generation unit of the backup device determines that the first clear data determination unit is cleared data, a clear data check code that is a check code of the cleared data calculated in advance is acquired. A first check code generation step for generating first check code information for performing a reliability check using the clear data check code;
When the compressed data generation unit of the backup device determines that the first clear data determination unit is the cleared data, obtains the clear data compressed data that is the compression result of the cleared data calculated in advance, A compressed data generation step for generating compressed data that is compressed data using the clear data compressed data;
A compressed data decompression step in which the compressed data decompression unit of the backup device decompresses the compressed data and generates decompressed data;
A second clear data determination step of determining whether or not the second clear data determination unit of the backup device is the cleared data with respect to the data obtained by subdividing the expanded data;
When the second check code generation unit of the backup device determines that the data is cleared by the second clear data determination unit, the clear data check code calculated in advance is acquired and the clear data check code is used. A second check code generation step for generating second check code information for performing a reliability check;
A check code comparison step in which the check code comparison unit of the backup device compares the first check code information and the second check code information;
A backup method characterized by comprising: A data compression method in a backup device for compressing data in a storage area of a backup target storage device,
An unused area data clear unit of the backup device clears unused area data among the storage areas, and uses the storage area data as cleared data.
A first clear data determination unit that determines whether the first clear data determination unit of the backup device is data that has been cleared with respect to data obtained by subdividing the cleared data;
When the first check code generation unit of the backup device determines that the first clear data determination unit is cleared data, a clear data check code that is a check code of the cleared data calculated in advance is acquired. A first check code generation step for generating first check code information for performing a reliability check using the clear data check code;
When the compressed data generation unit of the backup device determines that the first clear data determination unit is the cleared data, obtains the clear data compressed data that is the compression result of the cleared data calculated in advance, A compressed data generation step for generating compressed data that is compressed data using the clear data compressed data;
A data compression method comprising:   A backup program for causing a computer to execute the backup method according to claim 5.   A data compression program for causing a computer to execute the data compression method according to claim 6.

Images