JP2012164130A - Data division program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data division method which can enhance data compression efficiency while performing data deduplication.SOLUTION: When determining a position to divide data, a data division program preferentially saves a position nearer to the tail end of data to divide the data at the position.

Description

  The present invention relates to a program for dividing data.

  When data is stored in a computer, there is a case where processing for eliminating the overlapping portion and reducing the data size is performed. Specifically, the data is divided into blocks, and blocks having the same byte string are deleted.

  An example of performing deduplication processing on data includes processing for backing up data. This is because the backup data needs to be continuously stored, and therefore tends to increase in size, and therefore there is a need to delete duplicate data. In addition, the deduplication technology may be applied to the storage apparatus itself in the future.

  When deduplication processing is performed on backup data, the file to be backed up is first divided into blocks, and the block compressed by an appropriate method is saved in the backup medium using the hash value of the block as a key. Further, in order to make it easy to extract backup data, the hash value of each block may be held in a storage device different from the backup data.

  In order to increase the effect of the deduplication processing, it is desirable that the blocks generated by dividing the data be as identical as possible. For this reason, when performing deduplication processing, there is a problem of where to divide data.

  In Patent Document 1 below, a byte sequence is sequentially scanned from the beginning to the end, and the byte sequence is divided when a hash value of a predetermined length byte sequence becomes a predetermined numerical pattern. As a preliminary division condition, the byte string is divided even when the hash value partially matches the specified numerical value pattern.

  In Patent Document 2 below, a division frame is provided in a byte string, a feature value of each division position candidate in the division frame is obtained, and a division position is determined by comparing each feature value.

  In Patent Document 3 below, the division position is determined by evaluating the offset, hash value, and other feature values of the division position candidates.

  In Patent Document 4 below, a byte string is divided by some method, and whether or not the divided byte string includes a byte string written to the storage device first is determined by a hash value. As a result, the byte string already written in the storage device is not overwritten.

  The following non-patent document 1 describes a technique for copying data to a remote computer. Since it is difficult to remotely replicate all data at once, it is necessary to divide the target data into an appropriate size. Since overlapping blocks do not need to be duplicated, duplication efficiency can be improved by appropriately determining the division position. In this document, data is not divided at a fixed length, but is divided at a variable length to search for overlapping blocks.

US 6,810,398 B2, 2004, System and method for unorchestrated determination of data sequence using sticky byte factoring to determine breakpoints in digital sequences US 7,504,969 B2, 2009, Location-based stream segmentation for data deduplication US 2006/0047855 A1, 2006, Efficient Chunking Algorithm US 6,704,730 B1, 2004, Hash file system and method for use in a commonality factoring system

Andrew Tridgell, Efficient Algorithms for Sorting and Synchronization, Chapter 3: The rsync algorithm, 1999, dissertation, URL: http://samba.org/~tridge/phd_thesis.pdf (acquired on January 11, 2011)

  In order to increase the effect of deduplication processing, it is considered that the block size when dividing data into blocks should be as small as possible. This is because the smaller the block, the higher the probability of matching with another block.

  On the other hand, when compressing data, the larger the data size before compression, the better the compression efficiency. Therefore, when the data is compressed and stored for each block as in the above-described backup example, each block size should be as large as possible. This is in contradiction to the above requirement that is necessary to increase the efficiency of deduplication.

  Since both deduplication and data compression have the purpose of removing excess data, it is considered that priority should be given to methods that can reduce the data size overall. In particular, when both deduplication and data compression are performed, as in the backup example described above, it is important to reduce the data size as a comprehensive effect while balancing the conflicting requirements described above. It is.

  In addition, when backing up a large amount of data, it is necessary to consider the efficiency of backup processing. If the size of the divided block is reduced, the effect of the deduplication process is increased, but the number of write processes for storing the block in the backup medium is increased, resulting in an increase in the backup process time.

  For the above reasons, a data division method that can balance data deduplication and data compression is desired. In this regard, Patent Documents 1 to 4 and Non-Patent Document 1 focus on only deduplication processing, and thus do not describe the viewpoint of increasing the divided block size. Hereinafter, it demonstrates individually.

  Non-Patent Document 1 describes a method of searching for a block that overlaps with a certain block at high speed. However, as a premise thereof, it is clearly disclosed that data is divided so that the number of overlapping blocks increases. Absent.

  In Patent Literature 1, a plurality of division conditions are provided, and if no division candidate corresponding to the first division condition is found, the data is divided at a position corresponding to the preliminary second division condition. However, neither the first division condition nor the second division condition is explicitly configured to increase the size of the divided block. Therefore, in principle, it seems to operate so as to determine the division position so that the efficiency of the deduplication processing is increased.

  Even in Patent Documents 2 and 3, as in Patent Document 1, the size of the divided blocks is not necessarily increased. Further, since a plurality of division position candidates are compared, it is considered that the processing load is high and it is not suitable for an application where processing efficiency is required.

  In Patent Document 4, since it is necessary to read the written data from the storage device in order to confirm whether or not the byte string has been written, the time for accessing the storage device becomes long and the processing efficiency is not high. it is conceivable that.

  The present invention has been made in order to solve the above-described problems, and an object of the present invention is to provide a data division method capable of improving data compression efficiency while performing data deduplication.

  The data division program according to the present invention preferentially stores a position closer to the end of the data when determining the position to divide the data, and divides the data at that position.

  In the data division program according to the present invention, the position closer to the end of the data is preferentially used in the process of performing the data division for performing the data deduplication. Therefore, the processing is directed to improve the data compression effect. However, data deduplication can be implemented. Thereby, it is possible to comprehensively optimize the effect of reducing the data size by balancing data deduplication and data compression. This also makes it possible to shorten the time for backup processing comprehensively.

2 is a functional block diagram of a data dividing device 100 according to Embodiment 1. FIG. It is a program describing a process in which the data dividing device 100 divides data. It is a figure which shows the detailed process flow of the function bdb which the data storage part 140 implements. It is a program describing a process of dividing data by the data dividing device 100 according to the second embodiment. It is a program describing a process of dividing data by the data dividing device 100 according to the third embodiment. It is a figure which shows the detailed processing flow of the function bdb which the data division part 130 and the data storage part 140 implement in Embodiment 4. FIG.

<Embodiment 1>
FIG. 1 is a functional block diagram of a data dividing device 100 according to Embodiment 1 of the present invention. The data dividing device 100 is a device that performs processing for dividing data, and includes a data reading unit 110, a division condition determining unit 120, a data dividing unit 130, and a data storage unit 140.

  The data reading unit 110 reads data to be divided from a data source such as the file server 200. The division condition determination unit 120 sequentially acquires data read by the data reading unit 110 and determines whether or not a condition for dividing the data is satisfied. The data dividing unit 130 divides data into blocks at positions corresponding to the conditions determined by the dividing condition determining unit 120. The data storage unit 140 calculates a hash value of the data block generated by the data division unit 130 by the data division and stores it in the backup device 400. Further, the data block and the hash value are associated with each other and stored in the block DB 300.

  The block DB 300 is a database that stores data blocks generated by the data dividing unit 130 dividing the original data. The block DB 300 can be configured using a storage device such as a hard disk. The backup device 400 is a storage device that stores the hash value generated by the data storage unit 140.

  The data reading unit 110, the division condition determination unit 120, the data division unit 130, and the data storage unit 140 can be configured as individual functional units or can be configured integrally.

  The data reading unit 110, the division condition determination unit 120, the data division unit 130, and the data storage unit 140 can be configured as hardware such as a circuit device that realizes these functions, or may be a CPU (Central Processing Unit) or the like. It can also be configured by an arithmetic device and a program that defines its operation. Below, the example which comprised these function parts mainly with the program is demonstrated.

  FIG. 2 is a program describing a process in which the data dividing apparatus 100 divides data. For convenience of explanation, step numbers are also shown. Hereinafter, each step of FIG. 2 will be described.

(Figure 2: Step 200)
The data dividing apparatus 100 performs the following steps 210 to 223 for all the data read by the data reading unit 110. If the processing has not been completed for all the data, the process returns to this step using the “start” marker as a mark, and the same processing is repeated. It is assumed that the data read by the data reading unit 110 is stored as a byte string from the memory address indicated by the variable blockStart to the address immediately before the memory address indicated by the variable blockEnd.

(FIG. 2: Step 210)
The division condition determination unit 120 ends the process when blockEnd <= blockStart holds. The blockStart variable is updated sequentially in the following steps. This step is for confirming that the length of the byte string to be divided is 0 and determining whether or not to end the process.

(Figure 2: Step 211)
The division condition determination unit 120 determines whether blockEnd <= blockStart + blockMin holds. If it does, the data storage unit 140 writes the block between blockStart and blockEnd to the block DB 300. The data storage unit 140 calculates a hash value of the same block and stores it in the backup device 400. The above processing performed by the data storage unit 140 is described in the function bdb. Details of this function will be described later with reference to FIG.

(Figure 2: Step 211: Supplement)
blockMin is the minimum size of the divided block. Therefore, this step determines whether or not the byte sequence to be divided is smaller than the minimum size. If the target byte sequence is less than or equal to the minimum size, the process proceeds to near the end of the data, and the remaining byte sequence is small, so it is not meaningful to perform the following steps. Therefore, the remaining byte sequence is stored in the block DB 300 as it is.

(Figure 2: Step 212)
The division condition determination unit 120 initializes the variables cP1 and cP2 with 0. The variable cP1 is a variable for holding a data address corresponding to a first division condition that is a condition for dividing data. The variable cP2 is a variable for holding a data address corresponding to a second division condition that is a condition for dividing data.

(Figure 2: Step 213)
The division condition determination unit 120 initializes an address variable cP used when searching for a division position. The first position is an address one byte ahead of the position corresponding to the minimum block size. Hereinafter, each functional unit performs the following processing while sequentially reading data acquired from the file server 200 byte by byte.

(Figure 2: Step 214)
The division condition determination unit 120 repeats the processes of steps 215 to 223 while cP <blockEnd is satisfied, that is, until the address variable cP reaches the end of the division target data.

(Figure 2: Step 215)
The division condition determination unit 120 obtains a feature value hash used to determine whether the division condition is met. The process for obtaining the feature value hash is described in the function crc. Any known technique can be used as the processing content of the function crc.

(Figure 2: Step 215: Supplement)
As processing contents of the function crc, for example, it is conceivable to obtain a cyclic redundancy check value or a hash value for a byte string before a predetermined length from a specified address, and use the value as a feature value hash.

(Figure 2: Step 216)
The division condition determination unit 120 checks whether or not the feature value hash satisfies the first division condition. The first division condition is described in the function pattern1. If the feature value hash satisfies the first division condition, the current data address cP is set in the variable cP1, and the process proceeds to step 219. If the first division condition is not satisfied, the process proceeds to step 217.

(Figure 2: Step 216: Supplement)
As the first division condition, for example, whether or not the partial bit pattern of the feature value hash matches a specific bit pattern can be considered. In addition, it can be considered whether or not the remainder obtained by dividing the feature value hash by a specific value matches the specified value. The same applies to the second division condition and the third division condition described below.

(Figure 2: Step 217)
The division condition determination unit 120 checks whether or not the feature value hash satisfies the second division condition. The second division condition is described in the function pattern2. The second division condition is different from the first division condition in step 216. If the feature value hash satisfies the second division condition, the current data address cP is set in the variable cP2, and the process proceeds to step 219. If the second division condition is not satisfied, the process proceeds to step 218.

(Figure 2: Step 218)
The division condition determination unit 120 checks whether or not the feature value hash satisfies the third division condition. The third division condition is described in the function pattern3. The third division condition is different from the first division condition in step 216 and the second division condition in step 217. If the feature value hash satisfies the third division condition, the current data address cP is set in the variable cP2.

(Figure 2: Step 219)
The data dividing unit 130 checks whether or not the variable cP1 is set (not 0). If the variable cP1 is set, a position satisfying the first division condition is found in step 216. The data dividing unit 130 divides data into data blocks between the address blockStart and the address cP1. The data storage unit 140 writes the data block in the block DB 300. The data storage unit 140 calculates a hash value of the same data block and writes it in the backup device 400. The data dividing unit 130 returns block 210 to blockStart = cP1. If the variable cP1 is not set, the process proceeds to step 220.

(Figure 2: Step 220)
The data dividing unit 130 checks whether or not the address cP (current address for determining the dividing condition) has reached a position (blockStart + blockMax) corresponding to the maximum size of the divided block. If the position corresponding to the maximum size has been reached, the process proceeds to step 221; otherwise, the process skips to step 223.

(FIG. 2: Step 221)
The data dividing unit 130 checks whether or not the variable cP2 is set (not 0). If the variable cP2 is set, the position satisfying the second division condition in step 217 or the third division condition in step 218 is found. The data dividing unit 130 divides data into data blocks between the address blockStart and the address cP2. The data storage unit 140 writes the data block in the block DB 300. The data storage unit 140 calculates a hash value of the same data block and writes it in the backup device 400. The data dividing unit 130 returns block 210 to blockStart = cP2. If the variable cP2 is not set, the process proceeds to step 222.

(Figure 2: Step 222)
The data dividing unit 130 divides the data into data blocks (the size of the divided block is the maximum size according to the condition of step 220) between the address blockStart and the address cP. The data storage unit 140 writes the data block in the block DB 300. The data storage unit 140 calculates a hash value of the same data block and writes it in the backup device 400. The data dividing unit 130 returns block 210 to blockStart = cP.

(Figure 2: Step 223)
The data dividing unit 130 advances the address cP by 1 byte (in the direction toward the end of data), and returns to step 215.

(Figure 2: Step 225)
The data dividing unit 130 takes out the remaining data between the address blockStart and the address blockEnd as a data block. The data storage unit 140 writes the data block in the block DB 300. The data storage unit 140 calculates a hash value of the same data block and writes it in the backup device 400. This step is processing when a position corresponding to the first division condition is not found up to a position corresponding to the maximum size of the divided block.

  FIG. 3 is a diagram illustrating a detailed processing flow of the function bdb executed by the data storage unit 140. Hereinafter, each step of FIG. 3 will be described.

(FIG. 3: Step S301)
The data storage unit 140 calculates a hash value of a data block to be stored in the block DB 300. It is desirable to use a hash function with a low probability of collision. For example, hash functions such as SHA-1 and SHA-256 can be used.

(FIG. 3: Step S302)
The data storage unit 140 searches for a data block having the same content in the block DB 300 using the hash value obtained by the data storage unit 140 in step S301 as a key.

(FIG. 3: Step S303)
If the key is found in step S302, the data storage unit 140 outputs the hash value to the backup device 400 and ends this processing flow.

(FIG. 3: Step S304)
If the key is not found in step S302, the data storage unit 140 outputs the hash value to the backup device 400, and stores the data block in the block DB 300 using the hash value as a key. The data blocks stored in the block DB 300 are preferably compressed using an appropriate algorithm.

(FIG. 3: Steps S303 to S304: Supplement)
The reason why the hash value is used as a key in these steps is that the data block may be compressed when the data block is stored in the block DB 300 in step S304. When the data block is compressed and then stored in the block DB 300, it is not known whether the data block already stored in the block DB 300 and the new data block are the same unless the compression is solved. Become heavier. Therefore, the same determination can be easily made depending on whether or not the hash values match.

<Embodiment 1: Summary>
As described above, the data dividing apparatus 100 according to the first embodiment is configured to preferentially divide data among the data positions corresponding to the second division condition as close as possible to the end of the data. ing. As a result, while performing data division for deduplication processing, the processing policy is oriented so that the data block size after division is as large as possible. As a result, data deduplication and data compression are performed. Can be balanced.

  Specifically, if a division position corresponding to the first division condition is not found, a division position as close as possible to the end corresponding to the second division condition is stored in the variable cP2, and from blockStart to cP2 in step 221 Are written to the block DB300. By this processing procedure, the last remaining cP2 value is as close to the data end as possible.

  If the data block does not satisfy either the first division condition or the second division condition, a data block having the maximum size blockMax of the divided block is stored in the block DB 300 in step 222. As a result, data blocks that are not suitable for deduplication will store data blocks of the largest possible size with priority on data compression, so even if the deduplication effect cannot be demonstrated, this will be compensated by the data compression effect, The data size can be reduced overall.

<Embodiment 2>
In the second embodiment of the present invention, a method for dividing data at a position as close to the end of data as possible as in the first embodiment will be described. In the second embodiment, data to be divided is read sequentially from the end toward the beginning, and the data is divided when the division condition is met. Since the configuration of the data dividing apparatus 100 is the same as that of the first embodiment, the following description will focus on differences.

  FIG. 4 is a program describing a process of dividing data by the data dividing apparatus 100 according to the second embodiment. For convenience of explanation, step numbers are also shown. Hereinafter, each step of FIG. 4 will be described.

(FIG. 4: Steps 400 to 412)
These steps are the same as steps 200 to 212 in FIG.

(FIG. 4: Step 413)
When blockStart + blockMax <= blockEnd is established, the division condition determination unit 120 initializes the address variable cP used when searching for the division position to a position corresponding to the maximum size of the divided block.

(FIG. 4: Step 414)
If the conditional expression in step 413 is not satisfied, the division condition determination unit 120 initializes the variable cP to the end position of the data block.

(FIG. 4: Steps 413 to 414: Supplement)
In these steps, the condition determination position cP is initialized to a position as far back as possible in the data block.

(FIG. 4: Step 415)
The division condition determination unit 120 repeats the processing from step 416 to step 421 while cP> = blockStart + blockMin is satisfied, that is, until the address cP reaches the position corresponding to the minimum block size from the end of the data block.

(FIG. 4: Steps 416 to 417)
These steps are the same as steps 215 to 216 in FIG.

(FIG. 4: Steps 418 to 419)
These steps are the same as steps 217 to 218 in FIG. However, the division condition determination unit 120 determines whether or not the address cP2 is set (value is 0) before determining the second division condition pattern2 and the third division condition pattern3. This is because, in the second embodiment, since data is read from the end of the data block toward the beginning, the value of cP2 found first is closest to the end of the data block. In other words, from the viewpoint of dividing data as close to the end of the data block as possible, it is not necessary to update the value if cP2 is already found, and it is necessary to set the value only when cP2 is empty Because.

(FIG. 4: Step 420)
This step is the same as step 219 in FIG.

(FIG. 4: Step 421)
The data dividing unit 130 advances the address cP by 1 byte (in the direction toward the beginning of the data), and returns to step 416.

(FIG. 4: Step 423)
When the variable cP2 is set when this step is reached, in the loop of steps 415 to 421, a position that matches the first determination condition is not found, and only a position that matches the second determination condition is found. It will be. The data dividing unit 130 divides data into data blocks between the address blockStart and the address cP2. The data storage unit 140 writes the data block in the block DB 300. The data storage unit 140 calculates a hash value of the same data block and writes it in the backup device 400. The data dividing unit 130 returns block 410 to blockStart = cP2.

(FIG. 4: Step 424)
If the variable cP2 is not set in step 423, a position that matches both the first determination condition and the second division condition is not found in the loop of steps 415 to 421, so the minimum size of the block after division To divide the data block. The data dividing unit 130 divides data into data blocks between the address blockStart and the address blockStart + blockMin. The data storage unit 140 writes the data block in the block DB 300. The data storage unit 140 calculates a hash value of the same data block and writes it in the backup device 400. The data dividing unit 130 increments the address blockStart by blockMin and returns to Step 410.

<Embodiment 2: Summary>
As described above, the data division device 100 according to the second embodiment sequentially reads data from the end of the data toward the top, and stores the division position that is as close as possible to the end corresponding to the second division condition in the variable cP2. deep. In step 423, the data blocks from blockStart to cP2 are written in the block DB 300. By this processing procedure, the last remaining cP2 value is as close to the data end as possible.

  If the data block does not satisfy either the first division condition or the second division condition, in step 424, the data block having the minimum size blockMin of the divided block is stored in the block DB 300. As a result, even a data block that is not suitable for deduplication stores at least a data block of the minimum size blockMin, so that the minimum data compression effect can be exhibited.

  In step 424, the data block having the maximum size blockMax may be stored in the block DB 300. This can be said to employ the same processing as step 222 in FIG.

<Embodiment 3>
In the first and second embodiments, only the first division condition and the second division condition are used. However, more division conditions can be determined. In Embodiment 3 of the present invention, data is divided at a position corresponding to any of the first division condition, the second division condition, and the third division condition. Since the configuration of the data dividing apparatus 100 is the same as that of the first embodiment, the following description will focus on differences.

  FIG. 5 is a program describing a process of dividing data by the data dividing apparatus 100 according to the third embodiment. Here, in addition to the program described with reference to FIG. 2 in the first embodiment, an example is shown in which a process for dividing data at a position that matches the third division condition is newly provided. The program described with reference to FIG. 4 in the second embodiment In addition to the above, a third determination condition can be provided. In the following, the description will focus on the parts different from the program described in FIG.

(FIG. 5: Step 512)
The division condition determination unit 120 initializes the variable cP1, the variable cP2, and the variable cP3 with 0. The variable cP3 is a variable for holding a data address corresponding to a third division condition that is a condition for dividing data.

(FIG. 5: Step 518)
When the feature value hash satisfies the third division condition, the division condition determination unit 120 sets the current data address cP to the variable cP3. In the first and second embodiments, when the third division condition is satisfied, the data address cP is stored in the variable cP2, and thus either the first division condition or the second division condition is employed. The third embodiment is different in that a third division condition is newly provided in addition to this option.

(FIG. 5: Step 522)
The data dividing unit 130 checks whether or not the variable cP3 is set (not 0). If the variable cP3 is set, the position satisfying the third division condition in step 518 has been found. The data dividing unit 130 divides data into data blocks between the address blockStart and the address cP3. The data storage unit 140 writes the data block into the block DB 300. The data storage unit 140 calculates a hash value of the same data block and writes it in the backup device 400. The data dividing unit 130 returns block 510 to blockStart = cP3. If the variable cP3 is not set, the process proceeds to step 523.

<Embodiment 3: Summary>
As described above, the data division device 100 according to the third embodiment determines the division conditions in the order of the first division condition, the second division condition, and the third division condition, and divides the data at a position corresponding to any one of them. . Thereby, finer dividing conditions than those in the first and second embodiments can be set.

<Embodiment 4>
In the fourth embodiment of the present invention, an operation example in which the unique identification number of the data block is stored in the block DB 300 and the backup device 400 in addition to the hash value of the data block will be described. As a result, even if hash values collide, data stored in the block DB 300 or the backup device 400 is prevented from being destroyed. Since other configurations are the same as those in the first to third embodiments, the following description will focus on differences.

  FIG. 6 is a diagram illustrating a detailed processing flow of the function bdb executed by the data dividing unit 130 and the data storage unit 140 in the fourth embodiment. Hereinafter, each step of FIG. 6 will be described.

(FIG. 6: Step S601)
This step is the same as step S301 in FIG.

(FIG. 6: Step S602)
The data storage unit 140 initializes a unique identification number for uniquely identifying a data block with 0.

(FIG. 6: Step S603)
The data storage unit 140 uses the hash value obtained by the data storage unit 140 in step S601 and the unique identification number of the data block as a key to search whether a data block having the same content has already been registered in the block DB 300.

(FIG. 6: Step S604)
If a key is found in step S603, the data storage unit 140 retrieves the corresponding data block from the block DB 300, and compares whether or not the content is the same as the data block to be processed. If they are the same, the hash value and the unique identification number are output to the backup device 400, and this processing flow ends. If they are not the same, the unique identification number is incremented by 1, and the process returns to step S603.

(FIG. 6: Step S605)
If the key is not found in step S603, the data storage unit 140 outputs the hash value and the unique identification number to the backup device 400, and the data storage unit 140 uses the hash value and the unique identification number as a key to store the data block in the block DB 300. To store. The data blocks stored in the block DB 300 are preferably compressed using an appropriate algorithm.

<Embodiment 4: Summary>
As described above, the data dividing apparatus 100 according to the fourth embodiment can uniquely identify a data block by a unique identification number even when hash values collide. As a result, even if hash values collide, data stored in the block DB 300 or the backup device 400 can be prevented from being destroyed.

<Embodiment 5>
In the first to fourth embodiments described above, the data dividing device 100 determines whether the data is compressed or encrypted when the data reading unit 110 reads the data from the file server 200, and compresses or encrypts the data. In the case where it is configured, the processing described with reference to FIGS. 2 to 6 may be omitted and the data stored in the block DB 300 as it is. This is because even if deduplication processing or data compression processing is further performed on the compressed or encrypted data, it is considered that a large effect cannot be expected.

  When the processing described in FIGS. 2 to 6 is omitted, the entire data is handled as one data block, and the hash value is calculated for the entire data.

  DESCRIPTION OF SYMBOLS 100: Data division | segmentation apparatus, 110: Data reading part, 120: Division condition determination part, 130: Data division part, 140: Data storage part, 200: File server, 300: Block DB, 400: Backup apparatus.

Claims (9)

A program for causing a computer to execute a process of dividing data,
In the computer,
A reading step for sequentially reading the data for each predetermined reading unit;
A determination step of determining whether the data read in the reading step satisfies a condition for dividing the data and acquiring the data position;
A dividing step of dividing the data at a position where the data satisfies the condition;
And execute
In the determination step, the computer
A first determination step of determining whether or not the data read in the reading step satisfies a first division condition that is a condition for dividing the data;
If the data read in the reading step does not meet the first division condition, it is further determined whether the data read in the reading step satisfies a second division condition that is a condition for dividing the data. 2 determination steps;
And execute
The second determination step includes
A data division program characterized in that a position closer to the end of the data is preferentially used as a result of the determination step. Causing the computer to execute a step of predefining an upper limit size of the data after being divided in the dividing step;
In the reading step, the computer is
Read the data sequentially from the beginning to the end,
In the data dividing step, the computer is
2. The division step according to claim 1, wherein when the data read in the reading step does not satisfy either the first division condition or the second division condition, the division step is executed with the upper limit size. Data partitioning program. Causing the computer to execute a step of predefining a lower limit size of the data after being divided in the dividing step;
In the reading step, the computer is
Read the data sequentially from the end to the beginning,
In the data dividing step, the computer is
2. The division step according to claim 1, wherein when the data read in the reading step does not correspond to either the first division condition or the second division condition, the division step is executed with the lower limit size. Data partitioning program. In the determination step, the computer
If the data read in the reading step does not correspond to either the first division condition or the second division condition, the data read in the reading step corresponds to a third division condition that is a condition for dividing the data. A third determination step for further determining whether or not to perform,
In the dividing step, the computer is
Dividing the data at a position corresponding to any of the first division condition, the second division condition, or the third division condition;
The third determination step includes
The data division program according to any one of claims 1 to 3, wherein a position closer to the end of the data is preferentially used as a result of the determination step. The data division program according to any one of claims 1 to 4, wherein the computer executes a storage step of storing the data divided in the division step in association with a hash value of the data in a storage device. . In the storing step, the computer
The data division program according to claim 5, wherein a number that uniquely identifies the data divided in the division step is stored in the storage device in association with the hash value. Determining whether the data is compressed or encrypted;
When the data is compressed or encrypted, the reading step, the determining step, the dividing step, and the storing step are omitted, and the data is stored in the storage device in association with the hash value Steps to perform only,
The data division program according to claim 5 or 6, wherein the computer is executed. In the storing step, the computer
The data division program according to any one of claims 5 to 7, wherein the data or the data divided in the division step is compressed and stored in a storage device. A device for dividing data,
A reading unit that sequentially reads the data for each predetermined reading unit;
A division condition determination unit that determines whether the data read by the reading unit satisfies a condition for dividing the data;
A dividing unit that divides the data at a position where the data satisfies the condition;
With
The division condition determining unit
Determining whether the data read by the reading unit satisfies a first division condition that is a condition for dividing the data;
If the data read by the reading unit does not meet the first division condition, further determine whether the data read by the reading unit meets a second division condition that is a condition for dividing the data,
The division condition determining unit
A data division apparatus characterized by preferentially using a position closer to the end of the data when determining whether or not the data satisfies a second division condition.

Images