JP4852135B2 - Data division method and apparatus - Google Patents

Description

  The present invention relates to a data division method and apparatus suitable for dividing arbitrary data into variable-length data fragments while detecting overlapping data fragments between data.

  In recent years, the infrastructure for managing data of government offices, companies, and individuals has been rapidly enlarged and complicated, and the data stored in the storage device, which is the main component of the infrastructure, has been steadily increasing. As one technique for reducing such data storage and management costs, a deduplication technique has attracted attention.

  The deduplication technique is to detect whether data having the same content as the target data is already stored in the storage device when arbitrary data (hereinafter referred to as target data) is stored in the storage device. , And if the data is already stored, the target data is replaced with, for example, a link to combine (exclude) duplicate data into one. According to this deduplication technique, the storage capacity required for data storage can be reduced.

  In order to detect at high speed whether or not the same data is stored in the storage device, an identifier of the data is often used. That is, in general, the deduplication technique is not a method of comparing the target data itself with all the data already stored in the storage device to detect the duplication of data. A method of comparing with a group of already-identified data identifiers is applied.

  Data duplication is detected in a predetermined unit. As this unit, a first method for detecting data duplication by using a lump of data (content) such as a file has been known for a long time. Recently, a second method for detecting data duplication has been proposed by using a data fragment (hereinafter referred to as a chunk) obtained by dividing data such as a file in the above unit. . In the first method, even when part of the data is different, the entire data is processed as being different. On the other hand, the second method has an advantage that only a part of the above needs to be processed.

In the deduplication technique to which the second method is applied, deduplication is generally performed by repeating the following procedure.
Procedure 1) Cut out chunks from target data.
Procedure 2) Find the identifier of the cut chunk.
Step 3) The identifier of the cut chunk is compared with the identifier of each group of chunks already stored in the storage device. If there is a chunk whose identifier is the same as that of the cut chunk, it is assumed that the chunk has the same content as the cut chunk, and the cut chunk is stored in the storage device by replacing it with a link, for example, in a form that eliminates duplication. To store.

  In the deduplication technique to which the second method is applied, the chunk cutout method executed in the procedure 1, that is, the length of the chunk cutout is important. The deduplication technique to which the second method is applied is roughly classified into the following two types according to a method for obtaining a cut point of the chunk when the chunk is cut out from the target data.

A) Fixed-length deduplication method The fixed-length deduplication method is a method in which a chunk cut-out point is determined with a certain length, and duplication is detected and eliminated for each chunk.
B) Variable-length deduplication method The variable-length deduplication method is a method in which the data division length is dynamically adjusted according to the contents of the target data to determine the cut-out point, and duplicate detection / exclusion is performed for each chunk.

Hereinafter, differences between the fixed-length deduplication method and the variable-length deduplication method will be described with reference to FIG.
In FIG. 23, the chunk cut-out point is determined by the fixed-length deduplication method and the variable-length deduplication method for the two documents, the document 111 with the document name “document # 1” and the document 112 with the document name “document # 2”. Shows how it was found. The document 112 is a document generated by editing a part of the document 111, for example, by inserting the character string “ABCD” between the character strings “name” and “specified” in the document 111.

  According to the fixed-length deduplication method, for the document 111 and the document 112, as indicated by an arrow 113 in FIG. 23, for example, chunk cut points are determined in units of a fixed length of 10 characters. On the other hand, according to the variable-length deduplication method, chunk cut points are determined for the document 111 and the document 112 according to the contents of the data, as indicated by the arrow 114 in FIG. Details of this technique will be described later.

Attention should be paid to the following points.
In the fixed-length deduplication method, the document 111 and the document 112 differ in all chunks on the back side, that is, on the tail side of the document, from the position where the character string is inserted.
On the other hand, in the variable-length deduplication method, only the chunks around the portion where the character string is inserted are different between the document 111 and the document 112, and all the chunks after that are the same. Yes.

  In this way, compared to the fixed-length deduplication method, the variable-length deduplication method performs duplication while minimizing the impact even when partial insertion / deletion / change of data occurs between certain data. Exclusion can be realized.

  There are various known methods for obtaining chunk cut-out points with variable length as described above and methods for performing deduplication using the chunk cut-out points. Here, the method as described in Patent Document 1 will be described with an example.

In the method described in Patent Document 1, a chunk cut-out point is obtained by the following procedure.
1) A data fragment (byte string) in a certain continuous fixed-length section (hereinafter referred to as a window) on data is taken out, and an identifier of the data fragment is obtained. Here, it is assumed that the window length is 2 bytes. It should be noted that this data fragment is different from the data fragment as a chunk.

  2) When a part of the obtained identifier (for example, lower 2 bits) matches a predetermined value (for example, 0x01), this is set as a chunk cut point.

FIG. 24 obtains an identifier of a character string (data fragment) in a window W having a length of 2 bytes, and when the lower 2 bits of the identifier match a predetermined value 0x01, it is determined as a chunk cut-off point An example of the operation is shown. In the example of FIG. 24, a section having a length of 2 bytes (2 characters) from the beginning of the document data “The fil...” Is initially set as a window W, and then the window W is shifted by 1 byte and The identifier of the character string is obtained, for example, by calculating a hash value of the character string. A hash function used for calculating the hash value is represented by h _α (). If the character string in the window W is “Th”, the identifier is represented by h _α (“Th”).

If the identifier h _α (“Th”) of the character string “Th” in the window W is 0x1A, the lower 2 bits of the identifier 0x1A are 0x02. The lower 2 bits of the identifier 0x1A are obtained by a logical product operation 0x1A & 0x03 of the identifier 0x1A and the mask data 0x03. The lower 2 bits 0x02 of the identifier 0x1A are not the predetermined value 0x01. For this reason, the end of the window W at this time is not used as a chunk cut-out point.

If the identifier h _α (“f”) of the character string “f” in the window W is 0x99, the lower 2 bits of the identifier 0x99 are 0x01. Since this is the same as the predetermined value 0x01, the end of the window W at this time is set as a chunk cut-out point. Thereby, the character string “The f” is cut out.

In this example, the condition for determining the cut-out point of the chunk is used, and the match between the lower 2 bits of the identifier and a predetermined value is used, but it is noted that the average chunk size is determined by this number of bits. I want. For example, in the case of 2 bits, the average chunk size is 2 ² = 4 bytes.

After the chunk is cut out as described above, as shown in FIG. 25, the identifier of the chunk itself is obtained, for example, by calculating the hash value of the chunk. The cut chunk is represented by C _A , and the hash function used for calculating the hash value is represented by h _β (). In the example of FIG. 25 in which the character string constituting the chunk C _A is “The f”, the identifier of the chunk C _A is represented by h _β (“The f”) = h _β (C _A ). In the following description, it represents the identifier of the chunk C _A in H _A.

Next, similarly to the procedure 3, the obtained chunk identifier is compared with each identifier of the group of chunks already stored in the storage device. If there is a chunk having the same identifier as the obtained chunk, processing is performed assuming that a chunk having the same content as the obtained chunk is already stored in the storage device. On the other hand, if there is no chunk having the same identifier as the obtained chunk, the obtained chunk is processed as a new chunk that is not yet stored in the storage device.
By repeating the above process every time a chunk is obtained, duplicate detection / exclusion is performed.

According to the chunk cutout method described in Patent Document 1, for example, a document 111 (data) whose document name is “document # 1” shown in FIG. 26 includes nine chunks with identifiers H _x from H _{A to} H _I. They are divided into groups of chunks C _x containing. As shown in FIG. 26, each identifier H _x of the group of chunks C _x extracted from the document 111 with the document name “document # 1” is associated with the document name “document # 1” as shown in FIG. 251 is registered. Further, a list of each of the group of identifiers H _x including the identifiers H _{A to} H _I and the chunk C _x corresponding to the identifier H _x is registered in the chunk list table 252 as shown in FIG.

  Generally, when chunks are cut out, the lengths of the chunks are often limited by a minimum length and a maximum length. In such a case, the position reaching the maximum length is forcibly determined as the chunk cut-out point.

  Japanese Patent Application Laid-Open No. H10-228561 describes that a RollingHashing technique is applied to a method for obtaining an identifier while shifting a window in order to obtain a chunk cut-out point.

The variable length deduplication techniques described in Patent Documents 1 and 2 have the following two problems.
<First issue>
In order to improve the deduplication rate, it is necessary to shorten the average length of the chunk. However, if the average length of chunks is shortened, the number of chunks increases. For this reason, the number of identifiers (more specifically, pairs of identifiers and chunks) registered in the chunk list table increases. Generally, the chunk list table is often implemented as a hash table, but as the number of identifiers increases, the size of the hash table also increases. For this reason, it is difficult to expand all chunk list tables on a memory having a high access speed. For example, the chunk list table must be expanded on a disk device having a large capacity but a low speed as compared with the memory. This is a factor that greatly deteriorates the performance.

<Second problem>
On the other hand, in order to expand all the chunk list table on the memory, it is necessary to reduce the chunk list table to a size that can be expanded on the memory. For this purpose, the number of identifiers registered in the chunk list table must be reduced. In other words, this means that the average length of the chunk is lengthened, which leads to a decrease in the deduplication rate.

  An example will be described below. FIG. 27 shows an example of the first problem, and FIG. 28 shows an example of the second problem. In the example of FIG. 27, the deduplication rate is improved as compared with FIG. 28 by shortening the average length of the chunk as compared with FIG. It can be seen that the deduplication rate is high by summing the sizes of chunk groups registered in the chunk list table 252 in association with identifiers. However, there are many identifiers compared with FIG.

  Thus, it can be seen that the deduplication rate and the number of chunks (average length of chunks) are in a trade-off relationship.

  Recently, various methods have been studied and adopted to solve the above-described problems. For example, Patent Document 3 connects two or more cut chunks (here, for convenience, a group of connected chunks is referred to as a “connected chunk”), and performs duplicate detection at least in units of connected chunks. A technique for maintaining a high deduplication rate while reducing the number is disclosed.

  The feature of the method disclosed in Patent Document 3 is that a “buffer region” made up of a group of unconnected chunks is provided between the connected chunks while dynamically switching the connected / unconnected chunks. It is in. In this method, chunk connection / disconnection is dynamically switched based on the following first and second conditions.

The first condition is “whether or not a group of chunks temporarily determined as a connection target overlaps with a connection chunk already registered in the system”.
The second condition is “whether or not a group of chunks connected before and after a group of chunks determined as a connection target is a connected chunk” and “the group of consecutive chunks has already been registered in the system. Whether or not it overlaps with the connected chunk.

US Pat. No. 5,990,810 US Pat. No. 6,810,398 US Patent Application Publication No. 2008/0133561

  However, in the method disclosed in Patent Document 3, the condition for dynamically switching between connected / disconnected is complicated. It is not preferable in terms of implementation that the conditions are complicated. In the “buffer area” provided between the connected chunks, a large number of fine chunks are generated. This leads to an increase in the number of entries in the document configuration table and chunk list table, resulting in performance degradation.

  In the technique disclosed in Patent Document 3, a plurality of groups of unconnected chunks are always connected, and duplicate detection / exclusion is performed in units of this connection. This is because when the length of chunks cut out by chunk cutout at a variable length is long, duplicate detection / removal is performed in units of larger connected chunks, and the deduplication rate is reduced. It means to become.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a data division method and apparatus capable of realizing a high deduplication rate with a simple mechanism while suppressing the number of data fragments to be divided. .

  According to one aspect of the present invention, in an apparatus including an input unit, a first data fragment determination unit, a second data fragment determination unit, a third data fragment determination unit, a duplication detection unit, and a control unit, an arbitrary There is provided a data dividing method for dividing data into a plurality of first data fragments having an arbitrary length while performing duplication detection. The data dividing method includes: an input step in which the input means inputs the arbitrary data; and the remaining data portion not yet determined as the first data fragment of the input arbitrary data. The second data fragment determining means reaches a state satisfying a first condition and a first determination step of sequentially determining a second data fragment of an arbitrary length or a predetermined length, and a predetermined first condition Up to this point, the third fragment determining means determines one second data fragment itself or a combination of a plurality of second data fragments determined in the first step as one third data fragment. The first data fragment of the bit string that matches the determined third data fragment has already determined whether or not the determined third data fragment is duplicated The duplication detection step detected by the duplication detection means, and when the duplication is detected, the first data fragment decision means determines the determined third data fragment as the first data fragment. If the overlap is not detected and the third determination step is determined as follows, the first determination step and the second determination step are re-executed, so that a new state is reached until the state satisfying the first condition is reached. One second data fragment or a plurality of new second data fragments are determined, and the new one second data fragment itself, a combination of the new plurality of second data fragments, and the duplication A combination of a part of the third data fragment in which no duplication is detected and the new second data fragment, or a part of the third data fragment in which the duplication is not detected Control for determining a combination of the new plurality of second data fragments as one new third data fragment by the third data fragment determining means is performed under a predetermined second condition. The first control step repeated by the control means until the overlap is detected in a satisfied state and the overlap is not detected even if the first control step is repeated in a state satisfying the second condition One second data fragment itself or a combination of a plurality of second data fragments satisfying the first condition among the second data fragments determined in the meantime, A fourth determining step in which the first data fragment determining means determines as a new first data fragment, and the control procedure until all the input arbitrary data is divided into the first data fragments. The stage comprises a second control step for repeating the first control step.

  According to the present invention, in the data division method and apparatus for dividing arbitrary data into a plurality of first data fragments having an arbitrary length while performing duplication detection, the length of the second data fragment is determined. Is used as an offset interval for duplication detection, and duplication detection is performed with the length of the third data fragment that is likely to be longer than the length of the second data fragment and is likely to be used as the first data fragment. By adopting a configuration that performs the above, the deduplication ratio is maintained high while reducing the number of first data fragments (that is, the number of chunks or the number of divisions) by a simpler and faster method than the conventional technology. Duplicate detection can be performed.

1 is a block diagram showing a configuration of a storage system according to an embodiment of the present invention. The block diagram which shows the hardware constitutions of the document storage apparatus shown by FIG. FIG. 2 is a block diagram mainly showing a functional configuration of the document storage device shown in FIG. 1. 6 is an exemplary flowchart illustrating a procedure of document storage processing applied in the embodiment. 6 is an exemplary flowchart illustrating a procedure of document storage processing applied in the embodiment. The figure for demonstrating the relationship between the parent chunk and the group of a child chunk applied in the embodiment. The figure for demonstrating the method of determining the cut-out point of a child chunk. The figure which shows the 1st and 2nd document, and the state of the document structure table and chunk list table before the said 1st and 2nd document are stored. The figure which shows the storing operation (the 1) for storing a 1st document with the state of a document structure table. The figure which shows the storing operation (the 2) for storing a 1st document with the state of a document structure table. The figure which shows the storing operation (the 3) for storing a 1st document with the state of a document structure table. The figure which shows the storing operation (the 4) for storing a 1st document with the state of a document structure table. The figure which shows the storing operation (the 5) for storing a 1st document with the state of a document structure table. The figure which shows the state of a document structure table and a chunk list table after storage of a 1st document with the row | line | column of the parent chunk cut out from the said 1st document and the said 1st document. The figure which shows the storing operation (the 1) for storing the 2nd document performed after storage of a 1st document with the state of a document structure table. The figure which shows the storing operation (the 2) for storing a 2nd document with the state of a document structure table. The figure which shows the storing operation (the 3) for storing a 2nd document with the state of a document structure table. The figure which shows the storing operation (the 4) for storing a 2nd document with the state of a document structure table. The figure which shows the storing operation (the 5) for storing a 2nd document with the state of a document structure table. The figure which shows the state of a document structure table and a chunk list table after storage of a 2nd document with the row | line | column of the parent chunk cut out from the said 2nd document and the said 2nd document. The state of the document configuration table and the chunk list table after the storage of the first and second documents is shown together with the first and second documents and the parent chunk columns cut out from the first and second documents. Figure. 6 is a flowchart showing a procedure of document acquisition processing applied in the embodiment. The figure for demonstrating the difference of the fixed-length deduplication method and variable-length deduplication method in a prior art. The figure which shows an example of the process of the operation | movement which sets the chunk cutout point applied with the variable-length deduplication method in a prior art. The figure for demonstrating the chunk cutout method in a prior art. The figure which shows the example which performed the chunk cut-out for a document with the chunk cut-out method in a prior art, and comprised the document structure table and the chunk list table. The figure which shows the example of the 1st subject in a prior art. The figure which shows the example of the 2nd subject in a prior art.

Embodiments of the present invention will be described below with reference to the drawings.
<System configuration>
FIG. 1 is a block diagram showing a configuration of a storage system according to an embodiment of the present invention. This storage system includes a document storage device 10 and a client device 20. The document storage device 10 and the client device 20 are connected by a network 30, for example. The document storage device 10 is a data storage device for storing a document by dividing it into chunks. The client device 20 uses the document storage device 10 as its own storage device. That is, the client device 20 stores the document in the document storage device 10 by instructing the document storage device 10 to store the document, for example, in accordance with an application program running on the client device 20, and the document storage device 10 The document is acquired from the document storage device 10 by instructing the document acquisition to the document storage device 10. Note that the document storage device 10 and the client device 20 may be directly connected, or the function as the client device 20 may be built in the document storage device 10.

  When a document storage instruction for instructing storage of a document specified by a document name is given from the client apparatus 20, the document storage apparatus 10 divides the document specified by the document name into chunks according to a procedure described later. However, after duplicate detection / exclusion is performed, the document is stored in a document storage unit 32 (see FIG. 3) described later. When the document storage apparatus 10 is given a document acquisition instruction for instructing acquisition of a document specified by a document name from the client apparatus 20, the document storage apparatus 10 retrieves the document specified by the document name from the document storage unit 32 and performs client acquisition. Output to the device 20.

  The document here refers to, for example, a file or data in the file, and the document name refers to a file name. In order to distinguish a file from data in the file, the data in the file may be referred to as document data or document data. A chunk refers to data fragmented (data fragment). In this embodiment, “first data fragment”, “second data fragment”, and “third data fragment” are defined as data fragments. In the following description, the “second data fragment” is referred to as “child chunk” and the “third data fragment” is referred to as “parent chunk”. The “first data fragment” is referred to as “registered data fragment”, “registered parent chunk”, or simply “parent chunk”.

<Hardware Configuration of Document Storage Device 10>
In the present embodiment, the document storage device 10 is realized using a computer. FIG. 2 is a block diagram showing a hardware configuration of such a document storage device 10. As shown in FIG. 2, the document storage device 10, at least one processing unit 21, a main storage device 22, an auxiliary storage device 23, a communication mechanism 24, and an input / output device 25 have known hardware configurations. The auxiliary storage device 23 is configured using, for example, a hard disk drive. The auxiliary storage device 23 includes a storage medium 231 that stores a program 230 executed by the processing unit 21. In the present embodiment, the storage medium 231 is a disk medium.

<Functional Configuration of Document Storage Device 10>
FIG. 3 is a block diagram mainly showing a functional configuration of the document storage device 10. The document storage device 10 includes a document storage unit 31, an instruction reception module 32, a variable length deduplication module 33, and a work memory 34. In the present embodiment, the instruction receiving module 32 and the variable-length deduplication module 33 in the document storage device 10 are included in the computer when the document storage device 10 is configured from a computer having the hardware configuration shown in FIG. The processing unit 21 reads the program 230 stored in the auxiliary storage device 23 into the main storage device 22 and executes it. However, at least one of the instruction receiving module 32 and the variable length deduplication module 33 may be realized as hardware.

  The document storage unit 31 stores a group of documents using the document configuration table 311 and the chunk list table 312. The document storage unit 31 is realized by using a part of the storage area of the auxiliary storage device 23 shown in FIG. The document configuration table 311 and the chunk list table 312 correspond to the document configuration table 251 and the chunk list table 252 (see FIGS. 26 to 28) applied in the conventional technology, respectively.

  The document configuration table 311 includes, for each group of documents stored in the document storage unit 31, a document name of the document and an array (that is, a list) of identifiers (hash values) of the groups of chunks that configure the document. Hold in association. The chunk list table 312 holds the chunk data fragment and the chunk identifier (hash value) in association with each chunk constituting the document stored in the document storage unit 31. That is, the document storage unit 31 stores the document by dividing it into groups of chunks constituting the document.

  In the present embodiment, the document configuration table 311 and the chunk list table 312 are stored in the work memory 34 in order to speed up access when the document storage device 10 is activated (for example, when the document storage device 10 is turned on). Loaded and used. The document configuration table 311 and the chunk list table 312 loaded in the work memory 34 are, for example, when a state in which processing is not being performed in the document storage device 10 continues for a certain period of time or when the operation of the document storage device 10 is stopped. At this time (for example, when the power of the document storage device 10 is shut off), the data is written back to the document storage unit 31. However, for the sake of convenience, the following description will be made assuming that the document configuration table 311 and the chunk list table 312 in the document storage unit 31 are used even after the document storage device 10 is started.

  The instruction receiving module 32 receives an instruction from the client device 20 and operates according to the content of the instruction. When the instruction from the client device 20 is a document storage instruction, the instruction receiving module 32 passes the document storage instruction to the variable length deduplication module 33 to cause the variable length deduplication module 33 to perform document storage processing. The command reception module 32 includes a document acquisition unit 320 that operates when the instruction from the client device 20 is a document acquisition instruction. The document acquisition unit 320 performs document acquisition processing for acquiring data of a document having a designated document name from the document storage unit 31 in accordance with the document acquisition instruction. The document data acquired by the document acquisition unit 320 is output to the client device 20 by the command reception module 32.

  The variable-length deduplication module 33 detects chunk duplication for each chunk that has been cut out in accordance with the document storage instruction passed from the instruction receiving module 32 and cuts out chunks from the specified document data in variable length. Then, the document is stored in the document storage unit 31 while performing duplicate detection / exclusion processing for eliminating it. The variable length deduplication module 33 includes a child chunk determination unit 331, a parent chunk determination unit 332, an identifier generation unit 333, a duplication detection unit 334, a parent chunk registration unit 335, and a control unit 336.

  The child chunk determination unit 331 determines a variable-length chunk as a child chunk. The parent chunk determination unit 332 determines a sequence of consecutive child chunks or a single child chunk determined by the child chunk determination unit 331 as a parent chunk.

  The identifier generation unit 333 generates an identifier of the chunk that is used in chunk extraction (here, parent chunk) extraction and duplication detection. In this embodiment, a hash value of a chunk is used as an identifier. As this hash value, for example, a value generated using a hash function such as SHA1 is used.

  Based on the identifier of the parent chunk determined by the parent chunk determination unit 332, the duplication detection unit 334 detects the duplication in which the data fragment of the identifier is registered in the chunk list table 312.

The parent chunk registration unit 335 performs parent chunk registration processing for registering the parent chunk in the document configuration table 311 and the chunk list table 312 in the document storage unit 31 based on the duplicate detection result of the duplicate detection unit 334.
The control unit 336 controls operations of the child chunk determination unit 331, the parent chunk determination unit 332, the identifier generation unit 333, and the duplication detection unit 334.

The work memory 34 provides a working storage area for chunk cutout processing and duplication detection / exclusion processing by the variable-length deduplication module 33. The work memory 34 is realized by using a part of the storage area of the main storage device 22 shown in FIG. A part of the storage area of the work memory 34 is used as a document buffer 341 for temporarily storing document data to be processed. Another part of the storage area of the work memory 34 is used as a register unit 342 for temporarily storing various variables used for processing. The register unit 342 holds an i register, a j register, and a k register for holding the child chunk numbers i, j, and k, respectively, and a start offset c _k .offset described later of the child chunk of the child chunk number k. Child chunk start offset register and a child chunk length register for holding the length (child chunk length) c _k .len of the child chunk of child chunk number k.

<Document storage processing>
Next, document storage processing in the document storage device 10 will be described with reference to FIGS. 4 and 5 are flowcharts showing the procedure of document storage processing, and FIG. 6 is a group of parent chunks and child chunks when the document to be stored in the document storage device 10 is the document 111 shown in FIG. It is a figure for demonstrating the relationship with these.

  First, it is assumed that a document storage instruction is sent from the client device 20 to the document storage device 10 via the network 30. This document storage instruction includes a document name that specifies a document to be stored in the document storage device 10.

  The document storage instruction from the client device 20 sent to the document storage device 10 is received by the command reception module 32 of the document storage device 10. When receiving the document storage instruction, the instruction reception module 32 functions as an input unit, inputs document data of the document name specified by the document storage instruction from the client device 20, and stores the document buffer in the work memory 34. 341. Then, the command receiving module 32 passes the document storage instruction from the client device 20 to the variable length deduplication module 33. Then, the variable-length deduplication module 33 executes document storage processing according to the procedure shown in the flowcharts of FIGS. That is, the variable-length deduplication module 33 repeats the following processing from the beginning to the end of the document data stored in the document buffer 341, for example.

First, the control unit 336 of the variable-length deduplication module 33, initially for excision child chunk by children chunk determining section 331 (determining the cut-out point), the child chunk number k for designating a child chunk c _k 0 At the same time, an offset (start offset) c _k .offset of the child chunk _ck is initialized to 0 indicating the start position (here, the start byte position) of the document data (step 401). That is, the control unit 336 sets 0 (k = 0) as the child chunk number k in the k register in the register unit 342, and sets the start offset of the child chunk _ck in the child chunk start offset register in the register unit 342. as c _k .offset to set 0 (c _k .offset = 0). Start offset c _k .offset child chunk c _k marks the start clipping point of the child chunk c _k, an offset (relative position) from the head position of the document data of the starting cut point. Note that at this time, the end offset indicating the end cut-out point of the child chunk _ck has not been determined.

Next, the control unit 336 sets the child chunk numbers j and i of the child chunks c _j and c _i to k (step 402). That is, the control unit 336 sets k (k = 0) as the child chunk numbers j and i in the j register and i register in the register unit 342, respectively. The child chunk c _i indicates the first child chunk in the parent chunk that is obtained first in a series of processes (registered parent chunk determination process) for determining one parent chunk to be registered in the chunk list table 312. The child chunk c _j indicates the first child chunk in the latest parent chunk obtained in the registered parent chunk determination process.

Then, the child chunk determining section 331 in the variable length deduplication module 33, by determining the end offset indicating the end cutout of the child element chunk c _k, determine the length of the child chunk c _k, the length The child chunk length c _k .len indicating the length of the child chunk c _k is set in the child chunk length register in the register unit 342 (step 403). Method for determining the offset of the end of the child chunk c _k, that is, the method for determining the cut-out point of the child chunk c _k (variable length chunks), in addition to the manner as described in Patent Documents 1 and 2, It is possible to apply the method described later with reference to FIG.

Then the control unit 336 of the variable-length deduplication module 33, start offset c _k .offset child chunk length c _k .len value obtained by adding the child chunk _{c k (c k .offset + c} k .len) is, It is determined whether it is less than the size of the document data (step 404). This determination is performed to confirm that the end of the child chunk _kk has not reached the end (end position) of the document data.

If “c _k .offset + c _k .len” is less than the size of the document data (No in step 404), the control unit 336 determines from “c _k .offset + c _k .len” as a child chunk c. the value obtained by subtracting the offset c _j .offset of _{j (c k .offset + c k} .len - c j .offset) determines whether it is predetermined connection window size W or more (step 405). In the present embodiment, it is assumed that the connection window size W is 10 bytes (W = 10).

“C _k .offset + c _k .len−c _j .offset” is the length of the row of the connected child chunks when child chunks c _{j to} c _k having child chunk numbers j to _k are connected. Represents “C _j .offset”, that is, the offset of the first child chunk c _j of the connected child chunk sequence indicates the start offset of the connected child chunk sequence. The start offset of this connected row of child chunks is the start offset of the parent chunk p determined as described later. Therefore, in order to distinguish this start offset from the start offset of the child chunk, it is referred to as a parent chunk start offset.

When step 405 is executed first after steps 401 and 402 are executed, the child chunk _ck is the first child chunk whose child chunk number k is 0, and in the example of FIG. It is the first child chunk. At this time, since the child chunk c _k matches the child chunk c _j , “c _k .offset” matches “c _j .offset”. In this case, “c _k .offset + c _k .len−c _j .offset” matches the length “c _k .len” of the child chunk c _k .

If “c _k .offset + c _k .len−c _j .offset” is not equal to or larger than the concatenated window size W (No in step 405), the control unit 336 causes the child chunk start offset register in the register unit 342 to be stored. Indicates the start offset c _{k + 1} .offset of the next child chunk c _{k + 1} , so that the current child chunk is offset to the offset c _k .offset of the child chunk c _k currently held in the child chunk start offset register. The value obtained by adding the lengths of the child chunks _ck set in the length register is set in the child chunk start offset register as the start offset c _{k + 1} .offset of the next child chunk _{ck + 1} (step 406). In step 406, the control unit 336 increments the child chunk number k held in the k register by 1. As a result, the child chunk number k after one increment designates the child chunk _{ck + 1} that follows the child chunk _ck before step 406 is executed as a new child chunk _ck . At this time, the child chunk start offset register indicates the start offset of the new child chunk _ck .

Step 406 is executed by the control unit 336, the child chunk determining section 331 in search of the length of the new child chunk c _k by executing the step 403 again, the length of the child chunk length c _k .len Set as. In the example of FIG. 6, when the child chunk c _k before the execution of step 406 is the first child chunk “The f”, the data fragment “file” following “The f” is a new child chunk c _k. As determined. Then, the above-described processing is performed again based on the child chunk length c _k .len of the new child chunk _ck .

Then, in the example of FIG. 6, the data fragment “na” subsequent to “file” is determined as a new child chunk _ck . At this time, “c _k .offset + c _k .len−c _j .offset”, that is, child chunks c _j (“The f”) to c _k (“na”) whose child chunk numbers are j to k are connected. In this case, let L1 be the length of the row of the connected child chunks. This length L1 is not less than the connection window size W as shown in FIG.

As described above, when “c _k .offset + c _k .len−c _j .offset” becomes equal to or larger than the concatenated window size W (Yes in step 405), the parent chunk determination unit 332 determines that the child chunk numbers are j to k. Child chunks c _{j to} c _k are connected to one and defined as a parent chunk p (step 407). In the example of FIG. 6, three child chunks are concatenated from the top of the document 111 (document whose document name is “document # 1”), and determined as a parent chunk p1 (p = p1). In order to distinguish this parent chunk p1 from the subsequent parent chunks p2 and p3 determined by resetting the parent chunk start offset as will be described later, it may be called an initial parent chunk. As a determination condition in step 404, in addition to the above-described conditions, (1) “c _k .offset + c _k .len−c _j .offset” exceeds the connection window size W, (2) “c _k .offset + c _k .len -c _j .offset "matches the linked window size W, (3)" c _k .offset + c _k .len -c _j .offset "is closest to the linked window size W (4) “c _k .offset + c _k .len−c _j .offset” may be any of the maximum values that do not exceed the concatenated window size W.

In step 407, the parent chunk determination unit 332 obtains data c _j... K.data _obtained by concatenating the child chunks c _{j to} c _k as data (data fragment) p _data of the parent chunk p. In step 407, the parent chunk determination unit 332 causes the identifier generation unit 333 to generate the hash value p _hash of the data p _data of the parent chunk p, which is used as the identifier of the data p _data of the parent chunk p. If the hash function used to obtain the hash value is expressed as hash (), the hash value p _hash is a calculation process of hash (p _data ), that is, hash (c _{j ... k} .data) It is calculated | required by the calculation process of. If the child chunk number j matches k, that is, if only a single child chunk is equal to or larger than the concatenated window size W, the single child chunk itself is determined as the parent chunk p.

Next, the duplication detection unit 334 in the variable length deduplication module 33 determines whether the data fragment of the parent chunk p (p = p1) obtained in step 407 is already registered in the chunk list table 312 (step). 408). This step 408 is executed in order to detect a duplicate in which the parent chunk having the same content as the parent chunk p is already stored in the document storage unit 31. The determination in step 408 can be realized by checking whether an identifier (hash value) matching the identifier (hash value) p _hash of the data p _data of the parent chunk p is already registered in the chunk list table 312. However, the bit string of the data (data fragment) p _data of the parent chunk p and the bit string of the chunk data registered in the chunk list table 312 paired with the identifier that matches the identifier of the data p _data of the parent chunk p Do not always match depending on the hash function used to calculate the identifier (hash value). Therefore, duplicate detection unit 334 in step 408 determines whether an identifier (hash values) of the data p _data in the parent chunk p p _hash a pair of the data p _data is registered in the chunk list table 312. More specifically, the duplication detection unit 334 matches not only the identifier matching the identifier p _hash of the parent chunk p but also the bit string of the data p _data of the parent chunk p, as well as being registered in the chunk list table 312. It is determined whether the bit string of the data of the chunk to be registered is registered in the chunk list table 312 in association with the matching identifier. In this way, more accurate duplication detection can be performed and so-called hash collision can be prevented.

If the data (data fragment) p _{data of} the parent chunk p obtained in step 407 is not registered in the chunk list table 312 (No in step 408), the control unit 336 increments the child chunk number j by 1. (Step 409). The child chunk number j after this increment is a child chunk subsequent to the first child chunk in the row of child chunks constituting the parent chunk p obtained in step 407, and is to be determined next. Indicates the first child chunk (new child chunk) c _j .

The start offset c _j .offsetf of this new child chunk c _j is shifted toward the end of the document data by the length of the first child chunk in the row of child chunks constituting the parent chunk p obtained in step 407. , Indicates the new parent chunk start offset. That is, in step 409, the parent chunk start offset is reset. In the example of FIG. 6, the new (reset) parent chunk start offset matches the start offset of the second child chunk from the top of the document 111 (child chunk whose data fragment is “ile”). Step 409 is equivalent to removing the data fragment of the first child chunk from the parent chunk p determined in step 407, as will be described later.

Next, the control unit 336 determines whether “c _j .offset + c _j .len−c _i .offset” is equal to or larger than the connection window size W (step 410). At this time, c _j .offset indicates the parent chunk start offset reset as described above. On the other hand, “c _i .offset” indicates the start offset of the initial parent chunk p. Therefore, “c _j .offset + c _j .len−c _i .offset” is a sequence of concatenated child chunks when child chunks c _{i to} c _j having child chunk numbers i to _j are concatenated. Represents the length of. That is, “c _j .offset + c _j .len−c _i .offset” represents a “deviation” of the parent chunk start offset reset in step 409 from the start offset of the initial parent chunk p.

If “c _j .offset + c _j .len−c _i .offset” is not equal to or larger than the concatenated window size W (No in step 410), the control unit 336 proceeds to step 406 and the child chunk in the register unit 342 is obtained. The content of the register is updated to “c _k .offset + c _k .len” so that the start offset register indicates the start offset c _{k + 1} .offset of the next child chunk c _{k + 1} , and the child chunk number Increment k by 1. By this increment, the next child chunk c _{k + 1} is treated as a new child chunk c _k . As in this example, when Step 406 is executed because No is determined in Step 410, the new child chunk _ck is a child chunk that follows the sequence of chunks that make up the previously determined parent chunk p. It is. In the example of FIG. 6, the data fragment “me spe” following “na” is determined as the data fragment of the new child chunk _ck .

Step 406 is executed by the control unit 336, the child chunk determining section 331 in search of the length of the new child chunk c _k by executing the step 403 again, the length of the child chunk length c _k .len Set as. Thus, in the present embodiment, child chunks are determined until the column length of the child chunks c _{j to} c _k starting from the parent chunk start offset reset in step 409 is equal to or greater than the concatenated window size W. Here, as is clear from the processing in step 406, there is no need to re _- determine the child chunks c _{j to} c _k−1 that appear after the reset parent chunk start offset and are already determined in the previous processing. .

In the example of FIG. 6, the data fragments of the child chunks c _{j to} c _k at this time are “file” to “me spe”. When child chunks c _j (“file”) to c _k (“me spe”) are concatenated, the length “c _k .offset + c _k .len−c _j .offset” of the concatenated child chunks "Is L2. This length L2 is equal to or larger than the connection window size W as shown in FIG.

As in this example, if “c _k .offset + c _k .len−c _j .offset” becomes equal to or larger than the concatenated window size W (Yes in step 405), the parent chunk determination unit 332 determines as described above. , Child chunks c _{j to} c _k whose child chunk numbers are j to k are concatenated into one and defined as parent chunk p (step 407). In the example of FIG. 6, the second to fourth child chunks from the top of the document 111 are concatenated and determined as the parent chunk p2 (p = p2). The parent chunk p2 removes the first child chunk (that is, the first child chunk of the document 111) from the previously determined parent chunk p1 by the above-described step 409, and adds the new child chunk p1 to the parent chunk p1 from which the first child chunk has been removed. Is equivalent to a new parent chunk constructed by incorporating the fourth child chunk from the top of the document 111 in steps 406, 403, and 407.

Next, the duplication detection unit 334 determines whether the data (data fragment) p _{data of} the parent chunk p (p = p2) obtained in step 407 is already registered in the chunk list table 312 (step 408). If the data p _data of the parent chunk p is not registered in the chunk list table 312 (No in Step 408), the control unit 336 proceeds to Step 409 as described above and increments the child chunk number j by 1. As a result, the parent chunk start offset is reset. In the example of FIG. 6, the reset parent chunk start offset matches the start offset of the third child chunk from the top of the document 111 (child chunk whose data fragment is “na”).

In the example of FIG. 6, “c _j .offset + c _j .len−c _i .offset” at this time, that is, the “shift” of the parent chunk start offset is not equal to or larger than the linked window size W (No in step 410). In this case, the process including steps 406 and 403 is repeated. Accordingly, in the example of FIG. 6, the third to fifth child chunks from the top of the document 111 are connected and determined as a parent chunk p3 (p = p3) having a length L3 (L3 ≧ W) (step 407).

If the data p _{data of} the determined parent chunk p (p = p3) is not registered in the chunk list table 312 (No in step 408), the control unit 336 proceeds to step 409 as described above, and the child data Chunk number j is incremented by one. As a result, the parent chunk start offset is reset. In the example of FIG. 6, the reset parent chunk start offset matches the start offset of the fourth child chunk from the top of the document 111 (child chunk whose data fragment is “me spe”).

In the example of FIG. 6, “c _j .offset + c _j .len−c _i .offset” at this time, that is, the “shift” of the parent chunk start offset is equal to or larger than the linked window size W (Yes in step 410). . In this case, the control unit 336 resets the child chunk number k to a value obtained by subtracting 1 from the child chunk number j, that is, the child chunk number j before being incremented by 1 in Step 409 (Step 411). Thereby, the reset child chunk number k indicates the terminal-side child chunk ck in the initial parent chunk. When step 410 is executed, the process proceeds to step 510.

On the other hand, if the parent chunk p determined in step 407 is a registered parent chunk that has already been registered in the chunk list table 312 (Yes in step 408), the parent chunk registration unit 335 causes the document configuration table 311 and the chunk list to be registered. The parent chunk p is registered only in the document configuration table 311 in the table 312 (step 412). More specifically, the parent chunk registration unit 335 registers the identifier (hash value) p _hash of the parent chunk p in the document configuration table 311 in association with the document name of the document including the parent chunk p. If the document configuration table 311 and the chunk list table 312 are empty, that is, if one parent chunk has not been registered in the document configuration table 311 and the chunk list table 312, the determination is first made in step 407. The parent chunk p may be registered in the document configuration table 311 and the chunk list table 312.

Here, when the document name of the document including the parent chunk p is already registered in the document configuration table 311, the parent chunk registration unit 335 associates the identifier with the identifier already registered in the document configuration table 311. The identifier p _hash of the parent chunk p is added to the end of the array. Thereby, the arrangement order of the identifiers registered in the document configuration table 311 in association with the document name of the document including the parent chunk p is the order in which the corresponding parent chunk is extracted from the document, that is, the document of the corresponding parent chunk. Matches the order of

  When the parent chunk p is registered in the document configuration table 311 (step 412), the control unit 336 of the variable-length deduplication module 33 determines whether the child chunk number i is equal to the child chunk number j (step 413). That is, the control unit 336 determines whether or not the parent chunk p registered in the document configuration table 311 has been determined without resetting the parent chunk start offset (step 409). If the child chunk number i is not equal to the child chunk number j (No in step 413), the process proceeds from step 413 to step 501.

As described above, as a result of repeating the processes including steps 406 and 403, the parent chunk p is determined (step 407), and the data p _{data of the} determined parent chunk p is already registered in the chunk list table 312. Is detected (Yes in Step 408), and if i = j is not satisfied (No in Step 413), Step 501 is executed. If it is detected that the “shift” of the parent chunk start offset is equal to or larger than the connection window size W (Yes in Step 410), Step 501 is executed through Step 411.

In step 501, the parent chunk determination unit 332 concatenates the child chunks c _{i to} c _j−1 having child chunk numbers i to j−1 and determines it as the parent chunk p. In step 501, the parent chunk determination unit 332 obtains data c _i... J−1.data obtained by concatenating the child chunks c _{i to} c _j−1 as data p _data of the parent chunk p, and the parent chunk p The identifier generation unit 333 generates a hash value (identifier) p _hash (= hash (c _{i... j−1} .data)) of the data p _data of p.

Next, the parent chunk registration unit 335 registers the parent chunk p in the chunk list table 312 (step 502). More specifically, the parent chunk registration unit 335 registers the identifier (hash value) p _hash of the parent chunk p and the data (data fragment) p _data of the parent chunk p in the chunk list table 312. In addition, the parent chunk registration unit 335 registers the parent chunk p in the document configuration table 311 (step 503). That is, the parent chunk registration unit 335 registers the identifier p _hash of the parent chunk p in the document configuration table 311 in association with the document name of the document including the parent chunk p.

Now, when the process proceeds from step 410 to step 501 through step 411, the parent chunk p determined in step 501, that is, the parent chunk p composed of the columns of child chunks c _{i to} c _j−1 is the initial parent chunk p. It corresponds to p (in the example of FIG. 6, parent chunk p1).

On the other hand, when the process proceeds from step 413 to step 501, the columns of the child chunks c _{i to} c _j−1 constituting the parent chunk p determined in step 501 are the parent chunks registered in the chunk list table 312 most recently. And a column of child chunks existing between the parent chunk used in the determination in step 408. If i is equal to j-1, the child chunks c _{i to} c _j-1 mean a single child chunk.

When step 503 is executed by the parent chunk registration unit 335, one registration parent chunk determination process ends. Then, the control unit 336 determines whether “c _k .offset + c _k .len” is less than the size of the document data, similarly to the previous step 404 (step 504).

If “c _k .offset + c _k .len” is less than the size of the document data (Yes in step 504), the control unit 336 determines that the processing has not been completed up to the end of the document data. In this case, the control unit 336 causes the child chunk start offset register in the register unit 342 to indicate the start offset c _{k + 1} .offset of the next child chunk c _{k + 1 for} the next registered parent chunk determination process. Then, the contents of the register are updated to “c _k .offset + c _k .len” and the child chunk number k is incremented by 1 (step 505). After executing Step 505, the control unit 336 returns to Step 402 and sets both the child chunk numbers j and i to k. As a result, the parent chunk start offset is reset to the position (that is, the end position) of the end offset of the initial parent chunk in the previous registered parent chunk determination process. Thereafter, the registration parent chunk determination process including the step 403 and the same procedure as described above is repeated until the end of the document data.

As a result of processing up to the end of the document data, it is assumed that “c _k .offset + c _k .len” is not less than the size of the document data (No in step 404). In this case, similar to step 407, the parent chunk determination unit 332 concatenates the child chunks c _{j to} c _k whose child chunk numbers are j to k and determines it as the parent chunk p (step 414). . In this step 414, the variable-length deduplication module 33 obtains the data c _j... K.data _obtained by concatenating the child chunks c _{j to} c _k as the data p _data of the parent chunk p, and the data p of the parent chunk p. _The identifier generation unit 333 generates a hash value (identifier) p _hash (= hash (c _j.... data)) of data.

Then, the duplication detection unit 334 determines whether the data fragment of the parent chunk p obtained in step 414 is already registered in the chunk list table 312 (step 415). If the data fragment of the parent chunk p obtained in step 414 is not registered in the chunk list table 312 (No in step 415), the parent chunk registration unit 335 stores the identifier of the parent chunk p in the chunk list table 312. (Hash value) p _hash and the data fragment p _data of the parent chunk p are registered (step 416). Next, the parent chunk registration unit 335 proceeds to step 412 and registers the identifier p _hash of the parent chunk p in the document configuration table 311 in association with the document name of the document including the parent chunk p. On the other hand, if the data fragment of the parent chunk p is already registered in the chunk list table 312 (Yes in Step 415), the parent chunk registration unit 335 skips Step 416 and executes Step 412.

When step 412 is executed by the parent chunk registration unit 335, the control unit 336 determines whether the child chunk number i is equal to the child chunk number j (step 413). If the child chunk number i is equal to the child chunk number j (Yes in Step 413), the control unit 336 proceeds to Step 504. In step 504, the control unit 336 determines whether “c _k .offset + c _k .len” is less than the size of the document data, as in step 404. In this example in which the determination in step 404 is No, the determination in step 504 is also No. In this case, the variable-length deduplication module 33 finishes the process to the end of the document data, and ends the document storage process.

<Extracting child chunks>
Next, a method for determining a cut-out point of a child chunk (that is, a variable-length data fragment) will be described. As described above, as this method, it is possible to apply the methods described in Patent Documents 1 and 2. However, in addition to the methods described in Patent Documents 1 and 2, a novel method as described below can be applied. The feature of this new method is that when the lower m bits of an identifier (hash value) of a certain data fragment matches a predetermined value A, the end position of the data fragment is used as a cut-out point of a child chunk. is there.

Hereinafter, this new method will be described with reference to FIG. FIG. 7 shows that when the lower 2 bits (m = 2) of an identifier (hash value) of a certain data fragment matches a predetermined value 2 (A = 2), the end position of the data fragment is a child chunk. An example of the cut-out point is shown. A hash function used for calculating the identifier (hash value) of the data fragment is represented as h _β ().

In the example of FIG. 7, the identifier h _β (“Th”) of the data fragment “Th” in the document data “The fil...” Is 0x5A. The lower 2 bits of this identifier 0x5A are 0x02. The lower 2 bits of the identifier 0x5A are obtained by a logical product operation 0x5A & 0x03 of the identifier 0x5A and the mask data 0x03. The lower 2 bits 0x02 of the identifier 0x5A do not match the specified value 0x01. For this reason, the end position of the data fragment “Th” is not the cut-out point of the child chunk.

Therefore, the child chunk determination unit 331 extends the size (section) of the data fragment used for determining the cutout point by 1 byte toward the end of the document data. It is assumed that the identifier h _β (“The”) of the data fragment “The” after this size expansion is 0xF2. The lower 2 bits of this identifier 0xF2 are 0x02 and do not match the specified value 0x01. Therefore, the child chunk determination unit 331 further extends the size of the data fragment by 1 byte.

It is assumed that the identifier h _β (“The”) of the data fragment “The” after size expansion is 0x7C. The lower 2 bits of this identifier 0x7C are 0x00 and do not match the specified value 0x01. Therefore, the child chunk determination unit 331 further extends the size of the data fragment by 1 byte.

It is assumed that the identifier h _β (“The f”) of the data fragment “The f” after size expansion is 0x99. The lower 2 bits of this identifier 0x99 are 0x01, which matches the specified value 0x01. Therefore, the child chunk determination unit 331 determines the end position of the data fragment “The f” as a cutout point (end offset), and cuts out the data fragment “The f” as a child chunk.

  The main procedure of the document storage process described above is summarized below.

  The variable length deduplication module 33 repeatedly performs the following processing from the beginning to the end of the document data.

  a) The beginning of the document is set as a parent chunk start offset (steps 401 and 402).

  b) A column of child chunks (or a single child chunk) is determined from the parent chunk start offset to a place where the length after concatenation becomes equal to or larger than the concatenation window size W (steps 403 to 406).

  c) When the sequence of child chunks is determined in the process b, these are concatenated into one and defined as a parent chunk (initial parent chunk) (step 407). Even when a single child chunk is determined in the process b, it is determined as a parent chunk (initial parent chunk) (step 407). The length of the parent chunk (initial parent chunk) is equal to or greater than the linked window size W.

  d) The identifier (hash value) of the parent chunk is obtained (step 407).

  e) It is determined whether the identifier and data fragment of the parent chunk are already registered in the chunk list table 312 (step 408).

  e. 1) If registered, the identifier of the parent chunk is associated with the document name and registered in the document configuration table 311 (step 412).

  e. 2) If not registered, the following processing is performed.

  e. 2-1) Shifted to the back by the size of at least one child chunk at the beginning of the row of child chunks that make up the parent chunk, for example, the size of the first child chunk (that is, the child chunk closest to the parent chunk start offset side) The position is reset as the parent chunk start offset (step 409).

  e. 2-2) A sequence of child chunks (or a single child chunk) is determined from the parent chunk start offset to a position where the length after concatenation becomes equal to or larger than the concatenation window size W (steps 403 to 406). At this time, it is not necessary to re-determine the child chunks already determined in the previous process (step 406).

  e. 2-3) Process e. When the row of child chunks is determined in 2-2, these are concatenated into one and defined as a parent chunk (step 407). Process e. When a single child chunk is determined in 2-2, it is determined as a parent chunk (step 407).

  e. 2-4) An identifier (hash value) of the parent chunk is obtained (step 407).

  e. 2-5) It is determined whether the identifier and data fragment of the parent chunk are already registered in the chunk list table 312 (step 408).

  e. 2-6) Whether the parent chunk identifier and data fragment have already been registered in the chunk list table 312 (Yes in step 408), or the parent chunk start is performed in the above processing (e.2-1 to e.2-5) The process is repeated until the offset “deviation” exceeds the size of the initial parent chunk determined in the process c (Yes in step 410) (up to the parent chunk p3 in the example of FIG. 6).

  e. 2-7) If the data fragment is not yet registered, the identifier and data fragment of the initial parent chunk (parent chunk p1 in the example of FIG. 6) determined in process c are registered in the chunk list table 312 and The identifier is associated with the document name and registered in the document configuration table 311 (steps 502 and 503). Then, the start offset of the child chunk that becomes the next parent chunk start offset is reset to the position of the end offset of the initial parent chunk (that is, the end offset of the child chunk on the end side of the initial parent chunk) determined in the process c. (Steps 411 and 505), the process returns to b.

  When the identifier and the data fragment of the parent chunk are already registered in the chunk list table 312 (Yes in Step 408), the identifier of the parent chunk is registered in the document configuration table 311 in association with the document name (Step 412). When there is an unregistered data fragment in the chunk list table 312 between the parent chunk registered in the document configuration table 311 and the parent chunk registered in the previous chunk list table 312 (No in step 413). The data fragment and the identifier of the data fragment are registered in the chunk list table 312 as the parent chunk, and the identifier of the data fragment is registered in the document configuration table 311 in association with the document name (step 501). ~ 503). At this time, it is necessary to rewrite the chunks (data fragments) in the document configuration table 311 so that the order of the chunks constituting the document is correct. Then, the start offset of the child chunk that becomes the next parent chunk start offset is reset to the position of the end offset of the parent chunk registered this time (that is, the end offset of the child chunk at the end of the initial parent chunk) (step) 411, 505), the process returns to process b. When the identifier of the parent chunk is registered in the document configuration table 311 in association with the document name, information indicating the position / length of the parent chunk on the corresponding document data is added to the identifier of the parent chunk. Then, rewriting as described above is not always necessary.

  By repeating the above processing until the end of the data, it is possible to store data while performing deduplication with variable length.

<Specific example of document storage processing>
Next, a specific example of document storage processing in the document storage device 10 will be described with reference to FIGS. 8 to 21 in addition to the flowcharts of FIGS.
Here, an example will be described in which two documents, a document 111 with a document name “document # 1” and a document 112 with a document name “document # 2”, are sequentially stored in the document storage unit 31 while eliminating duplication. In the following description, storage processes for storing the documents 111 and 112 are referred to as storage processes SX and SY, respectively. In this example, the linked window size W is set to 10 (10 bytes).

(1) Before the start of the storage processes SX and SY Before the documents 111 and 112 are stored in the document storage unit 31, the document configuration table 311 and the chunk list table 312 in the document storage unit 31 are in an empty state. . FIG. 8 shows the documents 111 and 112 (first and second documents) and the state of the document configuration table 311 and chunk list table 312 before the documents 111 and 112 are stored.

(2) Storage process SX
A storage process (storage operation) SX for storing the document 111 will be described with reference to FIGS. 9 to 13 show the storage operation of the document 111 together with the state of the document structure table 311. FIG. 14 shows the state of the document structure table 311 and the chunk list table 312 after the document 111 is stored. It is shown together with a row of parent chunks cut out from 111.

A) Storage process SX 1
First, a storage process SX 1 for storing the document 111 (hereinafter referred to as a storage process SX1) will be described with reference to FIG. In FIG. 9, the document configuration table 311 is omitted.

A1)
A1-1) The variable-length deduplication module 33 sequentially determines child chunks from the beginning of the document 111 to a place where the connection window size W is greater than or equal to W (W = 10 in this example) (steps 403 to 406). Each child chunk is determined by the variable-length chunk cutout method described above. In the example of FIG. 9, the variable-length deduplication module 33 sets cut points at the fifth, eighth, and eleventh characters from the beginning of the document 111, and sets child chunks c ₀ (“The f”), c ₁ (“ile "), C ₂ (" na ").

A1-2) The variable-length deduplication module 33 sequentially determines child chunks c ₀ (“The f”), c ₁ (“ile”), and c ₂ (“na”) until the connection window size W or more is reached. Now, these child chunks are concatenated to define a parent chunk 901, and an identifier (hash value) of the parent chunk 901 is generated (step 407). The identifiers (hash values) of the child chunks c ₀ (“The f”), c ₁ (“ile”), and c ₂ (“na”) to be concatenated are expressed as c ₀ .hash = _HA and c ₁ .hash = H Let _B , c ₂ .hash = H _C. In this example, the hash values generated from the identifiers (hash values) H _A , H _B , and H _{C of the} child chunks c ₀ , c ₁ , and c ₂ constituting the parent chunk 901 are used as the hash values of the data of the parent chunk 901. The value H _ABC is used, and this is used as the identifier of the parent chunk 901.

A1-3) The variable-length deduplication module 33 determines whether the data fragment corresponding to the identifier H _ABC is registered in the chunk list table 312 based on the identifier H _ABC of the parent chunk 901 (step 408). In this example, the data fragment corresponding to the identifier H _ABC is not registered. For this reason, the determination result of step 408 regarding the parent chunk 901 becomes unregistered (No) as indicated by an arrow 911 in FIG. 9, and the process proceeds to the next process A2.

A2)
A2-1) length deduplication module 33, the length of the child chunk c ₀ of the column head of the configuration child chunk c _0, c _1, c ₂ parent chunks 901 determined in the above-described process A1, a document From the position shifted to the end of 111 (reset parent chunk start position) (step 409), the child chunks are sequentially determined from the position where the connection window size becomes W (W = 10) or more (steps 403 to 403). 406). At this time, it is not necessary to repeat the process for the portion where the child chunks c ₁ and c ₂ are determined in the process A1. In the example of FIG. 9, the variable-length deduplication module 33 defines a cut point at the 17-th character from the beginning of the document 111, and define those new children chunk c _3.

A2-2) length deduplication module 33, where defining a coupling window size children chunk c ₁ W until it becomes more, c _2, c _3, connected to their child chunks c _1, c _2, c ₃ Thus, the parent chunk 902 is determined, and an identifier (hash value) of the parent chunk 902 is generated (step 407). In this example, the variable-length deduplication module 33 determines the identifier (hash value) c ₁ .hash = of the child chunks c ₁ (“ile”), c ₂ (“na”), and c ₃ (“me spe”) to be connected _{H B, c 2 .hash = H} C, the hash value H _BCD generated from c ₃ .hash = H _D, an identifier of the parent chunk 902 (hash value).

A2-3) Based on the identifier H _BCD of the parent chunk 902, the variable length deduplication module 33 determines whether a data fragment corresponding to the identifier H _BCD is registered in the chunk list table 312 (step 408). In this example, the data fragment corresponding to the identifier H _BCD is not registered. For this reason, the determination result of step 408 regarding the parent chunk 902 becomes unregistered (No) as indicated by an arrow 912 in FIG. 9, and the process proceeds to the next process A3.

A3)
A3-1) The variable-length deduplication module 33 creates a document by the length of the _first child chunk c ₁ of the columns of the child chunks c ₁ , c ₂ , c ₃ constituting the parent chunk 902 defined in the above-described processing A2. From the position shifted to the end of 111 (step 409), child chunks are sequentially determined until the connection window size W (W = 10) or more is reached (steps 403 to 406). At this time, it is not necessary to repeat the process for the part where the child chunks c ₂ and c ₃ are determined in the process A 2. In the example of FIG. 9, the variable-length deduplication module 33 defines a point cut at the 20 th character from the beginning of the document 111, and define those new children chunk c _4.

A3-2) The variable-length deduplication module 33 determines the child chunks c ₂ , c ₃ , and c ₄ until the connection window size W is equal to or larger than the connection window size W, and connects these child chunks c ₂ , c ₃ , and c ₄ . Then, the parent chunk 903 is determined, and an identifier (hash value) of the parent chunk 903 is generated (step 407). In this example, the variable-length deduplication module 33 identifies the identifiers (hash values) c ₂ .hash = of the child chunks c ₂ (“na”), c ₃ (“me spe”), and c ₄ (“cif”) to be connected. The hash value H _CDE generated from H _C , c ₃ .hash = H _D and c ₄ .hash = H _{E is used as} the identifier (hash value) of the parent chunk 903.

A3-3) The variable-length deduplication module 33 determines whether a data fragment corresponding to the identifier H _CDE is registered in the chunk list table 312 based on the identifier H _CDE of the parent chunk 903 (step 408). In this example, the data fragment corresponding to the identifier H _CDE is not registered. For this reason, the determination result of step 408 regarding the parent chunk 903 becomes unregistered (No) as indicated by an arrow 913 in FIG. 9, and the process proceeds to the next process A4.

A4)
In the example of FIG. 9, the length of the first child chunk c _{2 in} the column of the child chunks c ₂ , c ₃ , c ₄ constituting the parent chunk 903 defined in the above process A 3 is shifted toward the end of the document 111. The position (that is, the reset parent chunk start position) exceeds the position of the end of the parent chunk (that is, the initial parent chunk) 901 determined in the process A1. Therefore, the “deviation” of the parent chunk start offset reset in step 409 from the start offset of the parent chunk 901 is equal to or larger than the linked window size W (Yes in step 410). In this case, the variable-length deduplication module 33 uses the chunk list table 312 as shown by an arrow 904 in FIG. 9 for the identifier H _ABC and the data (data fragment) “The file na” of the parent chunk 901 determined in the process A1. (Step 502). Although not shown in FIG. 9, the variable-length deduplication module 33 registers the document name “document # 1” of the document 111 and the identifier H _ABC of the parent chunk 901 in the document configuration table 311 (step 503). In this embodiment, the identifier H _ABC of the parent chunk 901 registered at step 502 is obtained again at step 501.

(B) Storage process SX 2
A storage process SX 2 for storing the document 111 (hereinafter referred to as a storage process SX2) executed subsequent to the above-described storage process SX1 will be described with reference to FIG. In FIG. 10, the document configuration table 311 is omitted.

B1)
B1-1) The variable-length deduplication module 33 removes child chunks from the end position of the parent chunk 901 determined in the storage process SX1 (step 505) until it reaches the connected window size W (W = 10) or more. These are determined sequentially (steps 403 to 406). At this time, it is not necessary to repeat the process for the part in which the child chunks c ₃ and c ₄ after the end position of the parent chunk 901 are determined in the storage process SX1. In the example of FIG. 10, it is assumed that the variable-length deduplication module 33 determines a cut-out point at the 24th character from the top of the document 111 and sets a new child chunk c ₅ (“cif”).

The subsequent processing is the storage processing SX1, is similar to the processing A2-2～A2-3, parent chunk 1001 is determined by linking the child chunk _{c 3, c 4, c 5} . In the example of FIG. 10, the data fragment corresponding to the identifier H _DEF of the parent chunk 1001 is not registered in the chunk list table 312. For this reason, the determination result of step 408 regarding the parent chunk 1001 becomes unregistered (No) as indicated by an arrow 1011 in FIG. 10, and the process proceeds to the next process B2.

B2)
The process B2 is the same as the process A2 in the storage process SX1. In the process B2, a new child chunk c ₆ (“by path is ope”) is determined. Then, the parent chunk 1002 is determined by connecting the child chunks c ₄ , c ₅ , and c ₆ . In the example of FIG. 10, the data fragment corresponding to the identifier _HEFG of the parent chunk 1002 is not registered in the chunk list table 312. For this reason, the determination result of step 408 regarding the parent chunk 1001 becomes unregistered (No) as indicated by an arrow 1012 in FIG. 10, and the process proceeds to the next process B3.

B3)
The process B3 is the same as the process A3 in the storage process SX1. The variable-length deduplication module 33 sets the end of the document 111 by the length of the first child chunk c _{4 in} the column of the child chunks c ₄ , c ₅ , c ₆ constituting the parent chunk 1002 defined in the above-described process A2. From the position shifted to (step 409), child chunks are sequentially determined until the connection window size W (W = 10) or more is reached (steps 403 to 406). At this time, it is not necessary to repeat the process for the portion where the child chunks c ₅ and c ₆ are determined in the process B2. In the example of FIG. 10, there is no newly defined child chunk from the position shifted by the length of the child chunk c _{4 to} the place where the connection window size W or more is reached. Therefore, the variable-length deduplication module 33 determines the parent chunk 1003 by concatenating the remaining child chunks c ₅ and c ₆ obtained by removing the first child chunk c ₄ from the parent chunk 1002 determined in the process A2. Data fragment corresponding to the identifier H _FG parent chunks 1003 in the example of FIG. 10 is not registered in the chunk list table 312. For this reason, the determination result of step 408 regarding the parent chunk 1003 becomes unregistered (No) as indicated by an arrow 1013 in FIG. 10, and the process proceeds to the next process B4.

B4)
The process B4 is the same as the process A4 in the storage process SX1. That is, in the example of FIG. 10, a position shifted toward the end of the document 111 by the length of the first child chunk c _{5 in} the column of the child chunks c ₅ and c ₆ constituting the parent chunk 1003 defined in the process B3 ( That is, the reset parent chunk start position) exceeds the end position of the parent chunk (that is, the initial parent chunk) 1001 determined in the process B1 (Yes in step 410). In this case, the variable-length deduplication module 33 stores the identifier H _DEF and data (data fragment) “me specified” of the parent chunk 1001 determined in the process B1 in the chunk list table 312 as indicated by an arrow 1004 in FIG. Register (step 502). Although not shown in FIG. 9, the variable-length deduplication module 33 registers the document name “document # 1” of the document 111 and the identifier H _DEF of the parent chunk 1001 in the document configuration table 311 (step 503).

(C) Storage process SX 3
A storage process No. 3 (hereinafter referred to as storage process SX3) for storing the document 111, which is executed subsequent to the above-described storage process SX2, will be described with reference to FIG. In FIG. 11, the document configuration table 311 is omitted.

C1)
The variable-length deduplication module 33 sequentially determines the child chunks from the end position of the parent chunk 1001 determined in the storage process SX2 (step 505) until the connection window size W (W = 10) or more. (Steps 403 to 406). In the example of FIG. 10, the variable-length deduplication module 33 defines a cut point at the 38-th character from the beginning of the document 111, it is assumed that defines the child chunk c _6.

The subsequent processing is the same as the processing A2-2 to A2-3 in the storage processing SX1. However, in the example of FIG. 11, since only the child chunk c ₆ (“by path is ope”) is equal to or larger than the linked window size W (W = 10), the child chunk c ₆ alone is determined as the parent chunk 1101. The variable-length deduplication module 33 uses the hash value H _G ′ generated from the identifier (hash value) c ₆ .hash = H _G of the child chunk c ₆ (“by path is ope”) as the identifier (hash value) of the parent chunk 1101. ). In the example of FIG. 11, the data fragment corresponding to the identifier H _G ′ of the parent chunk 1101 is not registered in the chunk list table 312. For this reason, the determination result of step 408 regarding the parent chunk 1101 becomes unregistered (No) as indicated by an arrow 1111 in FIG. 11, and the process proceeds to the next process C2.

C2)
The process C2 is the same as the process A4 in the storage process SX1. In the example of FIG. 11, the position shifted from the start position of the parent chunk 1101 determined in the above-described process C1 to the end side of the document 111 by the length of the child chunk c ₆ constituting the parent chunk 1101 (that is, reset) The parent chunk start position) exceeds the end position of the parent chunk (that is, the initial parent chunk) 1101 (Yes in step 410). In this case, the variable-length deduplication module 33 displays the identifier H _G ′ and data (data fragment) “by path is ope” of the parent chunk 1101 defined in the process C1 as a chunk list as indicated by an arrow 1102 in FIG. Register in the table 312 (step 502). Although omitted in FIG. 11, the variable-length deduplication module 33 registers the document name “document # 1” of the document 111 and the identifier H _G ′ of the parent chunk 1101 in the document configuration table 311 (step 503).

(D) Storage process SX 4
A storage process SX No. 4 (hereinafter referred to as a storage process SX4) for storing the document 111, which is executed subsequent to the above-described storage process SX3, will be described with reference to FIG. In FIG. 12, the document configuration table 311 is omitted.

D1)
D1-1) The variable-length deduplication module 33 removes the child chunk from the position of the end of the parent chunk 1101 determined in the storage process SX3 (step 505) until it reaches the connected window size W (W = 10) or more. These are determined sequentially (steps 403 to 406). In the example of FIG. 12, it is assumed that the variable-length deduplication module 33 defines cut points at the 41st, 47th, and 50th characters from the beginning of the document 111, and defines child chunks c ₇ , c ₈ , and c _9. .

The subsequent processing is the storage processing SX1, is similar to the processing A2-2～A2-3, parent chunk 1201 is determined by linking the child chunk _{c 7, c 8, c 9} . In the example of FIG. 12, the data fragment corresponding to the identifier H _HIJ of the parent chunk 1201 is not registered in the chunk list table 312. For this reason, the determination result of step 408 regarding the parent chunk 1201 becomes unregistered (No) as indicated by an arrow 1211 in FIG. 12, and the process proceeds to the next process D2.

D2)
Processing D2 is the same as the processing A2 in storage processing SX1, new child chunk c ₁₀ is determined. Then, the parent chunk 1202 is defined by connecting the child chunks c ₈ , c ₉ , and c ₁₀ . In the example of FIG. 12, the data fragment corresponding to the identifier H _IJK of the parent chunk 1202 is not registered in the chunk list table 312. For this reason, the determination result of step 408 regarding the parent chunk 1202 becomes unregistered (No) as indicated by an arrow 1212 in FIG. 12, and the process proceeds to the next process D3.

D3)
The process D3 is the same as the process A3 in the storage process SX1. Variable-length deduplication module 33, the length of the first child chunk c ₈ column configuration child chunk c _8, c _9, c ₁₀ a parent chunk 1202 defined by the above-described process D2, the end side of the document 111 From the position shifted to (step 409), child chunks are sequentially determined until the connection window size W (W = 10) or more is reached (steps 403 to 406). At this time, it is not necessary to repeat the process for the portion where the child chunks c ₉ and c ₁₀ are determined in the process D2. In the example of FIG. 12, the variable-length deduplication module 33 defines a cut point at the 57-th character from the beginning of the document 111, it is assumed that defines the child chunk c _11. The variable-length deduplication module 33 concatenates the child chunks c ₉ , c ₁₀ , and c ₁₁ to determine a parent chunk 1203. In the example of FIG. 10, the data fragment corresponding to the identifier H _JKL of the parent chunk 1203 is not registered in the chunk list table 312. For this reason, the determination result of step 408 regarding the parent chunk 1203 becomes unregistered (No) as indicated by an arrow 1213 in FIG. 12, and the process proceeds to the next process D4.

D4)
The process D4 is the same as the process A4 in the storage process SX1. That is, in the example of FIG. 12, the length of the first child chunk c _{9 in} the column of the child chunks c ₉ , c ₁₀ , c ₁₁ constituting the parent chunk 1203 defined in the process D3 is shifted toward the end of the document 111. The position exceeds the position of the end of the parent chunk 1201 determined in the process D1 (Yes in step 410). In this case, the variable-length deduplication module 33 uses the chunk list table 312 as shown by the arrow 1204 in FIG. 12 for the identifier H _HIJ and the data (data fragment) “ned for read” of the parent chunk 1201 determined in the process D1. (Step 502). Although omitted in FIG. 12, the variable-length deduplication module 33 registers the document name “document # 1” of the document 111 and the identifier H _HIJ of the parent chunk 1201 in the document configuration table 311 (step 503).

(E) Storage process SX 5
A storage process SX No. 5 (hereinafter referred to as storage process SX5) for storing the document 111, which is executed subsequent to the above-described storage process SX4, will be described with reference to FIG. In this example, for the sake of simplicity, the character string “and” is assumed to be the end of the document 111 for convenience. In FIG. 13, the document configuration table 311 is omitted.

E1)
The variable-length deduplication module 33 sequentially determines child chunks from the end position of the parent chunk 1201 determined in the storage process SX4 (step 505) (steps 403 to 406). In the example of FIG. 13, it is assumed that child chunks c ₁₀ and c ₁₁ are determined, and as a result, the cut-out point has reached the end of the document 111 (step 404). In this case, the variable-length deduplication module 33 determines whether the cut-out point is equal to or larger than the linked window size W (W = 10) from the end position of the parent chunk 1201 determined in the storage process SX4. A parent chunk 1301 is determined by connecting the chunks c ₁₀ and c ₁₁ (step 414).

E2)
The variable-length deduplication module 33 registers the identifier H _KL and data (data fragment) “ing and” of the parent chunk 1301 determined in the process E1 in the chunk list table 312 as indicated by an arrow 1302 in FIG. Step 412). Although omitted in FIG. 13, the variable-length deduplication module 33 registers the document name “document # 1” of the document 111 and the identifier H _KL of the parent chunk 1301 in the document configuration table 311 (step 416). Thereby, the storage process SX for storing the document 111 is completed.

  The state of the document configuration table 311 and the chunk list table 312 (that is, the registration state of the document 111) after the storage process SX (that is, the storage processes SX1 to SX5) is completed is cut out from the document 111 and the document 111. It is shown in FIG. 14 together with the parent chunk column.

(3) Storage processing SY
Next, storage processing SY for storing the document 112 will be described with reference to FIGS. 15 to 19 show the storage operation of the document 112 performed after the storage of the document 111 together with the state of the document configuration table 311. FIG. 20 shows the state of the document configuration table 311 and the chunk list table 312 after storage of the document 112 (that is, after storage of the documents 111 and 112), together with the column of the parent chunk extracted from the document 112 and the document 112, FIG. 21 shows the state of the document configuration table 311 and the chunk list table 312 after the documents 111 and 112 are stored, together with the parent chunk columns extracted from the documents 111 and 112.

F) Storage process SY 1
A storage process SY 1 (hereinafter referred to as storage process SY1) for storing the document 112 after storing the document 111 will be described with reference to FIG. In FIG. 15, the document configuration table 311 is omitted.

F-1) The variable-length deduplication module 33 sequentially determines child chunks from the top of the document 112 to a place where the linked window size W (W = 10) or more (steps 403 to 406). In the example of FIG. 15, the variable-length deduplication module 33 determines cut points at the fifth, eighth, and eleventh characters from the beginning of the document 112, and sets child chunks c ₀ (“The f”), c ₁ (“ile "), C ₂ (" na ").

F-2) The variable-length deduplication module 33 sequentially determines the child chunks c ₀ (“The f”), c ₁ (“ile”), and c ₂ (“na”) until the connection window size W is reached. Now, these child chunks are concatenated to define a parent chunk 1501, and an identifier (hash value) of the parent chunk 1501 is generated (step 407). In this example, the variable-length deduplication module 33 uses the identifiers (hash values) c ₀ .hash = of the child chunks c ₀ (“The f”), c ₁ (“ile”), and c ₂ (“na”) to be connected. _{H a, c 1 .hash = H} B, the hash value H _ABC generated from c ₂ .hash = H _C, an identifier of the parent chunk 1501 (hash value).

F-3) Based on the identifier H _ABC of the parent chunk 1501, the variable length deduplication module 33 determines whether a data fragment corresponding to the identifier H _ABC is registered in the chunk list table 312 (step 408). As shown in FIG. 15, a data fragment corresponding to the identifier H _ABC is registered in the chunk list table 312. For this reason, the determination result of step 408 regarding the parent chunk 1501 becomes Yes (registered) as indicated by an arrow 1511 in FIG. In this case, the chunk list table 312 does not change before and after the process F as indicated by an arrow 1502 in FIG. Although omitted in FIG. 15, the variable length deduplication module 33 registers the document name “document # 2” of the document 112 and the identifier H _ABC of the parent chunk 1501 in the document configuration table 311 before proceeding to the processing G. (Step 503).

G) Storage process SY 2
A storage process SY 2 (hereinafter referred to as storage process SY2) for storing the document 112, which is executed subsequent to the above-described storage process SY1, will be described with reference to FIG. In FIG. 16, the document configuration table 311 is omitted.

G1)
G1-1) The variable-length deduplication module 33 removes the child chunk from the end position of the parent chunk 1501 determined in the storage process SY1 (step 505) until it reaches the connected window size W (W = 10) or more. These are determined sequentially (steps 403 to 406). In the example of FIG. 16, it is assumed that the variable-length deduplication module 33 determines a cut-out point at the 13th and 22nd characters from the top of the document 112, and determines child chunks c ₃ and c ₄ .

G2-2) The variable length deduplication module 33 determines the child chunks c ₃ and c ₄ until the connection window size W is equal to or larger than the connection window size W, and concatenates the child chunks c ₃ and c ₄ to obtain the parent chunk 1601. The identifier (hash value) of the parent chunk 1601 is generated (step 407). In this example, the variable-length deduplication module 33 uses the identifiers (hash values) c ₃ .hash = H _X , c ₄ .hash = of the child chunks c ₃ (“me”) and c ₄ (“ABCD spe”) to be linked. The hash value H _XY generated from H _{Y is used as} the identifier (hash value) of the parent chunk 1601.

G2-3) length deduplication module 33 determines whether based on the identifier H _XY parent chunk 1601, data fragment corresponding to the identifier H _XY in the chunk list table 312 is registered (step 408). In this example, since the data fragment corresponding to the identifier _HXY is not registered, the process proceeds to the next process G3.

G3)
G3-1) length deduplication module 33, the length of the child chunk c ₃ of the column head of the configuration child chunk c _3, c ₄ parent chunks 1601 defined by the above-described process G2, the end of the document 112 From the position shifted to the side (step 409), child chunks are sequentially determined until the connection window size W (W = 10) or more is reached (steps 403 to 406). At this time, it is not necessary to repeat the process for the part for which the child chunk c ₄ is determined in the process G2. In the example of FIG. 16, the variable-length deduplication module 33 defines a point cut at the 25 th character from the beginning of the document 112, and define those new children chunk c _5.

G3-2) The variable length deduplication module 33 determines the child chunks c ₄ and c ₅ until the connection window size W is equal to or larger than the connection window size W, and concatenates the child chunks c ₄ and c ₅ to obtain the parent chunk 1602. Then, an identifier (hash value) of the parent chunk 1602 is generated (step 407). In this example, the variable-length deduplication module 33 uses the identifiers (hash values) c ₄ .hash = H _Y and c ₅ .hash = of the child chunks c ₄ (“ABCD spe”) and c ₅ (“cif”) to be connected. The hash value H _YE generated from H _{E is used as} the identifier (hash value) of the parent chunk 1602.

G3-3) The variable-length deduplication module 33 determines whether a data fragment corresponding to the identifier H _YE is registered in the chunk list table 312 based on the identifier H _YE of the parent chunk 1602 (step 408). In this example, since the data fragment corresponding to the identifier H _YE is not registered, the process proceeds to the next process G4.

G4)
G4-1) In the example of FIG. 16, only the length of the first child chunk c _{4 in} the column of the child chunks c ₄ and c ₅ constituting the parent chunk 1602 defined in the processing G3 described above is added to the end of the document 112. The shifted position exceeds the position of the end of the parent chunk (that is, the initial parent chunk) 1601 determined in the process G1. Therefore, the “deviation” of the parent chunk start offset reset in step 409 from the start offset of the parent chunk 1601 is equal to or larger than the linked window size W (Tes in step 410). In this case, the variable-length deduplication module 33 uses the chunk list table 312 as shown by the arrow 1603 in FIG. 16 for the identifier H _XY and the data (data fragment) “me ABCD spe” of the parent chunk 1601 determined in the process G1. (Step 502). Although omitted in FIG. 16, the variable-length deduplication module 33 registers the document name “document # 2” of the document 112 and the identifier H _XY of the parent chunk 1601 in the document configuration table 311 (step 503).

H) Storage processing SY 3
A storage process SY 3 (hereinafter referred to as storage process SY3) for storing the document 112, which is executed subsequent to the above-described storage process SY2, will be described with reference to FIG. In FIG. 17, the document configuration table 311 is omitted.

H1)
H1-1) The variable-length deduplication module 33 selects child chunks from the end position of the parent chunk 1601 determined in the storage process SY2 (step 505) until it reaches the connected window size W (W = 10) or more. These are determined sequentially (steps 403 to 406). At this time, it is not necessary to repeat the process for the part in which the child chunk c ₅ after the end position of the parent chunk 1601 is determined in the storage process SY2. In the example of FIG. 16, the variable-length deduplication module 33 sets a cut point at the 29th and 43rd characters from the top of the document 112, and creates new child chunks c ₆ (“ied”), c ₇ (“by path is ope ”).

H1-2) length deduplication module 33, where defining a child chunk c _5, c _6, c ₇ until it becomes a connecting window size W or more, connecting their child chunks c _5, c _6, c ₇ Then, the parent chunk 1701 is determined, and an identifier (hash value) of the parent chunk 1701 is generated (step 407). In this example, the variable-length deduplication module 33 identifies the identifiers (hash values) c _{5. Of} child chunks c ₅ (“cif”), c ₆ (“ied”), and c ₇ (“by path is ope”) to be connected. Hash value H _EFG generated from hash = H _E , c ₆ .hash = H _F , c ₇ .hash = H _{G is used as} the identifier (hash value) of the parent chunk 1701.

H1-3) length deduplication module 33 determines whether based on the identifier H _EFG parent chunk 1701, data fragment corresponding to the identifier H _EFG in the chunk list table 312 is registered (step 408). In this example, the data fragment corresponding to the identifier _HEFG is not registered. For this reason, the determination result of step 408 regarding the parent chunk 1701 becomes unregistered (No) as indicated by an arrow 1711 in FIG. 17, and the process proceeds to the next process H2.

H2)
H2-1) length deduplication module 33, the length of the first child chunk c ₅ columns parent chunks constituting 170101 child chunk c _5, c _6, c ₇ was determined by the above-described process H1, document Child chunks are sequentially determined from the position shifted to the end of 112 (step 409) to a position where the connection window size W (W = 10) or more is reached (steps 403 to 406). At this time, it is not necessary to repeat the process for the part where the child chunks c ₆ and c ₇ are determined in the process H1. In the example of FIG. 17, there is no newly defined child chunk from the position shifted by the length of the child chunk c ₅ to a place where the connection window size W or more is reached.

H2-2) Therefore, the variable-length deduplication module 33 determines the parent chunk 1702 by concatenating the remaining child chunks c ₆ and c ₇ excluding the first child chunk c ₅ from the parent chunk 1701 determined in the processing H1. An identifier (hash value) of the parent chunk 1702 is generated (step 407). In this example, the variable-length deduplication module 33 uses identifiers (hash values) c ₆ .hash = H _F , c _{7. For the} child chunks c ₆ (“ied”) and c ₇ (“by path is ope”) to be connected. the hash = H _G hash value H _FG generated from an identifier (hash values) of the parent chunk 1702.

H2-3) length deduplication module 33 determines whether based on the identifier H _FG parent chunk 1702, data fragment corresponding to the identifier H _FG in the chunk list table 312 is registered (step 408). In this example, the data fragment corresponding to the identifier _HFG is not registered. For this reason, the determination result of step 408 regarding the parent chunk 1702 becomes unregistered (No) as indicated by an arrow 1712 in FIG. 17, and the process proceeds to the next process H3.

H3)
H3-1) length deduplication module 33, the length of the child chunk _{c 6} the head of the column structure of the parent chunk 170,102 child chunk c _6, c ₇ was determined by the above-described processing H2, the end of the document 112 From the position shifted to the side (step 409), child chunks are sequentially determined until the connection window size W (W = 10) or more is reached (steps 403 to 406). In this case, again the process is not necessary for the portion defining the processing H2 child chunk c _7. In the example of FIG. 17, there is no newly defined child chunk from the position shifted by the length of the child chunk c _{6 to} the place where the connection window size W or more is reached. Further, only the child chunk c ₇ becomes the connection window size W (W = 10) or more.

H3-2) where the variable-length deduplication module 33 defines a remaining child chunk c ₇ alone, excluding the child chunk c ₆ from the parent chunk 1702 top of which defines the processing H2 as a parent chunk 1703, of the parent chunk 1703 An identifier (hash value) is generated (step 407). In this example, the variable-length deduplication module 33 uses the hash value H _G ′ generated from the identifier (hash value) c ₇ .hash = H _G of the child chunk c ₇ (“by path is ope”) as the parent chunk 1703. An identifier (hash value) is used.

H3-3) The variable length deduplication module 33 determines whether a data fragment corresponding to the identifier H _G ′ is registered in the chunk list table 312 based on the identifier H _G ′ of the parent chunk 1703 (step 408). . As shown in FIG. 17, in the chunk list table 312, data fragments corresponding to the identifier H _G ′ are registered. For this reason, the determination result of step 408 regarding the parent chunk 1703 becomes Yes (registered) as indicated by an arrow 1713 in FIG. 17, and the process proceeds to the next process H4. Although omitted in FIG. 17, the variable-length deduplication module 33 stores the document name “document # 2” of the document 112 and the identifier H _G ′ of the parent chunk 1501 in the document configuration table 311 before proceeding to the processing H4. Registration is performed (step 412).

H4)
H4-1) length deduplication module 33 includes a parent chunk 1601 defined by the storage processing SY2 described above (i.e., data fragments identified by the identifier H _XY), parent chunks 1703 defined by the process H3 (i.e., if a column of the child chunks or child chunks between data fragments) identified by the identifier H _G '(No in step 413), the columns of the child chunk or child chunk defines the parent chunk, the parent chunk Identifier (hash value) is generated (step 501). In the example of FIG. 17, child chunks c ₅ and c ₆ exist between the parent chunk 1601 and the parent chunk 1703. Therefore, the variable length deduplication module 33 concatenates the child chunks c ₅ and c ₆ to define a parent chunk 1704, and generates an identifier (hash value) of the parent chunk 1704 (step 501). That length deduplication module 33, connected child chunk _{c 5 ( "cif"),} c 6 the identifier (hash values) of _{( "ied") c 5 .hash} = H E, from c ₆ .hash = H _F The generated hash value H _EF is set as an identifier (hash value) of the parent chunk 1704.

H4-2) The variable-length deduplication module 33 registers the identifier H _EF of the parent chunk 1704 and the data (data fragment) “cified” in the chunk list table 312 as indicated by an arrow 1705 in FIG. 17 (step 502). . Although omitted in FIG. 17, the variable-length deduplication module 33 registers the document name “document # 2” of the document 112 and the identifier H _EF of the parent chunk 1704 in the document configuration table 311 (step 503).

I) Storage process SY 4
A storage process SY 4 (hereinafter referred to as a storage process SY4) for storing the document 112, which is executed subsequent to the above-described storage process SY3, will be described with reference to FIG. In FIG. 18, the document configuration table 311 is omitted.

I-1) The variable-length deduplication module 33 has a connection window size W (W = 10) or more from the position of the end of the parent chunk 1703 determined to have been registered in the storage process SY3 (step 505). Until then, child chunks are sequentially determined (steps 403 to 406). In the example of FIG. 18, the variable-length deduplication module 33 sets a cut point at the 46th, 52nd, and 56th characters from the top of the document 112, and creates new child chunks c ₈ (“ned”) and c9 (“for” r "), c10 (" ead ").

In the subsequent process storage processing SY1, is similar to the process F-2~F-3, parent chunk 1801 is determined by linking the child chunk _{c 8, c 9, c 10} . In the example of FIG. 18, the data fragment corresponding to the identifier H _HIJ of the parent chunk 1801 is registered in the chunk list table 312 (Yes in step 408). For this reason, the determination result of step 408 regarding the parent chunk 1801 becomes Yes (registered) as indicated by an arrow 1811 in FIG. In this case, the chunk list table 312 does not change before and after the process I as indicated by an arrow 1802 in FIG. Although omitted in FIG. 18, the variable-length deduplication module 33 registers the document name “document # 2” of the document 112 and the identifier H _HIJ of the parent chunk 1801 in the document configuration table 311 before proceeding to the process J. (Step 503).

J) Storage process SY, part 5
A storage process SY No. 5 (hereinafter referred to as a storage process SY5) for storing the document 112, which is executed subsequent to the above-described storage process SY4, will be described with reference to FIG. In FIG. 19, the document configuration table 311 is omitted.

J1)
J1-1) The variable-length deduplication module 33 sequentially determines child chunks (steps 403 to 406) from the end position of the parent chunk 1801 determined in the storage process SY4 (step 505). In the example of FIG. 19, it is assumed that child chunks c ₁₁ and c ₁₂ are determined, and as a result, the cut-out point has reached the end of the document 111 (step 404). In this case, the variable-length deduplication module 33 determines whether the cut-out point is equal to or larger than the linked window size W (W = 10) from the end position of the parent chunk 1801 determined in the storage process SY4. A parent chunk 1901 is determined by concatenating the chunks c ₁₁ (“in”) and c ₁₂ (“g and”) (step 414).

J1-2) The data fragment corresponding to the identifier (hash value) H _KL of the parent chunk 1901 is registered in the chunk list table 312 (Yes in step 415). For this reason, the determination result of step 415 regarding the parent chunk 1901 is Yes (registered) as indicated by an arrow 1911 in FIG. In this case, the process (step 416) of registering the identifier H _KL and data (data fragment) “ing and” of the parent chunk 1901 defined in step 414 in the chunk list table 312 is not performed. Therefore, the chunk list table 312 does not change before and after the process J, as indicated by an arrow 1902 in FIG. Although omitted in FIG. 19, the variable-length deduplication module 33 registers the document name “document # 2” of the document 112 and the identifier H _KL of the parent chunk 1501 in the document configuration table 311 (step 503). Thereby, the storage process SY for storing the document 112 is completed.

  After the storage process SX is completed following the storage process SX (that is, the storage processes SY1 to SY5), that is, after the documents 111 and 112 are stored, the state of the document configuration table 311 and the chunk list table 312 is changed to the document 112 and FIG. 20 shows the parent chunk columns cut out from the document 112.

  Similarly, the states of the document configuration table 311 and the chunk list table 312 after storing the documents 111 and 112 are shown in FIG. 21 together with the parent chunk columns cut out from the documents 111 and 112. In this embodiment in which the document 112 is stored after the document 111 is stored, it can be seen from FIG. 21 that the document 112 is registered while eliminating duplicate data by the storage process XY for storing the document 112.

<Document acquisition processing>
Next, document acquisition processing in the document storage device 10 will be described with reference to the flowchart of FIG.
First, it is assumed that a document acquisition instruction is sent from the client device 20 to the document storage device 10 via the network 30. This document acquisition instruction includes a document name that specifies a document to be acquired from the document storage device 10.

  The document acquisition instruction from the client device 20 sent to the document storage device 10 is received by the command reception module 32 of the document storage device 10. When this document acquisition instruction is received by the instruction reception module 32, the document acquisition unit 320 in the instruction reception module 32 is associated with the document name specified by the document acquisition instruction and is registered in the document configuration table 311. The identifier of the chunk (parent chunk) group is acquired (step 2201). As described above, the arrangement order of the identifiers of the acquired chunk groups matches the sequence of the chunk groups in the corresponding document.

  When the document acquisition unit 320 acquires a group of identifiers from the document configuration table 311, the document acquisition unit 320 acquires a group of chunks (data fragments) registered in the chunk list table 312 in association with the group of identifiers (step 2202). .

  Based on the group of chunks acquired from the chunk list table 312, the document acquisition unit 320 receives the chunk group from the client device 20 so that the order of the group of chunks matches the order of the identifiers of the group of chunks acquired previously. The document data of the document name designated by the document acquisition instruction is reconstructed (step 2203).

  The command reception module 32 returns the document data reconstructed by the document acquisition unit 320 to the client device 20 as a response to the document acquisition instruction from the client device 20 (step 2204).

  Incidentally, there are cases where the client device 20 wishes to acquire a data fragment on a document (document data) from the document storage device 10 in response to a request from the user. As a method for the client device 20 to acquire the data fragment on the document from the document storage device 10, in addition to the document name of the document, the position of the data fragment on the document and the length of the data fragment are specified. The method is known. In order for the document storage apparatus 10 to adapt to such a method, the document data of the document having the document name designated by the client apparatus 20 is reconstructed as described above, and the client apparatus 20 uses the document data from the document data. It is necessary to obtain a data fragment with the specified position and length.

  Therefore, for example, when registering an identifier of a chunk (parent chunk) in association with a specified document name in the document configuration table 311, information indicating the position / length of the chunk on the corresponding document data is displayed. It may be added to the identifier. In this way, by referring to this information and specifying the chunk held in the chunk list table 312 in association with the identifier to which this information is added, the specified position / length on the specified document is specified. Data fragment can be obtained.

<Summary of this embodiment>
As described above, in the present embodiment, the document storage device 10 divides arbitrary document data into variable-length chunks (parent chunk or registered parent chunk, or first data fragment) while performing duplication detection. Functions as a dividing device.

  According to the present embodiment, the length of a child chunk (second data fragment) is set as an offset interval for duplication detection, and is likely to be longer than the length of the child chunk and can be used as a registration target. By adopting a configuration in which duplication detection is performed with the length of the highly probable parent chunk (third data fragment), the registered parent chunk (first data) can be obtained by a simpler and faster method compared to the conventional technique. Duplicate detection can be performed while maintaining a high deduplication rate while reducing the number of data fragments) (that is, the number of divisions).

  Further, according to the present embodiment, a plurality of children that are determined when a certain condition is satisfied while determining child chunks (second data fragments) in order from one end of the data to be divided. Split by concatenating chunks (without linking when one child chunk is determined) and determining the parent chunk (third data fragment) and detecting whether the determined parent chunk is duplicated When the target data is input to the stream state, duplicate detection can be performed at high speed.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. For example, in the above embodiment, the document data is divided from the top of the document data to the end of the document data. However, the document data may be divided from the end of the document data to the start of the document data. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment.

  DESCRIPTION OF SYMBOLS 10 ... Document storage apparatus, 20 ... Client apparatus, 30 ... Network, 31 ... Document storage part, 32 ... Command reception module (input means), 33 ... Variable length deduplication module, 34 ... Working memory, 311 ... Document structure table 312 ... Chunk list table, 320 ... Document acquisition unit, 331 ... Child chunk determination unit (second data fragment determination unit), 332 ... Parent chunk determination unit (third data fragment determination unit), 333 ... Identifier generation unit 334... Duplicate detection unit, 335... Parent chunk registration unit (first data fragment determination means), 336... Control unit, 341... Document buffer, 342.

Claims (11)

In an apparatus including an input unit, a first data fragment determination unit, a second data fragment determination unit, a third data fragment determination unit, a duplication detection unit, and a control unit, a plurality of arbitrary data can be detected while performing duplication detection. A data dividing method for dividing the first data fragment of any length,
An input step in which the input means inputs the arbitrary data;
Of the input arbitrary data, the second data fragment determination means has an arbitrary length or a predetermined length from the remaining data portion not yet determined as the first data fragment. A first determining step for sequentially determining two data fragments;
The second data fragment itself or a combination of a plurality of second data fragments determined in the first step is changed to the third data fragment until the state that satisfies the predetermined first condition is reached. A second determining step in which the fragment determining means determines as one third data fragment;
The duplication detection means detects the duplication of the determined third data fragment depending on whether the first data fragment of the bit string that matches the determined third data fragment has already been determined. A detection step;
A third determining step in which the first data fragment determining means determines the determined third data fragment as the first data fragment when the duplication is detected;
If the duplicate is not detected, the first and second determination steps are re-executed, so that a new second data fragment or a new one is reached before reaching the state satisfying the first condition. A plurality of second data fragments are determined, and the new one second data fragment itself, a combination of the new plurality of second data fragments, and the third data fragment in which the duplication is not detected A combination of a part of and a new second data fragment, or a combination of a part of a third data fragment in which the duplication is not detected and the new plurality of second data fragments, Control for causing the third data fragment determining means to determine one new third data fragment is performed until the duplication is detected in a state where a predetermined second condition is satisfied. A first control step means repeats,
If the duplication is not detected even if the first control step is repeated in a state where the second condition is satisfied, the first condition among the second data fragments determined in the meantime. A fourth determination step in which the first data fragment determination means determines one second data fragment itself or a combination of a plurality of second data fragments as a new first data fragment that satisfies When,
A second control step for the control means to repeat the first control step until the input arbitrary data is all divided into the first data fragments. Split method. The second data fragment is determined in order from a first end of the remaining data portion toward a second end of the remaining data portion;
The first condition is the length of the one second data fragment determined in the first step, or the plurality of second data fragments sequentially determined in the first step. The length of the plurality of second data fragments after the connection,
When the plurality of second data fragments are determined in the first step before reaching the state satisfying the first condition, the plurality of second data fragments are connected in the order of determination. The data division method according to claim 1, wherein the third data fragment is determined as a combination of the plurality of second data fragments. The second determining step includes
At least one second data fragment closest to the first end of the remaining data portion included in the third data fragment in which no duplication is detected is extracted from the third data fragment. Removing step,
The new one second data fragment or the plurality of second plurality of second data fragments determined in the first determination step until the state satisfying the first condition is reached. The data division method according to claim 2, further comprising: determining a new third data fragment by incorporating the second data fragment into the removed third data fragment. The one second data fragment, which is determined as the new first data fragment when the duplicate is not detected even if the first control step is repeated within the range of the second condition, Itself or a combination of the plurality of second data fragments is a third fragment first determined after the most recent determination of the first data fragment;
The third determining step is determined last time when the first data fragment is determined as a result of detecting the duplication by repeating the first control step within the range of the second condition. The first data fragment determination means newly sets a data fragment that is sandwiched between the first data fragment and the first data fragment determined this time and has not yet been determined as the first data fragment. The data division method according to claim 1, further comprising a step of determining as a first data fragment. The first condition is:
The length of the one second data fragment determined in the first step, or the concatenation of the plurality of second data fragments sequentially determined in the first step The length of the plurality of second data fragments is
Exceeding the length of a predetermined standard,
Equal to the length of a predetermined standard,
Be closest to the length of the predetermined standard,
5. The data dividing method according to claim 4, wherein the data dividing method is any one of a maximum value not exceeding a predetermined reference length.   The second condition is from the first end portion of the remaining data portion to the end portion farthest from the first end portion of the third data fragment in which duplication has not been detected most recently. 5. The data division method according to claim 4, wherein the condition is related to a data fragment length which is a length. The second condition is that the data fragment length is:
Exceeding the length of a predetermined standard,
Equal to the length of a predetermined standard,
Be closest to the length of the predetermined standard,
The data dividing method according to claim 6, wherein the data dividing method is any one of a maximum value not exceeding a predetermined reference length. In a data dividing device for dividing arbitrary data into a plurality of first data fragments having an arbitrary length while performing duplication detection,
Input means for inputting the arbitrary data;
A second data fragment having an arbitrary length or a predetermined length is sequentially determined from the remaining data portion of the input arbitrary data that has not yet been determined as the first data fragment. Two data fragment determination means;
One second data fragment itself or a combination of a plurality of second data fragments determined by the second data fragment determining means until reaching a state satisfying a predetermined first condition, Third data fragment determining means for determining as one third data fragment;
Duplicate detection means for detecting whether or not the determined third data fragment is duplicated depending on whether or not the first data fragment of the bit string that matches the determined third data fragment has already been determined;
First data fragment determination means for determining the determined third data fragment as the first data fragment when the duplication is detected;
When the duplication is not detected, a new second data fragment or a plurality of new second data fragments is obtained by the second data fragment determining means until the state satisfying the first condition is reached. The new one second data fragment itself, a combination of the new plurality of second data fragments, a part of the third data fragment in which the duplication is not detected, and the new data fragment A combination of one second data fragment or a combination of a part of the third data fragment in which no duplication is detected and the new plurality of second data fragments is determined as the third data fragment. The control for determining as one new third data fragment by the means is repeated until the duplication is detected in a state satisfying a predetermined second condition, and this control is repeated. Teeth, any data that is the input comprises a further repeat control means until the split to all the first data fragments,
If the duplicate is not detected even if the control is repeated in a state where the second condition is satisfied, the first data fragment determination means includes the second data fragment determined in the meantime. One data fragment itself or a combination of a plurality of second data fragments that satisfies the first condition is determined as a new first data fragment. . The second data fragment determining means sequentially determines the second data fragment from the first end of the remaining data portion toward the second end of the remaining data portion;
The first condition is the length of the one second data fragment determined by the second data fragment determining means, or the plurality of second data sequentially determined by the second data fragment determining means. Is a condition regarding the length of the plurality of second data fragments after the connection,
When the plurality of second data fragments are determined by the second data fragment determination means before reaching the state satisfying the first condition, the third data fragment determination means The data dividing apparatus according to claim 8, wherein the third data fragment is determined as a combination of the plurality of second data fragments by concatenating the second data fragments in the order of determination.   The third data fragment determining means includes at least one second data on the side closest to the first end of the remaining data portion, which is included in the third data fragment in which the duplication is not detected. The fragment is removed from the third data fragment, and the new one second data fragment or the new data determined by the second data fragment determination means until the state that satisfies the first condition is reached. A new third data fragment is determined by incorporating a plurality of second data fragments into the third data fragment from which the at least one second data fragment has been removed. Item 12. The data dividing device according to Item 9. The one second data fragment itself, which is determined as the new first data fragment when the duplicate is not detected even if the control is repeated within the range of the second condition, or The combination of the plurality of second data fragments is a third fragment determined first after the most recently determined first data fragment;
The first data fragment determining means determines the first data fragment previously determined when the first data fragment is determined as a result of detecting the duplication by repeating the control within the range of the second condition. A data fragment that is sandwiched between the data fragment and the first data fragment that has not yet been determined as the first data fragment is determined as a new first data fragment. Item 11. The data dividing device according to any one of Items 8 to 10.

Images