JPWO2013108745A1 - Storage device, control method thereof, and program - Google Patents

Abstract

  The more data stored in the storage device, the more difficult it is for the user to search for the desired data. Therefore, a storage device that can be a similar search target even for unknown binary data is desired. Deduplication control unit that eliminates storage of duplicate file data, an index word storage unit that stores pseudo-random numbers based on the storage target file data as index terms, a search interface unit that acquires search target file data, and a search target file A search chunk data generation unit that divides data and generates a search chunk data sequence, a search pseudo random number unit that calculates a pseudo random number based on each chunk data in the search chunk data sequence, and a pseudo random number as a search term And a similarity search unit that extracts all index terms that satisfy the similarity criterion. (Figure 1)

Description

[Description of related applications]
The present invention is based on the priority claim of Japanese patent application: Japanese Patent Application No. 2012-006566 (filed on January 16, 2012), the entire description of which is incorporated herein by reference. Shall.
The present invention relates to a storage apparatus, a control method thereof, and a program. In particular, the present invention relates to a search system that uses a plurality of file paths similar to search target file data as search results, a control method thereof, and a program.

  When storing data in a storage device, it is useful to check for duplication with data already stored. Such a deduplication storage apparatus is being used in many systems as a technique for reducing data storage costs.

Here, Japanese Patent Application Laid-Open No. 2004-228620 discloses a technique that can write data at high speed by concentrating deduplication processing on a storage device.

JP 2011-203842 A

  The disclosure of the above prior art document is incorporated herein by reference. The following analysis has been made from the viewpoint of the present invention.

  As described above, a storage apparatus having a deduplication function can reduce physical storage capacity. Therefore, the storage apparatus can store a large amount of data by providing the deduplication function. However, the more data stored in the storage device, the more difficult it is for the user to search for the desired data.

  Patent Document 1 discloses a deduplication technique for a storage device. However, Patent Document 1 does not disclose a technique for extracting desired data from data stored in a storage device.

  Here, creating an index word for search in advance is a useful method for extracting data similar to a search condition. However, prior information about file data to be searched is required when creating an index word for search. There are various methods for extracting prior information about file data. For example, for document data, there is a method of extracting words from document data in order to create a search index. For image data, there is a method of extracting colors and shapes from image data in order to create a search index.

  However, there are cases where the file data format is unknown and prior information about the file data cannot be extracted. For this reason, when searching for data stored in the storage device, it is necessary to limit the types of file data used as search conditions.

  Therefore, a storage device, a control method thereof, and a program that can be used as a similar search target even for unknown binary data are provided.

  According to a first aspect of the present invention, a deduplication control unit that eliminates storage of duplicate file data, an index word storage unit that stores pseudo random numbers based on the storage target file data as index terms, and search target file data The search interface unit for acquiring the search target file data, the search chunk data generation unit for generating the search chunk data sequence by dividing the search target file data, and the pseudo random numbers are calculated based on the respective chunk data in the search chunk data sequence A storage device is provided that includes a search pseudorandom number section and a similar search section that extracts all index words that satisfy the similarity criterion using the pseudorandom number as a search word.

According to a second aspect of the present invention, there is provided a storage apparatus control method comprising: a deduplication control unit that eliminates storage of duplicate file data; and a search interface unit that acquires file data. A chunk data generation step of generating a chunk data sequence, a pseudo random number generation step of calculating a pseudo random number based on each chunk data in the chunk data sequence, and the pseudo random number as a search term A storage device control method including a similar search step for extracting all index terms satisfying a criterion is provided.
Note that this method can be implemented using hardware called a computer that constitutes the storage apparatus described above.

According to a third aspect of the present invention, there is provided a program that is executed by a computer controlled by a storage device that includes a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data. The chunk data generation process for dividing the file data to generate a chunk data string, the pseudo random number calculation process for calculating a pseudo random number based on each chunk data in the chunk data string, and the pseudo random number A program that executes a similar search process for extracting all index terms that satisfy the similarity criterion as a search term is provided.
This program can be recorded on a computer-readable storage medium. The storage medium can be non-transient such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, or the like. The present invention can also be embodied as a computer program product.

  According to each aspect of the present invention, there are provided a storage apparatus, a control method thereof, and a program that can be subjected to similar search even for unknown binary data.

It is a figure for demonstrating one Embodiment of this invention. It is a figure which shows an example of the system configuration | structure which introduced the storage apparatus 1 which concerns on the 1st Embodiment of this invention. 2 is a diagram showing an example of an internal configuration of a storage apparatus 1. 2 is a diagram illustrating an example of an internal configuration of a deduplication control unit 20. FIG. 3 is a diagram illustrating an example of an internal configuration of a similar search control unit 10. FIG. 4 is a diagram illustrating an example of an index word 250. FIG. 3 is a flowchart of data storage processing in the storage apparatus 1. It is a flowchart of the creation process of the index word 250. It is a figure which shows an example of the creation process of the index word. It is a flowchart of a similar search process.

  First, an outline of an embodiment will be described with reference to FIG. Note that the reference numerals of the drawings attached to this summary are attached to the respective elements for convenience as an example for facilitating understanding, and are not intended to limit the present invention to the illustrated embodiment.

  As described above, in order to search for similar file data, prior information regarding the file data may be required. However, there are cases where the format is unknown and prior information about file data cannot be extracted. Therefore, the search function of the storage device limits the types of file data that can be used as search conditions. Therefore, a storage device, a control method thereof, and a program that can be a similar search target even for unknown binary data is desired.

  Accordingly, the storage apparatus 1 shown in FIG. 1 is provided as an example. The storage apparatus 1 shown in FIG. 1 acquires a deduplication control unit that eliminates the storage of duplicate file data, an index word storage unit that stores pseudo random numbers based on the storage target file data as index terms, and search target file data A search interface part, a search chunk data generation part that divides search target file data to generate a search chunk data string, and a search pseudorandom number that calculates a pseudorandom number based on each chunk data in the search chunk data string And a similarity search unit that extracts all index words satisfying the similarity criterion using pseudo-random numbers as search words.

  First, the storage apparatus 1 includes a deduplication control unit 20. The deduplication control unit 20 has two functions. The first function is to perform processing for eliminating storage of duplicate file data. The second function is to perform processing for generating an index word for search processing.

  The storage device 1 also includes a search interface unit 11. The search interface unit 11 acquires search target file data. Here, the search target file data is file data designated as a search condition from the outside.

  Furthermore, the storage device 1 includes a search chunk data generation unit 12. The search chunk data generation unit 12 divides the file data into a plurality of block data (hereinafter referred to as chunk data). Chunk data is a chunk of data after division. Therefore, the search chunk data generation unit 12 can generate a chunk data string of file data.

  The storage device 1 includes a search pseudorandom number unit 13. The search pseudorandom number unit 13 calculates pseudorandom numbers of chunk data of file data by a pseudorandom number generation method.

  A pseudo-random number is a number sequence that looks like a random number but is actually obtained by deterministic calculation. A true random number is a sequence of numbers that is neither regular nor reproducible. On the other hand, the pseudo-random number is a reproducible number sequence created by calculation. The pseudo random number generation method means an algorithm for generating pseudo random numbers.

  Here, the storage apparatus 1 uses the same pseudo-random number generation method for all file data. There are various pseudo-random number generation methods such as SHA1 (Secure Hash Algorithm 1) and MD5 (Message Digest 5). However, it is preferable to select an algorithm that is less likely to cause pseudorandom collisions.

  Furthermore, the storage device 1 includes a similarity search unit 14. The similarity search unit 14 uses a pseudo-random number generated by the search pseudo-random number unit 13 as a search term. This pseudo random number does not depend on prior information regarding file data.

  Next, the storage device 1 includes an index word storage unit 30. The index word storage unit 30 stores index words. Here, the index word includes a pseudo-random number based on the storage target file data.

  Here, the deduplication control unit 20 registers a pseudo random number based on the file data as an index word. First, the deduplication control unit 20 generates chunk data of storage target file data by the same method as the search chunk data generation unit 12. Then, the deduplication control unit 20 calculates a pseudo random number by the same pseudo random number generation method as the search pseudo random number unit 13. Note that the pseudo-random numbers generated by the deduplication control unit 20 also do not depend on prior information regarding file data.

  And the similarity search part 14 collates the pseudorandom numbers contained in a search word, and the pseudorandom numbers contained in an index word, and extracts all the index words satisfying the similarity criterion.

  As described above, the storage apparatus 1 can acquire a characteristic amount unique to data by using a pseudo random number. In addition, the storage apparatus 1 can determine whether or not a part of the data matches by comparing the pseudo random numbers of the chunk data.

  Therefore, it is possible to provide a storage apparatus, a control method thereof, and a program that can be a similar search target even for unknown binary data.

  Hereinafter, specific embodiments will be described in more detail with reference to the drawings.

[First Embodiment]
The first embodiment of the present invention will be described in more detail with reference to the drawings.

  FIG. 2 is an example of a system configuration in which the storage apparatus 1 according to the present embodiment is introduced.

  A system in which the storage apparatus 1 is installed includes a storage apparatus 1, a data storage client 2, and a search client 3. The storage device 1, the data storage client 2, and the search client 3 are connected via a network. The network is realized by using an industry standard protocol such as NFS (Network File System) or CIFS (Common Internet File System). The protocol method is not limited.

  Here, the data storage client 2 issues a data read / write request to the storage apparatus 1. In other words, the data storage client 2 issues an access request for data stored in the storage device 1.

  In addition, the search client 3 issues a file search request to the storage device 1. Then, the search client 3 transfers file data to the storage device 1 as a search condition. Thereafter, the storage device 1 executes a search and returns the search result to the search client 3. Here, the search result is a list of file paths similar to the search condition file data.

  Note that the data storage client 2 and the search client 3 may each be composed of a plurality of machines. However, in the following description, for the sake of simplicity, the data storage client 2 and the search client 3 will be described as being configured from a single machine.

  Next, FIG. 3 is a diagram illustrating an example of the internal configuration of the storage apparatus 1.

  The storage device 1 includes a similar search control unit 10, a search interface unit 11, a deduplication control unit 20, an index word storage unit 30, a data storage interface unit 40, a file system 50, and a data storage unit 60. FIG. 3 shows only modules related to the storage apparatus 1 according to this embodiment for the sake of simplicity.

  First, the data storage interface unit 40 interacts with the data storage client 2. That is, the data storage interface unit 40 exchanges requests issued by the data storage client 2 and transmits responses. The data storage interface unit 40 transmits / receives a request to / from the data storage client 2 based on a remote file access protocol such as NFS or CIFS. As described above, the protocol method is not limited.

  Next, the file system 50 performs file management in the storage apparatus 1. First, the file system 50 controls file metadata. Here, the file metadata refers to additional data with a high level of abstraction for certain data. For example, the metadata includes data creation date and time, creator, data format, access information, and the like. The metadata includes the address of the data storage destination in the data storage unit 60. In addition, the file system 50 manages data storage in the data storage unit 60. Here, the data to be stored and managed is file data and its metadata.

  The data storage unit 60 includes a storage medium. Here, the data storage unit 60 responds to a read / write request from the file system 50. Then, the data storage unit 60 stores the data requested from the file system 50 in the storage medium. The data storage unit 60 and the index word storage unit 30 may share the same storage medium.

  In addition, the search interface unit 11 performs a dialog with the search client 3. That is, the search interface unit 11 exchanges requests issued by the search client 3 and returns search results. The search interface unit 11 transmits / receives a request to / from the search client 3 based on a remote file access protocol such as NFS or CIFS. As described above, the protocol method is not limited.

  Further, as described above, the index word storage unit 30 stores index words. Here, the index word storage unit 30 includes a storage medium. For example, the storage medium is preferably an HDD (Hard Disk Drive). Then, the index word storage unit 30 stores the index word generated by the deduplication control unit 20 and the data for duplication check in a storage medium.

  Next, FIG. 4 is a diagram illustrating an example of an internal configuration of the deduplication control unit 20.

  The deduplication control unit 20 includes a storage processing control unit 21, a storage chunk data generation unit 22, and a storage pseudo random number unit 23.

  First, the storage processing control unit 21 checks whether or not the storage target file data is duplicated. Here, the storage target file data refers to file data for which a write request has been made to the file system 50. In addition, the storage processing control unit 21 generates and updates the index word 250.

  Next, the storage chunk data generation unit 22 performs the same processing as the search chunk data generation unit 12. Therefore, the storage chunk data generation unit 22 can generate the chunk data string 210 of the storage target file data 200.

  The stored pseudorandom number unit 23 performs the same processing as the search pseudorandom number unit 13. Therefore, the stored pseudorandom number unit 23 can generate the pseudorandom number 240 from the chunk data sequence 210.

  Next, FIG. 5 is a diagram illustrating an example of an internal configuration of the similarity search control unit 10.

  The similar search control unit 10 includes a search chunk data generation unit 12, a search pseudorandom number unit 13, a similarity search unit 14, and a search word control unit 15. The search chunk data generation unit 12 and the search pseudorandom number unit 13 are as described above. Therefore, further explanation is omitted.

  As described above, the similarity search unit 14 collates the pseudo random number included in the search word with the pseudo random number included in the index word. Then, all index terms 250 that satisfy the similarity criterion are extracted. Here, the similarity search unit 14 refers to the index word 250 from the index word storage unit 30. Then, the similarity search unit 14 acquires a list of file identifiers 203 associated with the index word 250. Next, the similarity search unit 14 acquires a list of file paths based on the list of file identifiers 203. The file system 50 manages the file path. Therefore, the similarity search unit 14 acquires a list of file paths via the file system 50. Thereafter, the similarity search unit 14 returns the acquired file path list to the search client 3 via the search interface unit 11.

  The search word control unit 15 has two functions. The first function is a function for generating a search term based on the pseudorandom number 140 generated by the search pseudorandom number unit 13. The second function is a function that mediates data transfer within the similar search control unit 10.

  Next, the operation of the storage apparatus 1 will be described.

  First, the data structure of the index word 250 stored in the storage device 1 will be described.

  Here, FIG. 6 is a diagram illustrating an example of the index word 250. The index word 250 on the first line in FIG. 6 associates the pseudo random number 1 with two file identifiers. Also, the index word 250 in the second row in FIG. 6 associates the pseudo random number 2 with one file identifier. As described above, the index word 250 exists for all the file data stored in the storage device 1. Each index word 250 has a structure that associates a pseudo-random number with a file identifier.

  Next, file data storage processing in the storage apparatus 1 and index word generation processing will be described using the flowchart of FIG.

  First, the data storage client 2 issues a file data write request 202 to the storage apparatus 1. Here, the write request 202 includes storage target file data 200 and a file identifier 203 indicating the storage destination. Then, the data storage interface unit 40 acquires the storage target file data 200 and the file identifier 203 (step S110).

  Next, the data storage interface unit 40 transfers the write request 202 to the storage processing control unit 21 via the file system 50. Then, the storage processing control unit 21 transfers the storage target file data 200 to the storage chunk data generation unit 22. Thereafter, the storage chunk data generation unit 22 divides the storage target file data 200 (step S120). In the following description, the data (chunk data) sequence divided in step S120 will be described as the chunk data sequence 210. After the process of step S120, a number corresponding to the rank is assigned to each chunk data 230.

  Here, the storage chunk data generation unit 22 preferably employs a plurality of file data division methods. For example, it is also preferable to analyze file data and select a data division method. However, in the storage apparatus 1 according to the present embodiment, a method of dividing all file data with a fixed size will be described. A configuration employing a plurality of division methods will be described as the storage apparatus 1 according to the second embodiment.

  Next, the storage chunk data generation unit 22 transfers the respective chunk data 230 to the storage pseudorandom number unit 23 via the storage processing control unit 21. Thereafter, the stored pseudorandom number unit 23 calculates a pseudorandom number 240 for each chunk data 230 (step S130).

  Then, the stored pseudorandom number unit 23 transfers the pseudorandom number 240 to the storage processing control unit 21. Thereafter, the storage processing control unit 21 checks whether or not file data that overlaps with the storage target file data 200 is stored in the storage device 1 (step S140). In this case, the storage processing control unit 21 confirms whether or not the index word 250 including the pseudo random number 240 calculated in step S <b> 130 exists in the index word storage unit 30.

  Furthermore, the process of step S140 is demonstrated in detail using the flowchart of FIG.

  First, the storage processing control unit 21 refers to the index words 250 stored in the index word storage unit 30 in order. Then, the storage processing control unit 21 determines whether or not the index word 250 including the pseudo random number that matches the pseudo random number 240 is stored in the index word storage unit 30 (step S210).

  When the condition of step S210 is satisfied, the storage processing control unit 21 acquires the storage destination address 201 associated with the index word 250 that matches the pseudorandom number. In addition, the storage processing control unit 21 adds the file identifier 203 included in the write request 202 to the index word 250 matched with the pseudorandom number (step S220). Further, the storage processing control unit 21 transfers the chunk data 230 that is the calculation source of the pseudo random number 240 to the file system 50. In addition, the storage processing control unit 21 transfers a number corresponding to the rank of the chunk data 230 to the file system 50. Further, the storage processing control unit 21 transfers the pseudo random number 240 to the file system 50. In addition, the storage processing control unit 21 transfers the storage destination address 201 to the file system 50. Then, the storage processing control unit 21 notifies the file system 50 that the chunk data 230 from which the pseudo random number 240 is calculated is duplicated. Thereafter, the file system 50 updates the metadata managed by the file system 50 based on the notification from the storage processing control unit 21 (step S230).

  On the other hand, when the condition of step S210 is not satisfied, the storage processing control unit 21 newly creates an index word 250 (step S240). In this case, the storage processing control unit 21 associates the pseudo-random number 240 and the file identifier 203 included in the write request 202 with the index word 250 to be created. Then, the storage process control unit 21 stores the created index word 250 in the index word storage unit 30. Thereafter, the storage processing control unit 21 transfers the chunk data 23 from which the pseudo random number 240 is calculated to the file system 50. In addition, the storage processing control unit 21 transfers the number corresponding to the rank of the chunk data to the file system 50. Further, the storage processing control unit 21 transfers the pseudo random number 240 to the file system 50. In addition, the storage processing control unit 21 transfers the file identifier 203 to the file system 50. Then, the storage processing unit 21 notifies the file system 50 that the chunk data 230 from which the pseudo random number 240 is calculated is not duplicated.

  Next, the file system 50 secures a storage area for the chunk data 230 in the data storage unit 60 based on the notification from the storage processing unit 21. Thereafter, the file system 50 stores the chunk data 230 in the data storage unit 60 (step S250). Then, the file system 50 updates the file metadata (step S260). Note that the metadata to be updated includes the storage destination address 201 of the chunk data 230.

  Thereafter, the file system 50 transfers the pseudo random number 240 to the storage processing control unit 21. In addition, the file system 50 transfers the storage destination address 201 of the chunk data 230 to the storage processing control unit 21. Then, the storage process control unit 21 adds the received storage destination address 201 to the index word 250 created in the process of step S230 (step S270).

  The deduplication control unit 20 performs the above processing for all the write requests 202. Thereafter, the file system 50 notifies the data storage client 2 of the end of processing via the data storage interface unit 40.

  Here, an example of index word creation processing is shown using FIG.

  First, the storage apparatus 1 acquires a write request 202 (step S110). Here, the write request 202 includes storage target file data 200 and a file identifier 203. In the case of FIG. 9, the file identifier 203 indicates file_id3. Then, the storage device 1 generates a chunk data string 210 (step S120). In the case of FIG. 9, the chunk data sequence 210 has two chunk data 230. After the process of step S120, the storage apparatus 1 calculates a pseudo random number sequence 220 based on the chunk data sequence 210 (step S130). In the case of FIG. 9, the pseudo random number sequence 220 includes two pseudo random numbers D240p and D240r. Here, in the case of FIG. 9, the index word storage unit 30 stores an index word 250p and an index word 250q. The index word 250p includes a pseudo random number D240p. Furthermore, the index word 250p associates the file identifiers file_id1 and file_id2 with the pseudorandom number 240p. On the other hand, the index word 250q includes a pseudo random number D240q. Further, the index word 250q associates the file identifier file_id4 with the pseudorandom number D240q.

  Here, the pseudo random number D240p in the pseudo random number sequence 220p also exists in the pseudo random number sequence 250p. Therefore, the condition of step S210 is satisfied. Therefore, the process proceeds to step S220. As the processing in step S220, the storage apparatus 1 adds file_id3 to the index word 250p.

  As described above, the storage apparatus 1 according to this embodiment can eliminate the storage of duplicate data. In addition, the storage apparatus 1 uses a part of the data managed by the storage apparatus 1 at the time of index word registration. Therefore, in the storage device 1 according to the present embodiment, it is possible to suppress consumption of storage resources for storing index words.

  Next, the similarity search process will be described with reference to the flowchart of FIG.

  In the storage apparatus 1 according to the present embodiment, the search condition is file data. The search result is a list of path names in which files similar to the search condition are stored.

  First, the search client 3 transfers the search file data 150 to the storage device 1 as a search condition. At that time, the storage apparatus 1 acquires the search file data 150 via the search interface unit 11 (step S310). Then, the search interface unit 11 transfers the search file data 150 to the similarity search unit 14. Thereafter, the similarity search unit 14 transfers the search file data 150 to the search word control unit 15.

  Next, the search word control unit 15 transfers the search file data 150 to the search chunk data generation unit 12. Then, the search chunk data generation unit 12 performs the same process as the storage chunk data generation unit. Then, the search chunk data generation unit 12 generates a chunk data string 110 (step S320). However, the search chunk data generation unit 12 adopts the same chunk data generation method as the storage chunk data generation unit 22.

  Next, the search chunk data generation unit 12 transfers the chunk data sequence 110 to the search pseudorandom number unit 13 via the search word control unit 15. Thereafter, the search pseudorandom number unit 13 performs the same process as the index pseudorandom number unit 23. Then, the search pseudorandom number unit 13 calculates a pseudorandom number 140 for each chunk data 130 in the chunk data sequence 110 (step S330). However, the search pseudorandom number unit 13 adopts the same pseudorandom number generation method as the stored pseudorandom number unit 23.

  Next, the search pseudorandom number unit 13 transfers all the pseudorandom numbers 140 to the similarity search unit 14 via the search word control unit 14. Then, the similarity search unit 14 determines whether or not the index word 250 including the pseudo random number 140 exists in the index word storage unit 30 (step S340). When the condition of step S340 is satisfied, the similarity search unit 14 acquires a list of file identifiers 203 associated with the matched index word 250 (step S350). And the similarity search part 14 judges the conditions of step S340 about all the pseudorandom numbers 140. FIG.

  Thereafter, it is assumed that the similarity search unit 14 determines the condition of step S340 for all the pseudorandom numbers 140. In this case, the similarity search unit 14 acquires a file path from the file system 50 for each acquired file identifier. Then, the similarity search unit 14 transfers the acquired file path list to the search interface unit 11 as the similarity search result 160. Finally, the search interface unit 11 returns the similar search result 160 to the search client 3 (step S360).

  As described above, the first effect of the storage apparatus 1 according to the present embodiment is that a similarity search can be realized for stored file data without depending on the type of file data. Further, in the storage device 1 according to the present embodiment, prior information regarding file data is not required at the time of similarity search. Therefore, the storage apparatus 1 according to the present embodiment can provide a storage apparatus, a control method thereof, and a program that can be subjected to similarity search even for unknown binary data.

  A second effect of the storage apparatus 1 according to the present embodiment is that consumption of storage resources for storing index words can be suppressed. This is because the storage apparatus 1 according to the present embodiment registers a part of the storage management data as an index word. Therefore, the storage apparatus having the first effect can be provided with a small amount of memory resources.

[Second Embodiment]
Next, the second embodiment will be described in detail.

  The present embodiment is an embodiment that includes a plurality of chunk data generation methods and selects an appropriate chunk data generation method. In the description of the present embodiment, the description of the same parts as those in the first embodiment is omitted. Further, in the description of the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  As described above, in the present invention, pseudo-random numbers based on file data are collated to realize a similarity search. However, the same pseudo-random number may be calculated even for dissimilar file data. That is, pseudorandom numbers may collide. This pseudo-random number collision occurs when the chunk data is similar. Therefore, in the storage apparatus 1 according to the present embodiment, the chunk data generation method is changed according to the file data. Thereby, the possibility that the same pseudorandom number is calculated can be reduced.

  First, a case where the data format is known will be described. If the data format is known, chunk data is generated based on the data format structure. For example, consider data written in a markup language using tags. In that case, chunk data can be generated based on each tag.

  Next, a case where the data format is unknown will be described. When the data format is unknown, the first embodiment employs a fixed size division method. However, in this embodiment, in addition to the division method with a fixed size, a method for determining an appropriate division size (a division method with a variable length size) is also employed. For example, as a variable length size dividing method, there is a method of detecting a data break accompanying data insertion into file data. However, the storage apparatus 1 according to the present embodiment uses only one of a fixed size division method and a variable length size division method.

  As described above, in the storage apparatus 1 according to the present embodiment, the chunk data generation method is changed based on the analysis inside the file data. As a result, it is less likely that the same pseudorandom number is calculated for dissimilar file data. Therefore, the storage apparatus 1 according to the present embodiment improves the accuracy of the similarity search result compared to the storage apparatus 1 according to the first embodiment.

[Third Embodiment]
Next, the third embodiment will be described in detail.

  This embodiment is an embodiment that sorts and outputs similar search results based on the degree of similarity. In the description of the present embodiment, the description of the same parts as those in the first embodiment is omitted. Further, in the description of the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  As described above, the similarity search unit 14 determines whether or not the pseudo random number 140 matches the pseudo random number 240 stored in the index word storage unit 30 (step S340). For all the pseudo random numbers 140 in the pseudo random number sequence 120, the similarity search unit 14 determines the condition of step S340. Here, the more similar file data, the more pseudo-random numbers of chunk data match. That is, the number that the pseudo random number 140 and the pseudo random number 240 match represents the similarity of the file data. Therefore, in order to present the similarity of file data to the user, it is preferable to sort the corresponding file identifiers based on the number of matching pseudo-random numbers 140 and pseudo-random numbers 240.

  As described above, the storage apparatus 1 according to the present embodiment sorts and outputs the similarity search results based on the similarity. As a result, the user can grasp the similarity between the file data as the search condition and the file data of the search result. Therefore, the storage apparatus 1 according to the present embodiment can provide a storage apparatus that is more useful than the other embodiments.

  A part or all of the above embodiments can be described as in the following supplementary notes, but is not limited thereto.

  (Additional remark 1) Deduplication control part which excludes storage of the duplicate file data, Index word storage part which stores the pseudorandom number based on storage object file data as an index word, Search interface part which acquires search object file data A search chunk data generation unit that divides the search target file data to generate a search chunk data sequence, a search pseudo random number unit that calculates a pseudo random number based on each chunk data in the search chunk data sequence, and A storage device comprising: a similarity search unit that extracts all index terms that satisfy a similarity criterion using a pseudo-random number as a search term.

  (Additional remark 2) The said deduplication control part divides | stores storage object file data, a storage chunk data generation part which produces | generates a storage chunk data sequence, and a pseudorandom number based on each chunk data in the said storage chunk data sequence A storage apparatus comprising: a storage pseudo-random number unit that calculates an index word; and a storage processing control unit that generates an index word based on the pseudo-random number calculated by the storage pseudo-random number unit.

  (Supplementary Note 3) The storage apparatus, wherein the similarity criterion includes at least one or more pseudorandom numbers included in an index word in a pseudorandom number sequence based on the search chunk data sequence.

  (Additional remark 4) The said similar search part is a storage apparatus which sorts the index word which satisfy | filled the said similarity criteria.

  (Additional remark 5) The said similar search part is a storage apparatus which sorts an index word based on the number of the pseudorandom numbers which satisfy | filled the said similarity criterion.

  (Supplementary Note 6) A storage device that calculates a hash value as a pseudo-random number.

  (Supplementary note 7) A storage device that divides file data in a fixed size.

  (Supplementary Note 8) A storage apparatus that selects one data division method from two or more data division methods.

  (Supplementary note 9) A storage apparatus control method comprising a deduplication control unit for eliminating storage of duplicate file data and a search interface unit for acquiring file data, wherein the file data is divided and chunk data A chunk data generation step for generating a sequence, a pseudo random number generation step for calculating a pseudo random number based on each chunk data in the chunk data sequence, and all index terms satisfying a similarity criterion using the pseudo random number as a search term And a similar search step for extracting the storage device.

  (Supplementary Note 10) A storage apparatus control method including a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data, and generates an index word based on a pseudo-random number A method for controlling a storage apparatus, including an index word generation step.

  (Supplementary Note 11) A storage apparatus control method comprising a deduplication control unit that eliminates storage of duplicate file data, and a search interface unit that acquires file data, the pseudo random number based on the search chunk data string Sorting index words satisfying the similarity criterion that at least one or more pseudo-random numbers included in the index word are included in the column.

  (Supplementary Note 12) A storage apparatus control method comprising a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data, the step of dividing file data by a fixed size, A storage apparatus control method including:

  (Supplementary note 13) A storage apparatus control method comprising a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data, wherein one data from two or more data division methods A method for controlling a storage apparatus, comprising: selecting a division method.

  (Supplementary Note 14) A program that causes a computer that controls a storage apparatus to include a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data, the file data being divided A chunk data generation process for generating a chunk data string, a pseudo random number calculation process for calculating a pseudo random number based on each chunk data in the chunk data string, and using the pseudo random number as a search word, A similar search process for extracting all index words that satisfy the program.

  (Supplementary note 15) A program to be executed by a computer that controls a storage device that includes a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data. A program that executes index word generation processing for generating an index word.

  (Supplementary Note 16) A program that is executed by a computer that controls a storage apparatus that includes a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data, the search chunk data string A program for sorting index words satisfying a similarity criterion that includes at least one or more pseudo random numbers included in an index word in a sequence of pseudo random numbers based on.

  (Supplementary Note 17) A program that is executed by a computer that controls a storage apparatus that includes a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data, the file data having a fixed size The program that executes the process of dividing.

  (Supplementary note 18) A program that is executed by a computer that controls a storage apparatus that includes a deduplication control unit that eliminates storage of duplicate file data and a search interface unit that acquires file data. The program which performs the process which selects the one data division method from the method.

  The disclosure of the cited patent document is incorporated herein by reference. Within the scope of the entire disclosure (including claims) of the present invention, the embodiments and examples can be changed and adjusted based on the basic technical concept. Various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) within the scope of the claims of the present invention, Selection is possible. That is, the present invention of course includes various variations and modifications that could be made by those skilled in the art according to the entire disclosure including the claims and the technical idea. In particular, with respect to the numerical ranges described in this document, any numerical value or small range included in the range should be construed as being specifically described even if there is no specific description.

DESCRIPTION OF SYMBOLS 1 Storage apparatus 2 Data storage client 3 Search client 10 Similar search control part 11 Search interface part 12 Search chunk data generation part 13 Search pseudorandom number part 14 Similar search part 15 Search word control part 20 Deduplication control part 21 Storage process control part 22 Stored chunk data generation unit 23 Stored pseudo random number unit 30 Index word storage unit 40 Data storage interface unit 50 File system 60 Data storage unit 110, 210 Chunk data sequence 120, 220 Pseudo random number sequence 130, 230 Chunk data 140, 240 Pseudo random number 150 Search file data 160 Similar search result 200 Storage target file data 201 Storage destination address 202 Write request 203 File identifier 250 Index word

Claims (10)

A deduplication controller that eliminates the storage of duplicate file data;
An index word storage unit that stores pseudo random numbers based on file data to be stored as index words;
A search interface unit for acquiring search target file data;
A search chunk data generation unit that divides the search target file data and generates a search chunk data string;
A search pseudo-random number part for calculating a pseudo-random number based on each chunk data in the search chunk data string;
Using the pseudo-random number as a search term, a similarity search unit that extracts all index terms that satisfy a similarity criterion;
A storage apparatus comprising: The deduplication control unit
A storage chunk data generation unit that divides storage target file data and generates a storage chunk data string;
A stored pseudo-random number part for calculating a pseudo-random number based on each chunk data in the stored chunk data string;
A storage processing control unit that generates an index word based on the pseudo-random number calculated by the storage pseudo-random number unit;
The storage apparatus according to claim 1. The similarity criterion is
Including at least one or more pseudo-random numbers included in an index word in a pseudo-random number sequence based on the search chunk data sequence;
The storage apparatus according to claim 1 or 2. The similarity search unit includes:
Sorting index terms that meet the similarity criteria;
The storage device according to any one of claims 1 to 3. The similarity search unit includes:
Sorting the index terms based on the number of pseudo-random numbers that satisfy the similarity criterion;
The storage apparatus according to any one of claims 1 to 4. Calculating a hash value as a pseudo-random number,
The storage device according to any one of claims 1 to 5. Divide file data with a fixed size,
The storage apparatus according to any one of claims 1 to 6. Select one data division method from two or more data division methods.
The storage device according to any one of claims 1 to 7. A deduplication controller that eliminates the storage of duplicate file data;
A search interface unit for acquiring file data;
A storage apparatus control method comprising:
A chunk data generation step of dividing the file data to generate a chunk data string;
A pseudo random number generating step of calculating a pseudo random number based on each chunk data in the chunk data sequence;
Using the pseudo-random number as a search term, a similar search step for extracting all index terms that satisfy a similarity criterion;
A method for controlling a storage apparatus. A deduplication controller that eliminates the storage of duplicate file data;
A search interface unit for acquiring file data;
A program to be executed by a computer controlled by a storage device comprising:
Chunk data generation processing for dividing the file data and generating a chunk data string;
A pseudo-random number calculation process for calculating a pseudo-random number based on each chunk data in the chunk data sequence;
Using the pseudo-random number as a search term, a similar search process for extracting all index terms that satisfy a similarity criterion;
A program that executes.

Images