JP2015176393A - Storage device, storage system, control method for storage system, and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a storage system capable of increasing the efficiency of data synchronization between a storage device and terminal equipment.SOLUTION: A storage system includes: a data verification means for acquiring the meta data of user data from terminal equipment, and for searching external data having the features of the acquired meta data of the user data, and for comparing the featured values of the searched external data with the featured values of the user data to determine consistency/inconsistency between the user data and the external data, and for acquiring the substantial data of the user data or the external data on the basis of the determination result of the consistency/inconsistency between the user data and the external data; and a data storage means for storing the substantial data of the user data or the external data acquired by the data verification means.

Description

  The present invention relates to a storage apparatus, a storage system, a storage system control method, and a control program. In particular, the present invention relates to a storage apparatus, a storage system, a storage system control method, and a control program that can be applied to an online storage service such as a cloud storage service.

  There is a cloud storage service as one of services used for transferring and synchronizing data stored in personal devices to a storage service on the cloud, and for backup, data disclosure, and sharing between groups. The cloud storage service is realized by detecting a difference between a file in the device and a file on the cloud and transferring the detected difference. However, in many cases, the cloud storage service is performed via a communication means provided in a device owned by an individual and via a communication means that is not high-speed and large-capacity. In particular, in the case of portable devices, the tendency to go through communication means that is not high-speed and large-capacity is remarkable.

  Further, the data stored in the cloud storage includes not only unique data such as images taken by individuals but also data obtained from public information sources such as the web. Information data obtained from a publicly available information source is once collected by an individual and then transferred to a cloud storage service. Therefore, data already existing on the network is duplicated and stored in the cloud storage.

  In order to improve the efficiency of duplicate data in the cloud storage, there is a method of storing a position specifier such as a unified resource position specifier (hereinafter referred to as URL) in a format such as a bookmark instead of a data entity (URL: Uniform Resource Locators).

  Patent Document 1 discloses a backup server that can efficiently use radio resources when performing synchronization processing of user data and update processing of user data with regard to the above-described efficiency of duplicate data.

  In the backup server of Patent Document 1, update user data that overwrites and updates user data is acquired based on acquisition-related information stored in advance, and the stored user data is updated with the update user data. On the other hand, synchronization processing is performed so as to ensure consistency between user data stored in the mobile device and user data stored in advance. The synchronization process is configured to be performed using the updated user data.

  The acquisition related information stored in the backup server of Patent Document 1 is preferably acquisition destination information indicating a storage destination of updated user data. For example, it is disclosed that link destination information such as a URL can be used as acquisition destination information. Has been.

  Further, in a situation where widely spread public information is stored in the cloud storage, a situation occurs in which many users store the same information in a short time. Such a situation can be solved by a deduplication technique using deduplication storage. Deduplication technology has already been put to practical use in a closed environment such as in a corporate network.

  However, with current storage services, there is an increasing tendency that storage service providers are not allowed to refer to the contents of personal data. In an environment where data is stored by a third party such as the cloud, it is necessary to encrypt personal data, making it difficult to achieve compatibility with deduplication technology, making it difficult to make deduplication means function effectively. There was a problem that it was going.

  Patent Document 2 discloses a file storage system that can be used for an online storage server that targets an unspecified number of people and can prevent impersonation and unauthorized access while being able to upload a huge file quickly. Is disclosed.

  The server device of the file storage system of Patent Literature 2 verifies that the user has access rights to the file at the terminal device for the file searched with the search key received from the terminal device. When the verification result is affirmative, the access right of the server device is given to the user of the terminal device.

  In addition, distributed storage services that can store the same data in a plurality of storage services are increasing.

JP 2009-129195 A Japanese Patent No. 4376960

  Generally, in a storage device that backs up user data stored in a terminal device, the user data is acquired from the terminal device via a communication means provided in the terminal device. The capacity of communication means of a general terminal device is not sufficiently high speed and large capacity with respect to the data capacity transmitted from the terminal device to the storage device. Therefore, there is a problem that the efficiency of data synchronization between the terminal device and the storage device is limited by the capacity of the communication means of the terminal device.

  In the backup server of Patent Document 1, there is a problem that if the information included in the specified URL becomes invisible for some reason, the user data stored in the backup server cannot be updated. On the other hand, assuming that the information contained in the URL becomes invisible, there is also a technique that uses a clipping function of a Web site in order to save the information contained in the URL for each data entity. However, when using the clipping function, there is a problem that a plug-in is required for clipping in the browser, etc., and it is difficult to make efficient saving in units of individual files that can be obtained by methods other than the web browser There was a problem.

  According to the file storage system of Patent Document 2, impersonation and unauthorized access can be prevented even when using a technique for sending a file search key instead of uploading a file entity. In other words, according to the file storage system of Patent Document 2, it is possible to improve security with respect to access to the file stored in the server device. However, in the file storage system of Patent Document 2, when backing up the personal file of the terminal device in synchronization with the cloud, it is still necessary to communicate between the server device and the terminal device.

  In a distributed storage service that can use a plurality of storage services in parallel, the plurality of storage services are independent of each other. Therefore, there is a problem that it is difficult to improve the efficiency of client-initiated transfer except in the case where a data transfer service is specially prepared between a plurality of storage services.

  In order to solve the above-described problems, an object of the present invention is to provide a storage system that improves the efficiency of data synchronization between a storage device and a terminal device.

  The storage device of the present invention acquires user data metadata from a terminal device, searches for external data having the characteristics of the acquired user data metadata, and compares the searched external data and the feature amount of the user data. Data verification means for determining whether the user data and external data match or not, and obtaining the user data entity data or external data based on the determination result of the match or mismatch between the user data and external data, and data verification Data storage means for storing actual data of user data acquired by the means or external data.

  The storage system of the present invention acquires terminal data storing user data, user data metadata from the terminal apparatus, searches external data having the characteristics of the acquired user data metadata, and searches the external data The user data and the external data are compared to determine whether the user data and external data match, and based on the determination result of the user data and external data match / mismatch, the user data entity data or external data is determined. A storage device comprising: a data verification unit to acquire; and a data storage unit to store actual data of user data or external data acquired by the data verification unit.

  The storage system control method of the present invention acquires user data metadata from a terminal device, searches external data having the characteristics of the acquired user data metadata, and searches the external data and the feature amount of the user data. The user data and the external data are compared to determine whether the user data and the external data match, and based on the determination result of the user data and the external data match or mismatch, the user data entity data or the external data is obtained. Entity data or external data is stored in the storage device.

  The storage system control program of the present invention includes a process for acquiring user data metadata from a terminal device, a process for searching external data having the characteristics of the acquired user data metadata, and the searched external data and user The process of determining the match / mismatch between the user data and the external data by comparing the feature quantity of the data, and the actual data or the external data of the user data based on the match / mismatch determination result between the user data and the external data The computer is caused to execute the process of acquiring and the process of storing the actual data of user data or external data in the storage device.

  According to the present invention, instead of the communication means provided in the terminal device, it is possible to use the high-speed / broadband communication means provided in the storage server, thereby improving the efficiency of data synchronization between the storage device and the terminal device. Storage system can be provided.

1 is a block diagram showing a configuration of a storage system according to an outline of an embodiment of the present invention. It is a block diagram which shows an example of the hardware constitutions which implement | achieve the storage server which concerns on the outline | summary of embodiment of this invention. 1 is a block diagram showing a configuration of a storage system according to a first embodiment of the present invention. 4 is a flowchart for explaining an operation of a client of the storage system according to the first embodiment of the present invention. 3 is a flowchart for explaining the operation of the storage server of the storage system according to the first embodiment of the present invention. It is a block diagram which shows the structure of the storage system which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the storage system which concerns on the 3rd Embodiment of this invention. It is a block diagram which shows the structure of the storage system which concerns on the 4th Embodiment of this invention.

  EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated using drawing. However, the preferred embodiments described below are technically preferable for carrying out the present invention, but the scope of the invention is not limited to the following. In the following description, the same or similar components / parts may be denoted by the same reference numerals, and redundant description may be omitted.

(Overview)
First, the storage system 100 according to the outline of the embodiment of the present invention will be described. Note that the storage system 100 in FIG. 1 shows an outline of an embodiment of the present invention. Specific configurations, operations, and the like for realizing the present invention will be described in first to fourth embodiments to be described later.

  FIG. 1 is a block diagram showing a functional configuration of a storage system 100 according to the outline of the embodiment of the present invention. The storage system 100 includes a terminal device 10 owned by a user and a storage device 30 that can exchange data with the terminal device 10.

  First, as illustrated in FIG. 1, the storage apparatus 30 includes a data verification unit 400 and a data storage unit 408.

  The data verification unit 400 acquires the metadata of the user data, searches the external data having the characteristics of the acquired metadata, compares the searched external data with the user data, and matches the external data with the user data. Determine discrepancies. Then, the data verification unit 400 acquires the actual data of the user data or the external data based on the determination result of the match / mismatch between the metadata and the feature amount of the external data. Note that the actual data or external data of the user data acquired by the data verification unit 400 corresponds to the synchronization data.

  The data verification unit 400 includes a synchronous data determination unit 401, a data search unit 402, a data comparison unit 406, and an entity data request unit 407 in order to realize the above-described functions.

  The synchronization data determination unit 401 acquires the metadata of the synchronization target file from the terminal device 10 and determines whether the acquired metadata has characteristics of external data accessible by itself. When the synchronization data determination unit 401 determines that the received metadata has the characteristics of external data accessible by itself, the synchronization data determination unit 401 instructs the data search unit 402 to refer to the external data.

  When the external data reference is instructed from the synchronous data determination unit 401, the data search unit 402 takes in the corresponding data from an information source 50 (hereinafter referred to as a data source) such as an external website that can be referred to via a network, for example. Note that the data search unit 402 may use a line having a larger capacity than the line from the terminal device to the storage device, such as a large-capacity line for traders, when acquiring data from an external data source. preferable.

  The data comparison unit 406 compares the external data captured by the data search unit 402 with the user data (file) stored in the file storage unit 208 of the terminal device 10. The data comparison unit 406 determines whether the external data matches the user data based on the comparison result between the user data and the external data. Note that the comparison by the data comparison unit 406 is performed using a feature amount such as a hash value of data. That is, at this time point, the actual data from the terminal device 10 is not transferred.

  The entity data request unit 407 transfers the entity data to the terminal device 10 when the data comparison unit 406 determines that the external data captured by the data search unit 402 does not match the user data stored in the terminal device 10. To request. The entity data requesting unit 407 requests the terminal device 10 to transfer the entity data when external data cannot be acquired by the data search unit 402.

  The data storage unit 408 stores the external data acquired by the data search unit 402 when the data comparison unit 406 determines that the user data matches the external data. The data storage unit 408 stores user data (entity data) transferred from the terminal device 10 by the entity data request unit 407 when external data cannot be acquired by the data search unit 402.

  In addition, the terminal device 10 includes a synchronization control unit 201, a metadata transmission unit 202, an entity data transmission unit 203, a browsing unit 207, and a file storage unit 208.

  The synchronization control unit 201 sequentially synchronizes with the storage apparatus 30 with respect to a synchronization target such as a file stored in the file storage unit 208.

  The metadata transmission unit 202 acquires the synchronization target file from the synchronization control unit 201 and transfers the acquired metadata of the synchronization target file to the storage device 30.

  The entity data transmission unit 203 transmits the entity data stored in the file storage unit 408 when there is a request for the entity data of the file from the storage server 30.

  The browsing means 207 is realized by, for example, a browser including a position specifier such as a unified resource position specifier associated with data, and is a means used for capturing data from the outside such as a network. Hereinafter, the unified resource location specifier is referred to as URL (URL: URL: Uniform Resource Locators). The browsing means 207 stores in the file storage means 208 a file with the data acquired from the network or the like and the data acquisition source added as file attributes.

  The file storage unit 208 stores a file in which the data acquired by the browsing unit 207 and the acquisition source of the data are added as file attributes.

  The above is a schematic description of the storage system 1 according to the outline of the embodiment of the present invention.

  The method according to the outline of the present embodiment can be applied to a system that backs up a personal file obtained by a user in synchronization with the cloud. In the storage system 10 according to the outline of the present embodiment, when the source of the personal file is a public web server, not between the terminal device 10 and the storage device 30 but between the web server and the storage device 30. Exchange entity data. That is, in the present embodiment, the entity data is transferred using a relatively thick transmission path between the storage server 30 and the web server instead of a thin transmission path between the terminal device 10 and the storage apparatus 30. Therefore, the efficiency of data synchronization between the storage device 30 and the terminal device 10 can be increased. Note that the method of the present embodiment can be applied to a case where a file stored in another storage device that can be connected from the storage device 20 is a synchronization target, or a file stored in the terminal device 10 is a data copy from another storage service It can also be applied to cases where it is recognized.

  FIG. 2 is a block diagram showing an example of a hardware configuration for realizing the storage server 30 according to the outline of the embodiment of the present invention. The storage server 30 includes a central processing unit 301, a RAM 302 (Random Access Memory), a ROM 303 (Read Only Memory), an input device 304, an output device 305, a communication module 306, and an auxiliary storage device 307. Each hardware is connected to each other by a bus 308.

  The central processing unit 301 is a CPU (Central Processing Unit) that executes calculation and control of the storage server 30. The RAM 302 and the ROM 303 constitute a main storage device. The input device 304 is an input device such as a keyboard or a mouse, and the output device 305 is an output device such as a display. The communication module 306 is a data transmission / reception device such as a network card. The auxiliary storage device 307 is a storage device such as a hard disk.

  In order to realize each function shown in FIG. 1, first, software is read on hardware such as the central processing unit 301 and the RAM 302 shown in FIG. Under the control of the CPU 301, the input device 304, the output device 305, and the communication module 306 are operated, and data reading / writing in the RAM 302 and the auxiliary storage device 307 is executed. Note that the function shown in FIG. 1 may be realized not only by the hardware shown in FIG. 2 but also by using software loaded on the hardware.

  The outline of the embodiment of the present invention has been described above. Note that the method according to the present embodiment can be applied not only to the cloud storage service but also to a storage service using a storage device connected to an arbitrary terminal device.

  Hereinafter, the storage system according to this embodiment will be described in detail by showing a specific embodiment.

(First embodiment)
(Constitution)
Next, the storage system 1 according to the first embodiment of the present invention will be described with reference to FIG.

  A storage system 1 according to this embodiment includes a client 11 that is a terminal device owned by an individual user, and a storage server that is a storage device for providing a storage service for transferring a file in the client 11 and storing a copy. 31.

(client)
As shown in FIG. 3, the client 11 includes a synchronization control unit 21, a metadata transmission unit 22, an entity data transmission unit 23, a browser 27, and a file 28.

  The synchronization control unit 21 sequentially synchronizes with the storage server 31 for the synchronization target such as the file 28.

  The metadata transmission unit 22 acquires the synchronization target file from the synchronization control unit 21 and transfers the acquired metadata to the storage server 31.

  When there is a request for file entity data from the storage server 31, the entity data transmission unit 23 transmits the entity data.

  The browser 27 stores the data acquired from the network or the like and the data acquisition source as a file 28 added as file attributes.

  The client 11 can be realized as, for example, a personal computer such as a desktop PC, a notebook PC, or a tablet PC, or a terminal device such as a smartphone or a mobile phone (PC: Personal Computer).

(Storage server)
The storage server 31 includes synchronous data determination means 41, external data acquisition means 42, data comparison means 46, actual data request means 47, and data storage area 48.

  The synchronization data determination unit 41 acquires the metadata of the synchronization target file from the client 11 and determines whether or not the acquired metadata has a feature (also referred to as a feature amount) of external data accessible by itself. . When the synchronization data determination unit 41 determines that the data to be synchronized can be obtained from the outside, the synchronization data determination unit 41 transmits the extended attribute information such as the URL to the external data acquisition unit 42 and instructs to acquire the data from the external website 51. To do.

  When the external data acquisition unit 42 is instructed to reference the external data from the synchronous data determination unit 41, the external data acquisition unit 42 takes in the instructed external data from the data source. The data source of the external data is, for example, an external website 51 that can be referred to through a network. The external data acquisition unit 42 is a form of the data search unit 402 of the storage system 30 according to the outline of the present embodiment.

  The data comparison unit 46 compares the external data acquired by the external data acquisition unit 42 with the user data (file 28) stored in the client 11 to determine whether the data matches or does not match. The data comparing means 46 compares the external data and the user data by comparing the feature amounts of the external data and the user data.

  The entity data request unit 47 requests the client 11 to transfer the entity data when the data comparison unit 46 determines that the external data captured by the external data acquisition unit 42 and the user data stored in the client 11 do not match. To do. The entity data request unit 47 requests the client 11 to transfer the entity data when the external data acquisition unit 42 cannot acquire the data.

  The data storage area 48 stores the external data acquired by the external data acquisition unit 42 when the data comparison unit 46 determines that the external data matches the user data. The data storage area 48 stores user data transferred from the client 11 by the entity data request unit 47 when the data comparison unit 46 determines that the external data and the user data do not match. The data storage area 48 stores user data transferred from the client 11 by the entity data requesting unit 47 when data cannot be acquired by the external data acquiring unit 42.

(Operation)
Next, the operation of the storage system 1 according to the first embodiment of the present invention will be described with reference to FIG. 4 and FIG. The configuration of the storage system 1 is as shown in FIG. 3 (client)
First, the operation of the client 11 will be described with reference to FIG.

  In FIG. 4, the browser 27 of the client 11 adds information such as the URL used when downloading the file from the network to the file 28 as an extended attribute of the file 28 (step S11). Note that the technique of step S11 is already used for determining the reliability of a file by the file source.

  When synchronization of the file with the storage service 2 is started, the synchronization control unit 21 identifies one file 28 from the synchronization target file group, and instructs the metadata transmission unit 22 to perform synchronization processing of the identified file 28 (step S12). ).

  The metadata transmission means 22 transmits metadata including a feature quantity such as a hash value of data stored in the file 28 and an extended attribute such as an acquisition source URL stored as a file attribute to the storage server 31 (step). S13).

  Here, the client 11 side waits for processing on the storage server 31 side (step S14). At this time, the synchronization processing of steps S11 to S13 may be sequentially executed with respect to the other file 28 in the stage of step S14.

  When there is a request for entity data from the storage server 31 (Yes in step S14), the transmission means 13 transmits the entity data to the storage server 31 in response to the entity data request from the storage server 31 (step S15).

  If there is no actual data request from the storage server 31 (No in step S14), the flow of FIG. The case where there is no actual data request from the storage server 31 is a case where the storage server 31 acquires the data of the file 28 from the outside. The determination that there is no entity data request from the storage server 31 is made by telling the storage server 31 that the external data has been acquired from the storage server 31 to the client 11, or assuming that there is no entity data request after a specific time has elapsed after step S13. Judgment can be made based on arbitrary criteria.

  This completes the description of the operation of the client 11 of the storage system 1 according to this embodiment.

(Storage server)
Next, the operation of the storage server 31 will be described using the flowchart of FIG. The flowchart in FIG. 5 starts in response to the operation in step S13 in FIG.

  In FIG. 5, when the synchronization data determination unit 41 of the storage server 31 receives the metadata transmitted by the metadata transmission unit 22, it determines whether or not the data having the characteristics of the received metadata can be obtained from the outside. Determination is made (step S21).

When the synchronization data determination unit 41 determines that data having metadata characteristics can be obtained from the outside (Yes in step S21), the synchronization data determination unit 41 notifies the external data acquisition unit 42 of extended attribute information such as a URL, and the external website Data acquisition from 51 is instructed (step S22).
If the instructed data cannot be acquired (No in step S21), the process proceeds to step S26.

  The external data acquisition unit 42 acquires external data in accordance with an instruction from the synchronous data determination unit 41, and transmits the acquired external data to the data comparison unit 46 (step S23).

  The data comparison unit 46 uses the feature amount of the user data transmitted from the synchronization data determination unit 41 to determine whether or not the external data obtained by the external data acquisition unit 42 matches the file 28 to be synchronized. A determination is made (step S24).

  If the data obtained by the external data acquisition means 42 and the synchronization target file 28 match (Yes in step S24), the data comparison means 46 stores the external data in the data storage area 48 (step S25). . At this time, predetermined encryption or the like may be added to the data stored in the data storage area 48. If the data obtained by the external data acquisition unit 42 matches the file 28 to be synchronized, the flow in FIG. 4 ends in step S25. It should be noted that the synchronization target data matches at the stage where data obtained from the outside is stored in the data storage area 48 or when the obtained data is determined to match the synchronization target file 28. The client 11 may be notified.

  If the external data obtained by the external data acquisition unit 42 does not match the file 28 to be synchronized (No in step S24), the data comparison unit 46 sends a data transfer instruction from the client 11 to the substantial data request unit 47. (Step S26). Even when the external data acquisition unit 42 is notified that external data cannot be acquired, the data comparison unit 46 transmits a data transfer instruction for the client 11 to the entity data request unit 47.

  The entity data request unit 47 requests the client 11 to transfer the entity data of the file 28 to be synchronized (step S27).

  Here, the storage server 31 side waits for processing on the client 11 side (step S28). That is, the storage server 31 waits in step S28 unless it receives the entity data of the client 11 (No in step S28).

  When the entity data request unit 47 of the storage server 31 receives the entity data transmitted from the entity data transmission unit 23 of the client 11 (Yes in step S28), the received entity data is stored in the data storage area 48 (step S29). ). In step S22, the entity data may be encrypted at the time of transmission by the entity data transmission unit 23, or may be encrypted at the time of storage in the data storage area 48 by the entity data request unit 47. .

  The above is the description regarding the operation of the storage server 31 of the storage system 1 according to the present embodiment.

  According to the present embodiment described above, the same data can be referred from the storage server 31 through the URL serving as the data source, and if the referenced data and the file 28 are the same, the client 11 sends the storage server 31 to the same data. No entity data transfer to That is, if the actual data transfer from the client 11 to the storage server 31 is reduced by the method of the present embodiment, the efficiency of data synchronization can be obtained in personal communication means having a small upstream bandwidth. In addition, according to the present embodiment, even in an environment that is unstable and does not have a large bandwidth and has a practical transfer amount restriction, such as a portable device, a great effect can be obtained in the efficiency of data synchronization.

  Note that the configuration of the storage system 1 according to the present embodiment is effective not only for a large-scale storage server 31 installed by a so-called storage service provider but also for a server installed in a private house. In general, in the personal communication means, the downstream band is often set narrow, while the upstream band is often set narrow. In this situation, if it is attempted to synchronize the portable device and the server installed in the private home, it is more efficient for the storage server 31 to use only the downstream communication means for the data transfer without using the upstream communication means from the portable device. high.

(effect)
According to the first embodiment of the present invention, there is an effect that the efficiency of data synchronization between the client and the storage server can be increased. The reason is that the data transfer efficiency is improved by using the high-speed and broadband communication means provided in the storage server instead of the upstream communication means of the personal communication means used by the client.

(Second Embodiment)
Next, the storage system 2 according to the second embodiment of the present invention will be described with reference to FIG. The configuration of the client 12 according to the second embodiment is the same as the configuration of the client 11 according to the first embodiment, and thus detailed description thereof is omitted. Further, the description of the same components in the internal configuration of the storage server 32 is also omitted.

  First, differences between the storage system 2 according to the second embodiment and the storage system 1 according to the first embodiment will be described.

  In the storage system 1 according to the first embodiment, an external website is used as an information source. On the other hand, the storage system 2 according to the second embodiment includes a duplication detection shared cache area 60 that is shared among a plurality of storage servers 52 and stores recently transferred data and feature values such as hash values thereof. The configuration.

  In addition, the storage server 31 of the storage system 1 according to the first embodiment includes the external data acquisition unit 21. On the other hand, the storage system 2 according to the second embodiment determines whether or not a duplicate file exists in the duplicate detection shared cache area from the hash value of the file obtained from the synchronous data determination unit 41. Candidate search means 43 is provided. The duplicate detection candidate search unit 43 is one form of the data search unit 402 of the storage system 30 according to the outline of the present embodiment.

  In addition, a function of transferring the transferred entity data to the duplication detection shared cache area 60 is added to the entity data request unit 47 of the storage server 32 according to the second embodiment.

  Here, the operational characteristics of the storage system 2 according to the second embodiment will be described.

  In the system of the first embodiment, when the data acquisition source is disclosed as a URL, efficiency is improved by transferring data from the acquisition source URL. However, on the network, the same data is not always obtained from the same source.

  In many cases, the data on the network is replicated and there are many sources of data. Therefore, in order to make synchronization and backup more efficient when there are multiple sources of data, duplicate detection technology based on features such as the hash value of the data body is used instead of the identity of the sources. It has been. The duplicate detection technique is effective when the storage system 2 has a closed configuration or when the storage server operator can be trusted. In addition, there are various variations such as on the storage server and the client in the duplication detection target portion.

  However, an open storage service such as a cloud storage service generally requires data encryption for each user. For this reason, a technique for determining whether or not the data matches the already stored data, such as a normal deduplication technique, is not sufficiently functioned by encryption.

  In consideration of the operational characteristics of the storage service, a situation in which data spread over the network in a short period of time is stored in the user and is synchronized with the storage service by a plurality of users. That is, a situation in which the same data is written to the storage server by a plurality of users in a relatively short time often occurs.

  Here, the situation in which the acquisition source has various variations for the same data and the situation in which the same data is written to the storage server by a plurality of users in a relatively short time are considered. Certainly, the duplication detection performed on the entire stored data is hindered by encryption. However, if the feature amount of data requested to be stored within a certain period and the pre-encrypted data are retained, for each storage request that occurred during that period, a transfer request for the same data is sent from the storage server. You only need to get it once. At this time, the URL for obtaining the same data may be different. Further, the above-described method functions only with the feature amount even for data without the information of the source URL.

  From this point of view, the storage system 2 according to the second embodiment has a configuration in which a duplication detection shared cache area 60, which is a cache area that functions for a certain period, is added (FIG. 6).

  The duplicate detection shared cache area 60 is generally shared by a plurality of storage servers. This may mean that there are a plurality of logical servers corresponding to a plurality of users.

  The synchronous data discriminating means 41 causes the duplication detection candidate search means 43 to check whether there is data having the same characteristic quantity in the duplication detection shared cache area 60 using the data feature quantity.

  When data having the same feature amount is found in the duplicate detection shared cache area 60, the data comparison means 46 confirms the identity of the data. When the data comparison unit 46 confirms the identity of the data, the data comparison unit 46 stores the data obtained from the duplication detection shared cache area 60 in the data storage area 48.

  When data having the same feature amount is found in the duplicate detection shared cache area 60, the entity data requesting unit 47 obtains the entity data from the client 12 in the same manner as in the first embodiment. At this time, in the second embodiment, the entity data request unit 47 stores the entity data transferred from the client 12 together with the feature amount in the duplication detection cache area 60.

  Note that the duplicate detection shared cache area 60 is replaced with a new content of data to be stored in consideration of a reference situation due to a timed expression or a capacity constraint. Existing techniques can be applied to this technique.

  Although not illustrated in FIG. 6, when the acquisition source URL is included in the metadata, the external data of the storage system 31 according to the first embodiment when duplicate detection is not performed. A means corresponding to the acquisition means 42 may be activated. Also in this case, it is assumed that the data acquired by the external data acquisition unit 42 is stored in the duplication detection shared cache area 60 together with the feature amount.

  According to the second embodiment, when the same data is stored within a certain period by a plurality of users, an effect that the data transfer efficiency can be improved can be obtained. The reason for this is that the same data is easily uploaded to the storage server in a short time, and the duplicate detection that is difficult to apply to the storage server that normally requires encryption is executed by the cache to reduce the data transfer amount. It is to do.

(Third embodiment)
Next, the storage system 3 according to the third embodiment of the present invention will be described with reference to FIG. Note that the configuration of the storage server 23 according to the third embodiment is the same as the configuration of the storage server 22 according to the second embodiment, and a description thereof will be omitted. In addition, regarding the internal configuration of the storage server 23, description of similar components is omitted.

  First, differences between the storage system 3 according to the third embodiment and the storage system 2 according to the second embodiment will be described.

  The client 13 of the storage system 3 according to the third embodiment has a configuration in which a public data non-encryption consent unit 24 is added to the client 12 of the storage system 32 according to the second embodiment. The public data non-encryption acceptance means 24 stores the data in the duplicate detection shared cache area 60 without encryption only when the data source indicated by the file attribute of the file 28 indicates that it is public. I agree to do. That is, in the third embodiment, transfer of non-encrypted data to the duplication detection shared cache area 60 is performed only when the data source indicated by the file attribute of the file 28 is public. .

  In the storage system 3 according to the third embodiment, the entity data request unit 47 stores only the public data received in the non-encrypted state in the duplication detection shared cache area 60. On the other hand, in the storage system 2 according to the second embodiment, data is temporarily stored in the data storage area 48 of the storage server 32 in an unencrypted state. A user who dislikes storing unencrypted data in the storage server 32 is presumed not to store data not disclosed to the storage server 32 in an unencrypted state. Therefore, in the third embodiment, control for storing only public data in the storage server 32 in an unencrypted state is added.

  When the consent is obtained by the public data non-encryption consent means 24, the synchronization control means 21 performs the second implementation only when the attribute of the file 28 has an attribute presumed that the source URL is public. The efficiency improvement method described in the embodiment is used. On the other hand, if the consent is not obtained by the public data non-encryption consent means 24, a normal storage method is used. It should be noted that the normal storage method can be adopted consistently with either a method that has already been encrypted at the time of output from the client or a method that has been encrypted before being stored in the data storage area 48.

  Whether or not the source URL is public can be determined by using a private area exclusion method for normal personal use, or by using a white list method in situations where there is an intranet such as a company. .

  According to the third embodiment, there is an effect that control can be performed on the basis of information on whether or not a file acquisition source is open to a level at which data encryption is performed. The reason is that the source data attached to the file is used, the efficiency is applied to the public data, and the non-public data is encrypted to the storage server without giving information for efficiency. This is because only the processed data passes.

(Fourth embodiment)
Next, the storage system 4 according to the fourth embodiment will be described with reference to FIG. Note that the configuration of the storage server 14 according to the fourth embodiment is the same as the configuration of the storage server 33 according to the first embodiment, and a description thereof will be omitted. Also, the description of the same components as the internal configuration of the storage system 33 is omitted.

  First, differences between the storage system 4 according to the fourth embodiment and the storage system 1 according to the first embodiment will be described.

  The storage system 1 according to the first embodiment uses a public data source of the external website 51. In contrast, the storage system 4 according to the fourth embodiment uses a storage server 70 (also referred to as a transfer source storage device) different from the storage server 34 as a data source.

  An access right temporary setting unit 26 that outputs an access request from the storage server 34 to the storage server 70 is added to the client 14 of the storage system 4 according to the fourth embodiment.

  The storage server 70 includes an access right setting unit 71 that sets an access right to the storage system 34 in response to an access request from the temporary access right setting unit 26, and a data storage area 72 (also referred to as a storage unit) that stores data. .

  The storage server 34 includes data transfer means 44 for accessing the data on the storage server 70 instructed by the synchronous data determination means 41 instead of the external data acquisition means 42 of the storage system 31 according to the first embodiment. . The data transfer unit 44 is one form of the data search unit 402 of the storage system 30 according to the outline of the present embodiment.

  The storage system 4 according to the fourth embodiment is an application example in the case where the data acquisition source is not the public but the other storage server 70.

  In the configuration of the storage system 4 according to the fourth embodiment, synchronization information indicating that the data is synchronized with the other storage server 70 is added as an attribute of the file 28.

  When data already stored in a certain storage server 70 is to be copied to another storage server 70, it usually goes through a client. However, in general, it can be expected that the network between storage server vendors is faster than via a client.

  Therefore, when transferring data from another storage server 70, the access right temporary setting means 26 of the client 14 sets the storage server 70 that is the data transfer source so that the storage server 34 can access it. . Note that such a technique related to delegation of authority can be realized by an existing technique.

  Next, the metadata transmission means 22 gives the means for accessing the data on the storage server 70 to the synchronous data determination means 41 as metadata.

  Next, the synchronous data determination unit 41 instructs the data transfer unit 44 to transfer data.

  The data transfer unit 44 has substantially the same configuration as the external data acquisition unit 42 according to the first embodiment. Since data transfer means for general storage services are being standardized, the data transfer means 44 is not provided with a special function. That is, it is not necessary to provide a special advance function or the like that is conscious of each other between the storage servers 34 and 70 configured to apply the storage server 21 of the first embodiment.

  Since the operation after the data transfer is completed is the same as the operation of the storage system 1 according to the first embodiment, the description is omitted.

  Finally, the access right temporary setting unit 26 deletes the access right from the storage server 34 from the storage server 70 when the data storage in the storage server 34 is completed.

  Here, the difference between a general storage service and a storage service using the method according to the present embodiment will be described again.

  In general storage services, only data transfer from clients is used as a data source. By the way, in general, a storage system operated by an operator that provides storage services is connected to a high-speed network. Therefore, it is more efficient to transfer data directly from the data source to the storage system using the capacity of the network to which the storage system operated by the operator is connected rather than transferring data from the device owned by the user. There are many.

  In the storage system according to the embodiment of the present invention, data acquisition source information is added to a file on the client. In addition, in the storage system according to the present embodiment, even when there is no information on the data acquisition source, the match with other recently stored data is estimated based on the hash value information of the entire file. In the storage system according to the present embodiment, data is transferred not from the entity data from the client but from the data source of the entity data or the recently stored identical data. As a result, according to the storage system of this embodiment, it is possible to improve the efficiency of data synchronization between the client and the storage server (first and second embodiments: FIGS. 3 and 6).

  In addition, in the storage system according to the present embodiment, for users who do not want to reach the storage server in a state where user data is not encrypted, the function of applying the method according to the present embodiment only to public data is provided with consistency. (Third embodiment: FIG. 7).

  Further, in the storage system according to the present embodiment, when the data source is not a public one and is a data copy between storage servers, the storage server that is the transfer source is connected to the storage server that is the transfer destination. Is temporarily granted access rights. As a result, the storage system according to this embodiment enables direct transfer between storage servers (fourth embodiment: FIG. 8).

  The storage apparatus according to the embodiment of the present invention described above may be realized entirely by hardware, may be realized entirely by software, or may be realized by combining hardware and software. A storage control method for controlling a storage apparatus using the method according to the present embodiment is also included in the present invention. Furthermore, a storage control program that causes an information processing apparatus such as a computer, a server, or a storage apparatus to execute the storage control method according to the embodiment of the present invention is also included in the scope of the present invention. A program storage medium such as a storage medium or a semiconductor storage medium applying an electronic, magnetic, optical, or electromagnetic property storing the storage control program according to the embodiment of the present invention is also included in the scope of the present invention. . The storage control program according to the embodiment of the present invention includes a program acquired through a network. In addition, a server system and a storage system that store the storage control program according to the present embodiment are also included in the program storage medium range according to the present embodiment.

  Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

1, 2, 3, 4 Storage system 10 Terminal device 11, 12, 13, 14 Client 21 Synchronization control means 22 Metadata transmission means 23 Substantive data transmission means 24 Public data non-encryption acceptance means 26 Access right temporary setting means 27 Browser 28 File 30 Storage device 31, 32, 33, 34 Storage server 41 Synchronous data determination means 42 External data acquisition means 43 Duplicate detection candidate search means 44 Data transfer means 46 Data comparison means 47 Entity data request means 48 Data storage area 50 Information source 51 External Website 52 Storage Server 60 Duplicate Detection Shared Cache Area 70 Storage Server 71 Access Right Setting Unit 72 Data Storage Area 100 Storage System 201 Synchronization Control Unit 202 Metadata Transmission Stage 203 the entity data transmission unit 207 reading unit 208 file storage unit 301 central processing unit 302 RAM
303 ROM
304 Input Device 305 Output Device 306 Communication Module 307 Auxiliary Storage Device 400 Data Verification Unit 401 Synchronous Data Determination Unit 402 Data Search Unit 406 Data Comparison Unit 407 Entity Data Request Unit 408 Data Storage Unit

Claims (10)

Obtaining user data metadata from a terminal device, searching for external data having the characteristics of the acquired user data metadata, and comparing the searched external data and the feature amount of the user data Data verification means for determining whether the data and the external data match or not, and acquiring the user data entity data or the external data based on the match / mismatch determination result between the user data and the external data;
A storage apparatus comprising: data storage means for storing the actual data of the user data acquired by the data verification means or the external data. The data verification means includes
Synchronous data determination means for acquiring metadata of the user data from the terminal device, and determining presence or absence of the external data having the characteristics of the acquired metadata of the user data;
When the external data is determined by the synchronous data determination means, the external data is acquired from an external information source via a line having a capacity larger than the line from the terminal device to the storage device. Data retrieval means;
Data comparing means for comparing the user data and the external data, and determining whether the user data and the external data match or not;
If it is determined by the data comparison means that the user data and the external data do not match, or if the external data cannot be obtained by the data search means, the actual data of the user data to the terminal device Entity data requesting means for requesting to transfer,
The data storage means includes
If the data comparison means determines that the user data and the external data match, the external data is stored, and the user data and the external data do not match by the data comparison means The storage apparatus according to claim 1, wherein when it is determined that the external data cannot be acquired by the data search unit, the actual data of the user data is acquired from the terminal device and stored. A terminal device for storing user data;
By acquiring metadata of the user data from the terminal device, searching for external data having the characteristics of the acquired metadata of the user data, and comparing feature values of the searched external data and the user data Data verification means for determining whether the user data and the external data match or not, and obtaining the user data entity data or the external data based on the match / mismatch determination result between the user data and the external data And a storage apparatus comprising: data storage means for storing the actual data of the user data acquired by the data verification means or the external data. The terminal device
File storage means for storing the user data;
Browsing means used to capture data from outside,
Synchronization control means for sequentially synchronizing the user data to be synchronized with the storage device;
Metadata transmission means for acquiring metadata of the user data from the synchronization control means, and transmitting the acquired metadata to the storage device;
Entity data transmission means for transmitting the user data entity data to the storage device when there is a transfer request of the user data entity data from the storage device;
The data verification means includes
Synchronous data determination means for acquiring metadata of the user data from the terminal device, and determining presence or absence of the external data having the characteristics of the acquired metadata of the user data;
When the external data is determined by the synchronous data determination means, the external data is acquired from an external information source via a line having a capacity larger than the line from the terminal device to the storage device. Data retrieval means;
Data comparing means for comparing the user data and the external data, and determining whether the user data and the external data match or not;
If it is determined by the data comparison means that the user data and the external data do not match, or if the external data cannot be obtained by the data search means, the actual data of the user data to the terminal device Entity data requesting means for requesting to transfer,
The data storage means includes
If the data comparison means determines that the user data and the external data match, the external data is stored, and the user data and the external data do not match by the data comparison means 4. The storage system according to claim 3, wherein when the external data cannot be acquired by the data search means, the actual data of the user data is stored. The browsing means is:
Realized as a browser with a unified resource location specifier associated with the data,
The data search means includes
The storage system according to claim 4, wherein the external data is acquired from a website connected via the unified resource location specifier. A shared cache area for duplicate detection that stores a feature amount including a recently transferred data and a hash value of the data shared by a plurality of storage devices;
The data search means includes
Using the feature amount of the data stored in the duplicate detection shared cache area, the presence or absence of data having the same feature as the metadata of the user data is searched, and data having the same feature as the user data is searched. If so, the corresponding data is acquired from the duplicate detection shared cache area,
The entity data request means includes
5. The storage system according to claim 4, wherein when the user data entity data is stored in the data storage means, the user data entity data is also stored in the duplication detection shared cache area. The terminal device
7. The storage system according to claim 6, further comprising public data non-encryption consent means that consents to store the user data without encryption in the duplication detection cache area only when the user data is public data. . A transfer source storage device serving as a data transfer source for the storage device;
The terminal device
Access right temporary setting means for setting access rights from the storage device to the transfer source storage device;
The transfer source storage device is
An access right setting unit that sets an access right to the storage device as a transfer destination in response to an access request from the access right temporary setting unit;
Storage means for storing data,
The data search means includes
5. The storage system according to claim 4, wherein the data stored in the storage unit of the transfer source storage apparatus is acquired according to an instruction from the synchronous data determination unit. Get user data metadata from the terminal device,
Search external data having the characteristics of the metadata of the acquired user data,
By comparing feature values of the searched external data and the user data to determine whether the user data and the external data match,
Based on the determination result of coincidence / mismatch between the user data and the external data, obtain the actual data of the user data or the external data,
A storage system control method for storing actual data of the user data or the external data in a storage device. Processing to obtain user data metadata from the terminal device;
A process of searching for external data having metadata characteristics of the acquired user data;
A process of determining whether the user data and the external data match or not by comparing feature values of the searched external data and the user data;
A process of acquiring entity data of the user data or the external data based on a determination result of coincidence / mismatch between the user data and the external data;
A storage system control program for causing a computer to execute processing for storing the actual data of the user data or the external data in a storage device.

Images