WO2014180348A1

WO2014180348A1 - Method and device for processing versioned file duplicate removal

Info

Publication number: WO2014180348A1
Application number: PCT/CN2014/077168
Authority: WO
Inventors: 袁鹏; 吴馨; 修文飞; 周国胜
Original assignee: 中兴通讯股份有限公司
Priority date: 2013-11-22
Filing date: 2014-05-09
Publication date: 2014-11-13
Also published as: CN104657386A

Abstract

A method and device for processing versioned file duplicate removal. The method comprises: acquiring a first temporary full file corresponding to a current version file; comparing a distinguishing part of an uploaded version file with the first temporary full file; and, updating the distinguishing part as a new version file. The present invention solves the problem in the relevant art that document duplicate removal or block duplicate removal processing cannot effectively and fundamentally implement storage space conservation, thus achieving the effect of greatly conserved storage space by storing only a modified part.

Description

The present invention relates to the field of communications, and in particular to a version file deduplication processing method and apparatus. BACKGROUND OF THE INVENTION An object storage system is an infrastructure of cloud storage services in the field of cloud computing, and has continued to grow at a relatively high rate in recent years. In object storage, deduplication is a core technology that saves storage space and directly affects hardware costs. Objects in the object store are saved as files in the storage system. At present, each vendor object storage supports file deduplication and block deduplication functions. File deduplication and block deduplication are for files or blocks with the same signature, and only one copy of the same file or file block for the signature is achieved. Save storage space. The file is deduplicated in file units and requires the contents of the two files to be identical. The block deduplication is in the file block unit. Therefore, in the two blocks that cannot be deduplicated, the different places may be only a part of the block, and other identical parts in the block cannot be deduplicated. Therefore, in the related art, file deduplication or block deduplication processing cannot effectively achieve storage space savings. SUMMARY OF THE INVENTION The present invention provides a method and apparatus for deduplicating a version file to at least solve the problem that the processing of file deduplication or block deduplication in the related art cannot effectively realize the saving of the storage space. According to an aspect of the present invention, a method for deduplicating a version file is provided, including: obtaining a first temporary complete file corresponding to a current version file; comparing a difference portion between the uploaded version file and the first temporary complete file; The difference is updated to a new version file. Preferably, the obtaining the first temporary complete file corresponding to the current version file includes: determining whether a parent file of the current version file exists; and if the determination result is yes, the current version file and All previous version files are merged into the first temporary complete file. Preferably, after updating the difference part to a new version file, the method further includes: receiving version file download information, where the version file download information includes a download version of the version file; acquiring a second corresponding to the download version Temporary complete file; download the second temporary complete file. Preferably, the obtaining the second temporary complete file corresponding to the downloaded version comprises: determining whether a parent file corresponding to the downloaded version exists; and if the determination result is yes, downloading the downloaded version and all previous The version files are merged into the second temporary complete file. Preferably, before acquiring the first temporary complete file corresponding to the current version file, the method further includes: determining an enabled version policy and/or a version file deduplication function. According to another aspect of the present invention, a version file deduplication device is provided, including: a first obtaining module, configured to acquire a first temporary complete file corresponding to a current version file; and a comparison module configured to compare an uploaded version file a difference portion with the first temporary complete file; an update module configured to update the difference portion to a new version file. Preferably, the first obtaining module includes: a first determining unit, configured to determine whether a parent file of the current version file exists; and a first merging unit configured to determine that the result of the first determining unit is yes In the case, the current version file and all previous version files thereof are merged into the first temporary complete file. Preferably, the device further includes: a receiving module, configured to receive the version file downloading information, where the version file downloading information includes a download version of the version file; and the second obtaining module is configured to obtain the first corresponding to the downloaded version Second temporary complete file; download module, set to download the second temporary complete file. Preferably, the second obtaining module includes: a second determining unit, configured to determine whether a parent file corresponding to the downloaded version exists; and a second merging unit configured to determine that the second determining unit is yes In the case, the downloaded version and all previous version files are merged into the second temporary complete file. Preferably, the apparatus further comprises: a determining module configured to determine an enable version policy and/or a version file deduplication function. According to the present invention, the first temporary complete file corresponding to the current version file is obtained; the difference portion between the uploaded version file and the first temporary complete file is compared; and the difference portion is updated to the new version file, and the related technology is solved. The process of deduplication or block deduplication in the middle of the file does not effectively realize the problem of saving the storage space, thereby achieving the effect of storing only the modified part and greatly saving storage space. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are set to illustrate,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, In the drawing: 1 is a flowchart of a method for deduplicating a version file according to an embodiment of the present invention; FIG. 2 is a block diagram showing a structure of a deduplication processing device for a version file according to an embodiment of the present invention; FIG. 3 is a version file according to an embodiment of the present invention. FIG. 4 is a block diagram showing a preferred structure of a version file deduplication processing apparatus according to an embodiment of the present invention; FIG. 5 is a block diagram deduplication processing apparatus according to an embodiment of the present invention; FIG. 6 is a block diagram showing a preferred structure of a version file deduplication processing apparatus according to an embodiment of the present invention; FIG. 7 is a schematic structural diagram of a version file deduplication system according to an embodiment of the present invention; FIG. 9 is a flowchart of an upload control process according to an embodiment of the present invention; FIG. 10 is a flowchart of a version file deduplication request process according to an embodiment of the present invention; Flowchart of deduplication processing of version file according to an embodiment of the present invention; FIG. 12 is a diagram of a version file deduplication system for uploading according to an embodiment of the present invention; FIG. 13 is a flowchart of deduplication processing of an uploaded modified partial version file by a version file deduplication system according to an embodiment of the present invention; FIG. 14 is a user according to an embodiment of the present invention; FIG. 15 is a flowchart of processing a version file download when a version policy is opened according to an embodiment of the present invention; FIG. 16 is a flowchart of a version file assembly and merge process according to an embodiment of the present invention; 17 is a flowchart of assembling and merging version files according to an embodiment of the present invention; FIG. 18 is a flowchart of processing a user uploading a full version file of a specified object in the case of version deduplication according to an embodiment of the present invention; 19 is a flow diagram of processing a user to download a specified version of a specified object in the case of version deduplication in accordance with a preferred embodiment of the present invention. BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict. In this embodiment, a method for deduplicating a version file is provided. FIG. 1 is a flowchart of a method for deduplicating a version file according to an embodiment of the present invention. As shown in FIG. 1, the process includes the following steps: Step S102, Obtaining a first temporary complete file corresponding to the current version file; Step S104, comparing a difference portion between the uploaded version file and the first temporary complete file; Step S106, updating the difference portion to a new version file. Through the above steps, only the modified part (ie, the difference part) is updated and stored, and the processing of file deduplication or block deduplication in the related art cannot effectively realize the problem of saving the storage space. It not only solves the problem of storage waste in related technologies, but also stores only the modified part, which greatly saves the effect of storage space. When the first temporary complete file corresponding to the current version file is obtained, all the files before the current version file are merged to form a complete file. For example, the following processing may be used to determine whether the parent file of the current version file exists; If the result of the determination is yes, the current version file and all the previous version files are merged into the first temporary complete file. It should be noted that the parent file may still exist in the parent file, and therefore, the stored version file is continuously For example, you can add the parent file to the merge list, and then loop through the parent files until the initial version is added to the list, and then merge to form a first temporary complete file in the order from old to new (Vl~Vn). After the difference portion is updated to the new version file, the stored version file may also be downloaded according to the version file stored in the version, for example, the version file download information is received first, where the version file download information includes the download version of the version file; Download the second temporary complete file corresponding to the version; download the second temporary complete file. Similar to the uploading process of the version file, when downloading, obtaining the second temporary complete file corresponding to the downloaded version may also adopt a similar process, that is, first determining whether the parent file corresponding to the downloaded version exists; if the judgment result is yes, Combine the downloaded version and all previous version files into a second temporary complete file. That is, both the version file upload and download processing involve the merge processing of the version file. Preferably, before obtaining the first temporary complete file corresponding to the current version file, the method further includes: determining an enabled version policy and/or a version file deduplication function, and determining that the version and/or version file deduplication function is enabled Then, the deduplication processing of the version file is performed, which can effectively improve the processing efficiency of the file. In this embodiment, a version file deduplication device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and has not been described again. As used hereinafter, the term "module" may implement a combination of software and/or hardware of a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and conceivable. 2 is a structural block diagram of a version file deduplication processing apparatus according to an embodiment of the present invention. As shown in FIG. 2, the apparatus includes a first obtaining module 22, a comparing module 24, and an updating module 26. The apparatus will be described below. The first obtaining module 22 is configured to obtain the first temporary complete file corresponding to the current version file; the comparing module 24 is connected to the first obtaining module 22, and is configured to compare the difference between the uploaded version file and the first temporary complete file; The update module 26, coupled to the comparison module 24 described above, is arranged to update the difference portion to a new version file. FIG. 3 is a structural block diagram of a first obtaining module 22 in a version file deduplication processing apparatus according to an embodiment of the present invention. As shown in FIG. 3, the first obtaining module 22 includes a first determining unit 32 and a first combining unit 34. The first acquisition module 22 will be described below. The first determining unit 32 is configured to determine whether the parent file of the current version file exists; the first merging unit 34 is connected to the first determining unit 32, and is set to be, if the judgment result of the first determining unit is yes, The current version file and all previous version files are merged into the first temporary complete file. 4 is a block diagram of a preferred structure of a version file deduplication processing apparatus according to an embodiment of the present invention. As shown in FIG. 4, the apparatus includes a receiving module 42 and a second acquiring module, in addition to all the modules shown in FIG. 44 and the download module 46, the version file deduplication device will be described below. The receiving module 42 is connected to the update module 26, and is configured to receive the version file download information, where the version file download information includes a download version of the version file, and the second obtaining module 44 is connected to the receiving module 42 and configured to acquire and download. The second temporary complete file corresponding to the version; the downloading module 46, connected to the second obtaining module 44, is configured to download the second temporary complete file. FIG. 5 is a structural block diagram of a second obtaining module 44 in a version file deduplication processing apparatus according to an embodiment of the present invention. As shown in FIG. 5, the second obtaining module 44 includes a second determining unit 52 and a second combining unit 54, The second acquisition module 44 will be described below. The second determining unit 52 is configured to determine whether the parent file corresponding to the downloaded version exists; the second merging unit 54 is connected to the second determining unit 52, and is configured to, if the determination result of the second determining unit is YES, The download version and all previous version files are merged into a second temporary complete file. 6 is a block diagram of a preferred structure of a version file deduplication processing apparatus according to an embodiment of the present invention. As shown in FIG. 6, the apparatus includes a determination module 62 in addition to all the modules shown in FIG. The determination module 62 is described. The determining module 62 is connected to the first obtaining module 22, and is configured to determine an enabled version policy and/or a version file deduplication function. The version file refers to the different historical versions of the same object after the object storage has opened the file version function. Different historical versions reflect the user's modification of the file. Generally, the amount of modification is not too large each time, that is, the difference between two adjacent historical versions is not too large, but even if one byte is modified, the file deduplication function is not applicable. The block deduplication function has the previous problem. That is to say, in the version file, there is also a problem that the above file is deduplicated and the block is deduplicated. Based on the above problem, in the present embodiment, a method and system for deduplicating a version file in object storage is provided. The method includes: a process in which the object storage system applies a version policy to a new version of a user-uploaded object and deduplicates the version file. The process includes the steps of the object storage system determining whether to use the version policy for the target object, creating a version file, deduplicating the version file, and assembling the deduplicated version file when the user downloads according to the steps set by the system and the user. The version file deduplication system, which is part of the object storage system, is used to set the system version policy for the system and the user and to create a deduplicated version file. 7 is a schematic structural diagram of a version file deduplication system according to an embodiment of the present invention. As shown in FIG. 7, the system includes: a version policy setting module 70 (corresponding to the function of the determining module 62), and a request processing module 72 (with the above) The receiving module 42 functions equivalently, the version file processing module 74 (corresponding to the comparison module 24, the first obtaining module 22, and the second obtaining module 44), and the file reading and writing module 76 (corresponding to the function of the download module 46 described above), Explain the system in the face. The version policy setting module 70 provides an operation interface for the administrator and is set to set the version policy. The version file deduplication system does not perform version file processing when the version policy setting module 70 does not enable the version policy, and performs version file processing but does not perform version file processing when the version policy setting module 70 enables the version policy but does not set the version deduplication policy. Go heavy. The request processing module 72 receives the upload/download request of the user, and calls the version file processing module 74 to process the response to the user. For the upload operation, the request processing module 72 receives the file uploaded by the user and returns the result of the operation. For the download operation, the request processing module 72 transmits the file requested by the user to the user. The version file processing module 74 performs version file processing according to the requirements of the request processing module 72, and mainly performs two functions of version file merging and version file creation. The file reading and writing module 76 completes the reading and writing operations of the file on the file system according to the requirements of the version file processing module 74. Through the above-mentioned version file de-duplication processing method and system, the version file is assembled, so that each version file only needs to save the modified part except the original original version, and the user-specified version file is generated by orderly merging; The creation of the version file allows the object storage system to save the entire version file when the user uploads a new version of the object, and only needs to save the new version with the current version, which greatly saves storage space. FIG. 8 is a schematic diagram of a file merging method according to an embodiment of the present invention. As shown in FIG. 8, the file used in the de-duplication processing has a special file header, and the file header includes a parent file pointer, that is, path information of the parent file. When accessing a file, you need to merge it with its parent file to get the complete file. At the same time, the parent file may have a parent file again, so you need to continue the merge in an orderly manner until the last parent file does not have a parent file. The parent file pointer is empty. For example, the parent file can be added to the merge list first, and the parent file is looped through until the initial version is added to the list, and then merged in order from old to new (Vl~Vn), and finally a the complete document. The operation of the file by the user mainly includes two parts of uploading and downloading. When the version file is deduplicated, both parts need the version file to be processed by the system. FIG. 9 is a flowchart of an upload control process according to an embodiment of the present invention. As shown in FIG. 9, the process includes the following steps: Step S902: determining whether a version policy is set in the current system, if not, proceeding to step S904, if If it is set, go to step S906; Step S904, if the version policy is not set, the version processing is not required, and the file can be directly saved, if the original file with the same name exists, the direct coverage is performed; Step S906, if the version policy is set, the uploaded file is required. If the version file is processed, it will enter the version file processing. FIG. 10 is a flowchart of a version file de-requesting process according to an embodiment of the present invention. As shown in FIG. 10, when a version file upload is processed when a version policy is enabled, the process includes the following steps: In step S1002, the object storage system determines whether the version file is deduplicated. If not, the process goes to step S1004. Otherwise, the process goes to step S1006. In step S1004, when the version file is not set, the uploaded version file is saved. Re-processing, setting the newly uploaded file as the current version of the object, and the original current version file becomes the historical version; Step S1006, setting the version file to be de-duplicated when the version file is deduplicated, then performing the version file de-duplication processing. FIG. 11 is a flowchart of a deduplication process of a version file according to an embodiment of the present invention. As shown in FIG. 11, the process includes the following steps: Step S1102: Determine whether an uploaded version file is a complete file or only a modified part of a file segment. If it is a complete file, go to step S1104, otherwise go to step S1106; step S1104, perform deduplication processing of full version file uploading; step S1106: perform deduplication processing for modifying partial version file upload. FIG. 12 is a flowchart of deduplication processing of a full version file uploaded by a version file deduplication system according to an embodiment of the present invention. As shown in FIG. 12, the process includes the following steps: Step S1202: assembling and merging the current version file , obtaining a complete current version file; Step S1204, comparing the difference between the uploaded version file and the current version file of the object, and saving the modified part as a new version file; Step S1206, designating the parent file of the new version file as the current version file Step S1208, the new version file is set as the current version of the object, and the previous current version file becomes the historical version of the object. FIG. 13 is a flowchart of a deduplication process performed by a version file deduplication system on a modified partial version file according to an embodiment of the present invention. As shown in FIG. 13, the process includes the following steps: Step S1302: Uploading a modified partial file Save as a new version of the object; Step S1304, designate the parent file of the new version file as the current version file; In step S1306, the new version file is set as the current version file, and the previous current version file becomes the historical version of the object. The download file is taken as an example to describe the deduplication of the version file. FIG. 14 is a flowchart of a user downloading a version file request process according to an embodiment of the present invention. As shown in FIG. 14, the process includes the following steps: Step S1402: determining whether a version policy is set in the current system, and if not, proceeding to the step S1404, if it is set, go to step S1406; Step S1404, when the version policy is not set, the version processing is not required, and the user specified object is directly downloaded. Step S1406: If the version policy is set to indicate the uploaded file, the version file processing is performed. FIG. 15 is a flowchart of processing a version file download when a version policy is enabled according to an embodiment of the present invention. As shown in FIG. 15, the process includes the following steps: Step S1502: The object storage system determines whether a version file is set to be de-duplicated. If it is not set, go to step S1504, otherwise go to step S1506; Step S1504, when the version file is not deduplicated, do not need to do assembly and merge processing, directly download the specified version of the user-specified object; Step S1506, set the version file to de-weight When the version file is assembled and merged. 16 is a flowchart of a version file assembly and merge process according to an embodiment of the present invention. As shown in FIG. 16, the process includes the following steps: Step S1602: Obtain a complete current version file by assembling and merging the current version of the object; S1604: Download the specified version of the assembled user-specified object. FIG. 17 is a flowchart of assembling and merging a version file according to an embodiment of the present invention. As shown in FIG. 17, the process includes the following steps: Step S1702: Obtain a specified version file of a user-specified object and join a to-be-combined list, and query the file. Parent file information; Step S1704, if the parent file information that is queried does not exist, go to step S1706, otherwise go to step S1708; Step S1706, sequentially merging the files in the to-be-combined list, generating a complete version file, and ending the merging; Step S1708, adding the parent file to the to-be-combined list, and querying the parent file information of the parent file. DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the preferred embodiments of the present invention will be described. Preferred Embodiment 1 It should be noted that the following is applied to an object storage system as an example. The initial conditions of this embodiment include:

(1) The version policy is enabled, and the version file is set to be de-duplicated;

(2) The user uploaded the latest version of a file, Vn, and uploaded the full file;

(3) The file already has a historical version, which is version Vl~Vn-l, where VI is the first original version, which is a complete file, and V2~Vn-l is a modified partial version file based on the previous version. . FIG. 18 is a flowchart of processing a user uploading a full version file of a specified object in the case of version deduplication according to an embodiment of the present invention. As shown in FIG. 18, when a user uploads a Vn version complete file, the version file deduplication system performs a version file. The reprocessing includes the following steps: Step S1802: determining that the current system has enabled the version policy; Step S1804, determining that the current system has set the version deduplication; Step S1806, determining that the specified object of the uploaded version is Vn is a complete file; Step S1808, Querying the current version of the object, which is Vn-1; Step S1810, obtaining the parent file information Vn-2... of the version file Vn-1 until VI; Step S1812, combining Vl~Vn-1 into the temporary of Vn-1 The complete version file Vn-Γ; step S1814, comparing the uploaded file Vn with the temporary complete file Vn-Γ, saving the difference portion as the modified partial file Vn; Step S1816, setting the current version of the object to Vn, Vn- 1 becomes the historical version; in step S1818, the version file is de-duplicated successfully. Preferred Embodiment 2 It should be noted that the application is applied to an object storage system as an example. The initial conditions of this embodiment include:

(2) The current version of the user download object is Vm, and the user specifies to download the Vn version thereof, where m>=n; FIG. 19 is a flowchart of processing the user to download the specified version of the specified object in the case of version deduplication according to a preferred embodiment of the present invention. As shown in FIG. 19, when the user downloads the Vn version file, the version file deduplication system performs the version file merging process including the following steps:

Niu Peng

Less S1902, judge that the current system has enabled the version policy; Niu Peng

Less S1904, judge that the current system has set the version to de-weight; Niu Peng

Less S1906, judge the download target is the Vn version of the object; Niu Peng

Less S1908, get the parent file information of version Vn, Vn-1... until VI;

Niu Peng

Less S1910, merge Vl~Vn into a temporary complete file Vn of Vn,;

Niu Peng

Less S1912, download the file Vn'. Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computing device, which can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device, such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein. The steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps are fabricated as a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software. The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and scope of the present invention are intended to be included within the scope of the present invention. INDUSTRIAL APPLICABILITY As described above, a version file deduplication processing method and apparatus provided by an embodiment of the present invention have the following The beneficial effects: solving the problem that the file deduplication or the block deduplication in the related art can not effectively realize the saving of the storage space, thereby achieving the effect of only storing the modified part and greatly saving the storage space. .

Claims

Claim

1. A version file deduplication method, including:

Obtaining the first temporary complete file corresponding to the current version file;

Comparing the difference between the uploaded version file and the first temporary complete file;

Update the difference portion to a new version file.

2. The method according to claim 1, wherein the obtaining the first temporary complete file corresponding to the current version file comprises:

Determining whether a parent file of the current version file exists;

In the case where the judgment result is YES, the current version file and all previous version files thereof are merged into the first temporary complete file.

3. The method according to claim 1, wherein after updating the difference portion to a new version file, the method further comprises:

Receiving version file download information, where the version file download information includes a download version of the version file;

Obtaining a second temporary complete file corresponding to the downloaded version;

Download the second temporary complete file.

4. The method according to claim 3, wherein obtaining the second temporary complete file corresponding to the downloaded version comprises:

Determining whether a parent file corresponding to the downloaded version exists;

In the case where the judgment result is YES, the downloaded version and all of the previous version files are merged into the second temporary complete file.

The method according to any one of claims 1 to 4, further comprising: before acquiring the first temporary complete file corresponding to the current version file,

Be sure to enable versioning policy and/or version file deduplication.

6. A version file deduplication device, comprising:

The first obtaining module is set to obtain the first temporary end corresponding to the current version file. And a comparison module configured to compare a difference portion between the uploaded version file and the first temporary complete file; and an update module configured to update the difference portion to a new version file. The device according to claim 6, wherein the first obtaining module comprises:

a first determining unit, configured to determine whether a parent file of the current version file exists; the first merging unit is configured to: when the determination result of the first determining unit is yes, the current version file and All previous version files are merged into the first temporary complete file. The device according to claim 6, further comprising:

a receiving module, configured to receive version file download information, where the version file download information includes a download version of the version file;

The second obtaining module is configured to obtain a second temporary complete file corresponding to the downloaded version; and the downloading module is configured to download the second temporary complete file. The device according to claim 8, wherein the second obtaining module comprises: a second determining unit, configured to determine whether a parent file corresponding to the downloaded version exists; and a second merging unit, configured to be in the second If the judgment result of the judgment unit is YES, the download version and all previous version files are merged into the second temporary complete file. The apparatus according to any one of claims 6 to 9, further comprising: a determination module configured to determine an enable version policy and/or a version file deduplication function.