CN103455631A - Method, device and system for processing data - Google Patents
Method, device and system for processing data Download PDFInfo
- Publication number
- CN103455631A CN103455631A CN2013104329367A CN201310432936A CN103455631A CN 103455631 A CN103455631 A CN 103455631A CN 2013104329367 A CN2013104329367 A CN 2013104329367A CN 201310432936 A CN201310432936 A CN 201310432936A CN 103455631 A CN103455631 A CN 103455631A
- Authority
- CN
- China
- Prior art keywords
- index value
- index
- blocks
- files
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 72
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000003860 storage Methods 0.000 claims abstract description 77
- 238000003672 processing method Methods 0.000 claims description 15
- 238000000638 solvent extraction Methods 0.000 claims description 12
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 108010001267 Protein Subunits Proteins 0.000 claims description 2
- 230000001960 triggered effect Effects 0.000 claims description 2
- 230000008030 elimination Effects 0.000 abstract 1
- 238000003379 elimination reaction Methods 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 10
- 238000007418 data mining Methods 0.000 description 9
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a device and a system for processing data. The method is characterized in that a current index list in advance is created in advance and at least contains one index value, and each index value corresponds to one stored file block. The method includes acquiring a to-be-stored target file which contains at least one file block; acquiring a target index value of each file block; inquiring index values consistent with the target index values in the current index list; abandoning the file blocks corresponding to the certain target index values when the certain index values are consistent with the inquired index values in the current index list, or writing the certain target index values into the current index list and storing the file blocks corresponding to the certain target index values when the certain index values are not consistent with the inquired index values in the current index list. The method, the device and the system in an embodiment of the invention have the advantages that redundancy elimination processing can be effectively performed on to-be-stored data when the method, the device and the system are applied to a distributed file system, so that the storage utilization rate of space is effectively increased, and resources are saved.
Description
Technical field
The application relates to technical field of data processing, particularly a kind of data processing method, Apparatus and system.
Background technology
Development along with internet, network data information increases with the magnanimity formula, and have a state of data high redundancy in the data of storage more, and, As time goes on the phenomenon of data redundancy is more and more obvious, increasing redundant data takies more storage space, causes the waste of storage resources.
Summary of the invention
The application's technical matters to be solved is to provide a kind of data processing method, Apparatus and system, obvious in order to solve in prior art the data redundancy phenomenon, causes the technical matters of storage resources waste.
The application provides a kind of data processing method, sets up in advance current index, and described current index comprises at least one index value, and each described index value is corresponding with a blocks of files of having stored respectively, and described method comprises:
Obtain file destination to be stored, described file destination comprises at least one blocks of files;
Obtain respectively the target index value of each described blocks of files;
To each described target index value, the inquiry index value consistent with described target index value in described current index;
When in described current index, inquiring consistent index value, the corresponding blocks of files of described target index value is abandoned, otherwise, write in described current index by described target index value and store blocks of files corresponding to described target index value.
Said method, preferred, describedly obtain file destination to be stored, comprising:
Receive the file storage resource request, described file storage resource request comprises file data to be stored;
Described file data is divided in logic, obtained including the file destination of at least one blocks of files.
Said method, preferred, the described target index value that obtains each described blocks of files comprises:
The Hash hash calculated value that obtains respectively each described blocks of files is target index value separately as it.
Said method, preferred, blocks of files corresponding to the described target index value of described storage comprises:
Create the metadata of the blocks of files that described target index value is corresponding;
Obtain the memory address of described target index value respective file piece;
With described memory address, blocks of files corresponding to described target index value stored as DSB data store block.
Said method, preferred, set up the secondary index table is arranged in described current index, wherein, described to each described target index value, the inquiry index value consistent with described target index value in current index comprises:
To each described target index value, the corresponding index key value of the described target index value of inquiry in described secondary index table;
When inquiring the index key value corresponding with described target index value in described secondary index table, in described current index with in the corresponding sub-list of described index key value, inquire about the index value consistent with described target index value, if while in described sub-list, inquiring consistent index value, carry out and described the corresponding blocks of files of described target index value is abandoned, otherwise, carry out describedly writing in described current index by described target index value and storing blocks of files corresponding to described target index value;
When not inquiring the index key value corresponding with described target index value in described secondary index table, carry out describedly writing in described current index by described target index value and storing blocks of files corresponding to described target index value.
The application provides a kind of data processing equipment, sets up in advance current index, and described current index comprises at least one index value, and each described index value is corresponding with a blocks of files of having stored respectively, and described device comprises:
The file acquisition unit, for obtaining file destination to be stored, described file destination comprises at least one blocks of files;
The index value acquiring unit, for obtaining respectively the target index value of each described blocks of files;
The index value query unit for to each described target index value, is inquired about the index value consistent with described target index value in current index, when in described current index, inquiring consistent index value, trigger the first processing unit, otherwise, the second processing unit triggered;
The first processing unit, for abandoning blocks of files corresponding to described target index value;
The second processing unit, for writing to described target index value described current index and storing blocks of files corresponding to described target index value.
Said apparatus, preferred, described file acquisition unit comprises:
The request receiving subelement, for receiving the file storage resource request, described file storage resource request comprises file data to be stored;
The logical partitioning subelement, for described file data is divided in logic, obtain including the file destination of at least one blocks of files.
Said apparatus, preferred:
Described index value acquiring unit, specifically for the Hash hash calculated value that obtains respectively each described blocks of files as it target index value separately.
Said apparatus, preferred, described the second processing unit comprises:
Index value writes subelement, for described target index value is write to described current index;
The blocks of files storing sub-units, for storing blocks of files corresponding to described target index value;
Concrete, described blocks of files storing sub-units comprises:
The metadata store module, for creating the metadata of the blocks of files that described target index value is corresponding;
Address acquisition module, for obtaining the memory address of described target index value respective file piece;
The blocks of files memory module, for described memory address, stored blocks of files corresponding to described target index value as DSB data store block.
Said apparatus, preferred, set up the secondary index table is arranged in described current index, wherein, described index value query unit comprises:
The first inquiry subelement, be used for each described target index value, the corresponding index key value of the described target index value of inquiry in described secondary index table, when inquiring the index key value corresponding with described target index value in described secondary index table, trigger the second inquiry subelement, otherwise, trigger described the second processing unit;
The second inquiry subelement, for sub-list corresponding with described index key value in described current index, inquire about the index value consistent with described target index value, if while in described sub-list, inquiring consistent index value, trigger described the first processing unit, otherwise, trigger described the second processing unit.
The application also provides a kind of data handling system, comprise at least one index server, at least one is the operations server corresponding with each described index server respectively, meta data server, at least one storage server, and data processing equipment as described as above-mentioned any one, described data processing equipment is arranged on its corresponding index server of described operations server, and described current index is placed in described index server, described storage server is in order to deposit stored blocks of files, described meta data server is in order to deposit in described storage server the metadata of storage file piece.
Said system, preferred, also comprise the synchronizer be connected with each described index server respectively, wherein:
Described synchronizer, carry out data synchronous operation for the current index of each the described index server to coupled.
From such scheme, a kind of data processing method that the application provides, Apparatus and system, carrying out index value by each blocks of files in the file destination to be stored obtains, and then the state whether be stored in current index of its index value is judged to inquiry, thereby draw whether stored state of this document piece, and then the blocks of files that stored index value is corresponding is given up, not stored blocks of files is stored and corresponding index value writes in current index by it, realization is to the superfluous processing of disappearing of blocks of files, reach the application's purpose.Can be effectively when the embodiment of the present application is applied in distributed file system to the data to be stored superfluous processing that disappeared, avoid redundant data in prior art to take the problem that a large amount of storage spaces cause the wasting of resources, effectively improve the space utilisation in space, saving resource.
The accompanying drawing explanation
In order to be illustrated more clearly in the technical scheme in the embodiment of the present application, in below describing embodiment, the accompanying drawing of required use is briefly described, apparently, accompanying drawing in the following describes is only some embodiment of the application, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain according to these accompanying drawings other accompanying drawing.
The process flow diagram of a kind of data processing method embodiment mono-that Fig. 1 provides for the application;
The part process flow diagram that Fig. 2 is the embodiment of the present application one;
The part process flow diagram of a kind of data processing method embodiment bis-that Fig. 3 provides for the application;
The process flow diagram of a kind of data processing method embodiment tri-that Fig. 4 provides for the application;
The structural representation of a kind of data processing equipment embodiment tetra-that Fig. 5 provides for the application;
The part-structure schematic diagram that Fig. 6 is the embodiment of the present application four;
The part-structure schematic diagram of a kind of data processing equipment embodiment five that Fig. 7 provides for the application;
Another part structural representation that Fig. 8 is the embodiment of the present application five;
The structural representation of a kind of data processing equipment embodiment six that Fig. 9 provides for the application;
The structural representation of a kind of data handling system embodiment seven that Figure 10 provides for the application;
Figure 11 is that in the embodiment of the present application seven, the data between the structure composition are moved towards figure;
The certain applications exemplary plot that Figure 12 is the embodiment of the present application seven;
Another structural representation that Figure 13 is the embodiment of the present application seven;
The application example figure that Figure 14 is the embodiment of the present application seven.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present application, the technical scheme in the embodiment of the present application is clearly and completely described, obviously, described embodiment is only the application's part embodiment, rather than whole embodiment.Embodiment based in the application, those of ordinary skills are not making under the creative work prerequisite the every other embodiment obtained, and all belong to the scope of the application's protection.
With reference to figure 1, the process flow diagram of a kind of data processing method embodiment mono-provided for the application, described method can be applied in the superfluous processing that disappears to file to be stored data in distributed file system, the embodiment of the present application is set up current index in advance, described current index comprises at least one index value, each described index value is corresponding with a blocks of files of having stored respectively, wherein, the blocks of files that described index value is corresponding to it has unique identification, and described method can comprise the following steps:
Step 101: obtain file destination to be stored, described file destination comprises at least one blocks of files.
Wherein, between each described blocks of files, have logical interdependency, each described blocks of files is corresponding with a copy.With reference to figure 2, be the realization flow figure of described step 101, described step 101 can realize by following steps:
Step 201: receive the file storage resource request, described file storage resource request comprises file data to be stored.
Wherein, described file storage resource request can be generated according to self-demand by the user, and this request comprises file data to be stored.
Step 202: described file data is carried out to logical partitioning, obtain including the file destination of at least one blocks of files.
Wherein, described logical partitioning refers to, to described file data, the division rule default according to it adopts the method for mark as memory address division rule etc., described file data is divided into to a plurality of sections, and the starting and ending position of recording each file section, extremely the file content between end position is as a blocks of files for a reference position, and the All Files piece forms file destination, i.e. described file data to be stored.It should be noted that, after by logical partitioning, described file destination is still a complete file, and this logical partitioning is different from the division of physical address.
Step 102: the target index value that obtains respectively each described blocks of files.
Wherein, described target index value has unique identification to its blocks of files, and each blocks of files all has a unique target index value corresponding with it, and this target index value can be with the Hash hash value representation of this document piece, therefore, described step 102 can realize in the following manner:
The hash calculated value that obtains respectively each described blocks of files is as its each self-corresponding target index value.
In the embodiment of the present application, to the account form of hash value, can be realized by application such as MD5 or SHA.
Step 103: to each described target index value, in described current index, the inquiry index value consistent with described target index value, when in described current index, inquiring consistent index value, perform step 104, otherwise, perform step 105 and described step 106.
Wherein, described current index comprises each index value of stored blocks of files, thus, while in described step 103, inquiring the index value consistent with described target index value in described current index, execution step 104, otherwise, perform step 105 and step 106.
Step 104: the corresponding blocks of files of described target index value is abandoned.
Wherein, described step 104 is to inquire the index value consistent with described target index value in described step 103 in described current index, now show in described current index to have existed described target index value, the blocks of files that this target index value is corresponding is stored, therefore, blocks of files corresponding to this target index value is without being stored again, can perform step 104 abandons the corresponding blocks of files of described target index value, can also this target index value be abandoned simultaneously simultaneously, realize the data superfluous purpose that disappears.
Step 105: described target index value is write in described current index.
Step 106: store blocks of files corresponding to described target index value.
Wherein, described step 105 and described step 106 are not inquire the index value consistent with described target index value in described step 103 in described current index, now show that the blocks of files that this target index value is corresponding is not stored, now perform step 105 and described step 106 described target index value is write in described current index, basic as inquiry while inquiring about for index value next time, blocks of files corresponding to described target index value stored in the disappear superfluous processing of realization to this target index value respective file piece simultaneously.
It should be noted that, in described step 106 storage blocks of files comprise this document piece itself with and copy.
From such scheme, a kind of data processing method embodiment mono-that the application provides, carrying out index value by each blocks of files in the file destination to be stored obtains, and then the state whether be stored in current index of its index value is judged to inquiry, thereby draw whether stored state of this document piece, and then the blocks of files that stored index value is corresponding is given up, not stored blocks of files is stored and corresponding index value writes in current index by it, realization is to the superfluous processing of disappearing of blocks of files, reach the application's purpose.Can be effectively when the embodiment of the present application is applied in distributed file system to the data to be stored superfluous processing that disappeared, avoid redundant data in prior art to take the problem that a large amount of storage spaces cause the wasting of resources, effectively improve the space utilisation in space, saving resource.
With reference to figure 3, the process flow diagram of step 106 described in a kind of data processing method embodiment bis-provided for the application, described step 106 can be realized by following each step:
Step 301: the metadata that creates the blocks of files that described target index value is corresponding.
Wherein, described metadata be describe its respective file piece content, blocks of files copy information (each blocks of files can comprise a plurality of copies) with and the respective file piece under the attribute information etc. of file destination.In distributed file system, described metadata can be placed in the meta data server of this system.
Step 302: the memory address of obtaining described target index value respective file piece.
Wherein, described memory address comprises the first memory address of this document piece self and the second memory address of this document piece part.
It should be noted that, set in advance the blocks of files storage rule in the embodiment of the present application, the memory address obtained in described step 302 can be the memory address of the described target index value respective file piece that gets according to described blocks of files storage rule, and described memory address means this document piece can be stored in which storage server in distributed file system.Wherein, described blocks of files storage rule can be placed in meta data server in advance, thus, described step 302 is when realizing, can, by sending the address acquisition request to meta data server, after generating memory address by meta data server, by the application, this memory address be obtained.
Step 303: with described memory address, blocks of files corresponding to described target index value stored as DSB data store block.
Wherein, in described step 303, can comprise: the blocks of files that described target index value is corresponding is stored with the first memory address of the blocks of files in described memory address self, and by described target index value, the copy of corresponding blocks of files is stored successively with the second memory address of the blocks of files copy in described memory address, realizes the storing process to blocks of files.
It should be noted that, described step 301 can be changed with the execution sequence of described step 302, and also can first carry out described step 302 and described step 303, then carry out described step 301, or, carry out described step 301 and described step 302 simultaneously.
In order to improve the efficiency of search index value in described step 103, can when setting up, described current index set up its secondary index table, and described secondary index table comprises at least one index key value.For example, index value in described current index is set up to second layer concordance list, be specially: described current index is carried out to index value and sorted according to the ordering rule set in advance, choose an index value at interval of at least one index value in current index after sequence afterwards, the index value of choosing is set up to second layer concordance list, and described second layer concordance list comprises at least one index key value.Now, with reference to figure 4, the process flow diagram of a kind of data processing method embodiment tri-provided for the application, wherein, described step 103 can comprise the following steps:
Step 401: to each described target index value, the corresponding index key value of the described target index value of inquiry in described secondary index table, while inquiring the index key value corresponding with described target index value in described secondary index table, execution step 402, otherwise, perform step 105 and described step 106;
Step 402: in described current index with in the corresponding sub-list of described index key value, inquire about the index value consistent with described target index value, if while in described sub-list, inquiring consistent index value, carry out described step 104, otherwise, carry out described step 105 and described step 106.
Such scheme is: to each described target index value, and the corresponding index key value of the described target index value of inquiry in described secondary index table;
When inquiring the index key value corresponding with described target index value in described secondary index table, in described current index, with in the corresponding sub-list of described index key value, inquire about the index value consistent with described target index value;
If while in described sub-list, inquiring consistent index value, the corresponding blocks of files of described target index value is abandoned, otherwise, write in described current index by described target index value and store blocks of files corresponding to described target index value;
And, when not inquiring the index key value corresponding with described target index value in described secondary index table, write in described current index by described target index value and store blocks of files corresponding to described target index value.
From such scheme, in the embodiment of the present application three, current index is set up to the secondary index table, utilize the search index mode of binary tree, improve the inquiry velocity to index value.
Wherein, in above-mentioned each embodiment, can also carry out data mining to the index value in described current index, be that the maintenance data mining algorithm carries out the extraction of valuable information to index value, for example according to the index value in described current index, learn that the repeat usage of the blocks of files which index value is corresponding is higher.
The example of data mining process is as follows: have section to be called the film of " robot Great War ", the big or small 1G of this film, A people has mixed English subtitles to this one, uploaded this film, now the film size is 1.01G, B people has mixed Chinese subtitle to this one, uploaded this film, now the film size is 1.02G, and C people has mixed Russian captions to this one, has uploaded this film, now the film size is 1.01G, D people has mixed the French captions to this one, has uploaded the now big or small 1.01G of being of film of this film.Owing to being same film, everyone is a very little part to the modification of this film.Generally system is all assigned these four films as different films, therefore storing these films needs 4.05G, while adopting the scheme of data mining in this application, just can judge which content is identical, data for identical content are not preserved, therefore generally only need storage 1.05G just can, save storage space.
With reference to figure 5, the structural representation of a kind of data processing equipment embodiment tetra-provided for the application, described device can be applied in the superfluous processing that disappears to file to be stored data in distributed file system, the embodiment of the present application is set up current index in advance, described current index comprises at least one index value, each described index value is corresponding with a blocks of files of having stored respectively, wherein, the blocks of files that described index value is corresponding to it has unique identification, and described device can comprise:
Wherein, between each described blocks of files, have logical interdependency, each described blocks of files is corresponding with a copy.With reference to figure 6, be the structural representation of described file acquisition unit 501, described file acquisition unit 501 can comprise:
Wherein, described file storage resource request can be generated according to self-demand by the user, and this request comprises file data to be stored.
Wherein, described logical partitioning refers to, to described file data, the division rule default according to it adopts the method for mark as memory address division rule etc., described file data is divided into to a plurality of sections, and the starting and ending position of recording each file section, extremely the file content between end position is as a blocks of files for a reference position, and the All Files piece forms file destination, i.e. described file data to be stored.It should be noted that, after by logical partitioning, described file destination is still a complete file, and this logical partitioning is different from the division of physical address.
Index value acquiring unit 502, for obtaining respectively the target index value of each described blocks of files.
Wherein, described target index value has unique identification to its blocks of files, and each blocks of files all has a unique target index value corresponding with it, and this target index value can be with the Hash hash value representation of this document piece, therefore, described index value acquiring unit 502 can be realized in the following manner:
The hash calculated value that obtains respectively each described blocks of files is as its each self-corresponding target index value.
In the embodiment of the present application, to the account form of hash value, can be realized by application such as MD5 or SHA.
Wherein, described file acquisition unit 501 and index value acquiring unit 502 can be arranged at as shown in Figure 10 in operations server, carry out file destination obtain and the index value of each blocks of files obtains.
Index value query unit 503, be used for each described target index value, the inquiry index value consistent with described target index value in current index, when in described current index, inquiring consistent index value, trigger the first processing unit 504, otherwise, trigger the second processing unit 505.
Wherein, described current index comprises each index value of stored blocks of files, thus, when described index value query unit 503 inquires the index value consistent with described target index value in described current index, trigger described first and process 504, otherwise, trigger described the second processing unit 505.
Wherein, described index value query unit 503 can be arranged at as shown in Figure 10 in index server, carries out the inquiry of index value.
The first processing unit 504, for abandoning blocks of files corresponding to described target index value.
Wherein, when described index value query unit 503 inquires the index value consistent with described target index value in described current index, now show in described current index to have existed described target index value, the blocks of files that this target index value is corresponding is stored, therefore, blocks of files corresponding to this target index value is without being stored again, can trigger described the first processing unit 504 abandons the corresponding blocks of files of described target index value, can also this target index value be abandoned simultaneously simultaneously, realize the data superfluous purpose that disappears.
The second processing unit 505, for writing to described target index value described current index and storing blocks of files corresponding to described target index value.
Wherein, while in described index value query unit 503, not inquiring the index value consistent with described target index value in described current index, now show that the blocks of files that this target index value is corresponding is not stored, now triggering described the second processing unit 505 writes to described target index value in described current index, basic as inquiry while inquiring about for index value next time, blocks of files corresponding to described target index value stored in the disappear superfluous processing of realization to this target index value respective file piece simultaneously.
It should be noted that, in described the second processing unit 505 storage blocks of files comprise this document piece itself with and copy.
Wherein, described the first processing unit 504 and described the second processing unit 505 can be arranged at as shown in Figure 10 in operations server, carry out the operation to blocks of files and target index value thereof, as abandon or the processing such as storage.
From such scheme, a kind of data processing equipment embodiment tetra-that the application provides, carrying out index value by each blocks of files in the file destination to be stored obtains, and then the state whether be stored in current index of its index value is judged to inquiry, thereby draw whether stored state of this document piece, and then the blocks of files that stored index value is corresponding is given up, not stored blocks of files is stored and corresponding index value writes in current index by it, realization is to the superfluous processing of disappearing of blocks of files, reach the application's purpose.Can be effectively when the embodiment of the present application is applied in distributed file system to the data to be stored superfluous processing that disappeared, avoid redundant data in prior art to take the problem that a large amount of storage spaces cause the wasting of resources, effectively improve the space utilisation in space, saving resource.
With reference to figure 7, the structural representation of the second processing unit 505 described in a kind of data processing method embodiment five provided for the application, wherein, described the second processing unit 505 can comprise:
Index value writes subelement 551, for described target index value is write to described current index.
Blocks of files storing sub-units 552, for storing blocks of files corresponding to described target index value.
Concrete, with reference to figure 8, be the structural representation of blocks of files storing sub-units 552 described in the embodiment of the present application five, wherein, described blocks of files storing sub-units 552 can comprise:
Wherein, described metadata be describe its respective file piece content, blocks of files copy information (each blocks of files can comprise a plurality of copies) with and the respective file piece under the attribute information etc. of file destination.In distributed file system, described metadata can be placed in the meta data server of this system.
Wherein, described memory address comprises the first memory address of this document piece self and the second memory address of this document piece part.
It should be noted that, set in advance the blocks of files storage rule in the embodiment of the present application, the memory address obtained in described address acquisition module 5522 can be the memory address of the described target index value respective file piece that gets according to described blocks of files storage rule, and described memory address means this document piece can be stored in which storage server in distributed file system.Wherein, described blocks of files storage rule can be placed in meta data server in advance, thus, described address acquisition module 5522 is when realizing, can be by sending the address acquisition request to meta data server, after generating memory address by meta data server, by the application, this memory address is obtained.
Blocks of files memory module 5523, for described memory address, stored blocks of files corresponding to described target index value as DSB data store block.
Wherein, described blocks of files memory module 5523 specifically can realize in the following manner:
The blocks of files that described target index value is corresponding is stored with the first memory address of the blocks of files in described memory address self, and by described target index value, the copy of corresponding blocks of files is stored successively with the second memory address of the blocks of files copy in described memory address, realizes the storing process to blocks of files.
In order to improve the efficiency of search index value in described index value query unit 503, can when setting up, described current index set up its secondary index table, and described secondary index table comprises at least one index key value.For example, index value in described current index is set up to second layer concordance list, be specially: described current index is carried out to index value and sorted according to the ordering rule set in advance, choose an index value at interval of at least one index value in current index after sequence afterwards, the index value of choosing is set up to second layer concordance list, and described second layer concordance list comprises at least one index key value.Now, with reference to figure 9, the structural representation of a kind of data processing equipment embodiment six provided for the application, wherein, described index value query unit can comprise:
The first inquiry subelement 531, be used for each described target index value, the corresponding index key value of the described target index value of inquiry in described secondary index table, when inquiring the index key value corresponding with described target index value in described secondary index table, trigger the second inquiry subelement 532, otherwise, trigger described the second processing unit 505;
The second inquiry subelement 532, for sub-list corresponding with described index key value in described current index, inquire about the index value consistent with described target index value, if while in described sub-list, inquiring consistent index value, trigger described the first processing unit 504, otherwise, trigger described the second processing unit 505.
Such scheme is: to each described target index value, and the corresponding index key value of the described target index value of inquiry in described secondary index table;
When inquiring the index key value corresponding with described target index value in described secondary index table, in described current index, with in the corresponding sub-list of described index key value, inquire about the index value consistent with described target index value;
If while in described sub-list, inquiring consistent index value, the corresponding blocks of files of described target index value is abandoned, otherwise, write in described current index by described target index value and store blocks of files corresponding to described target index value;
And, when not inquiring the index key value corresponding with described target index value in described secondary index table, write in described current index by described target index value and store blocks of files corresponding to described target index value.
From such scheme, in the embodiment of the present application six, current index is set up to the secondary index table, utilize the search index mode of binary tree, improve the inquiry velocity to index value.
Wherein, in above-mentioned each embodiment, described device can also arrange data-mining module, carry out data mining in order to the index value in described current index, be that the maintenance data mining algorithm carries out the extraction of valuable information to index value, for example according to the index value in described current index, learn that the repeat usage of the blocks of files which index value is corresponding is higher.
The example of data mining process is as follows: have section to be called the film of " robot Great War ", the big or small 1G of this film, A people has mixed English subtitles to this one, uploaded this film, now the film size is 1.01G, B people has mixed Chinese subtitle to this one, uploaded this film, now the film size is 1.02G, and C people has mixed Russian captions to this one, has uploaded this film, now the film size is 1.01G, D people has mixed the French captions to this one, has uploaded the now big or small 1.01G of being of film of this film.Owing to being same film, everyone is a very little part to the modification of this film.Generally system is all assigned these four films as different films, therefore storing these films needs 4.05G, while adopting the scheme of data mining in this application, just can judge which content is identical, data for identical content are not preserved, therefore generally only need storage 1.05G just can, save storage space.
With reference to Figure 10, the structural representation of a kind of data handling system embodiment seven provided for the application, described system comprises at least one index server 1001, at least one is the operations server 1002 corresponding with each described index server respectively, one meta data server 1003, at least one storage server 1004, and as above in the data processing equipment 1005 described in any embodiment, wherein, described storage server 1004 is connected with described meta data server 1003 respectively, described storage server 1004 is in order to the storing documents piece, described meta data server 1003 is in order to deposit the metadata of the blocks of files of having stored in described storage server 1004, described storage server 1004 and described meta data server 1003 form distributed file system.
Wherein, the operations server 1002 that described index server 1001 is corresponding with it is logically separated, but it physically can be on same equipment.
In the embodiment of the present application, described data processing equipment 1005 is arranged on the index server of described operations server 1002 and correspondence thereof, as mentioned in above, described data processing equipment 1005 File acquiring units, the index value acquiring unit, the first processing unit and the second processing unit are arranged on described operations server, the index value query unit is arranged on described index server, and deposited current index in described index server 1001, contain in described storage server 1004 index value of storage file piece in described current index, wherein:
When needs carry out the storage of file data, described data processing equipment 1005 on a certain described operations server 1002 receives the file storage resource request, this document storage resource request comprises file data to be stored, data between operations server 1002 as shown in Figure 11 and described index server 1001 are moved towards shown in figure, 1005 pairs of described file datas of described data processing equipment carry out logical partitioning, obtain including the file destination of at least one blocks of files.
Afterwards, described data processing equipment 1005 obtains respectively the target index value of each described blocks of files in described file destination, and to each described target index value, the traversal queries index value consistent with described target index value in current index in described index server, when in described current index, inquiring consistent index value, the corresponding blocks of files of described target index value is abandoned and returns to the current storage information of the corresponding blocks of files of this target index value, otherwise, described target index value is write in described current index, the metadata that creates the blocks of files that described target index value is corresponding is placed in described meta data server 1003, and obtain the memory address of the described target index value respective file piece that described meta data server 1003 sends, the blocks of files that described target index value is corresponding is stored in the storage server corresponding with described memory address 1004 as DSB data store block.
Wherein, as shown in Figure 12, described data processing equipment 1005 can pass through user management module in specific implementation, task scheduling modules, superfluous module and communication module and the index module in index server of disappearing forms, and the described superfluous module that disappears comprises the piecemeal module, hash computing module and sending module, user management module is mainly used to receive user's request, communicate the request to task scheduling modules, at first the superfluous module that disappears can logically be divided file, each data block is carried out to hash calculating, and send in the index module on index server changing hash calculating, index module is carried out traversal search to the hash value sended over, again the information of search is sent back to the data processing equipment 1005 on operations server, described data processing equipment 1005 directly sends and is stored to storage server to corresponding data block by communication module by dose the storage instruction in the information of returning.
Wherein, the file storage resource request that described data processing equipment 1005 sends by its user management module accepted user, when the file storage resource request is more, the file storage resource request that can can't accept by task scheduling modules is sent to the data processing equipment 1005 of other comparatively idle operations server 1002 and is processed, thereby accelerates the disappear efficiency of superfluous processing storage operation of data.It should be noted that, described task scheduling modules can realize by load balancing module is set the equilibrium treatment of 1002 couples of users' of each operations server file storage resource request, improves the efficiency that data are processed.
It should be noted that, described meta data server 1003 is central servers, is in charge of the access to file of the NameSpace of file system and server.Meta data server is stored the metadata of whole file system.Any operation that file metadata is produced to modification, meta data server all can be recorded in transaction journal.
Wherein, described storage server has a plurality of in cluster, is generally that node exists one, is in charge of their subsidiary storages on himself node.Storage server under the commander of meta data server, carry out piece establishment, delete and copy.Meta data server is responsible for keeping and is managed the metadata of all file system.
In addition, with reference to Figure 13, be another structural representation of the embodiment of the present application seven, described system can also comprise the synchronizer 1006 be connected with each described index server 1001 respectively, wherein:
Described synchronizer 1006, carry out data synchronous operation for the current index of each the described index server 1001 to coupled.
In Figure 14, the schematic diagram of minimum 4 station servers of building for native system, wherein two is operations server and index server, and one is meta data server, and one is storage server.When the user uploads a file to server, after file enters the superfluous module that disappears, system can be carried out the fixed length piecemeal to file, and calculate the hash value of each piece, the hash computing module can send to index server to this hash and be judged, safeguards the mapping table of whole data block hash value and block address in index server.As have identical recordings, this document piece will not be preserved, and if there is not identical recordings, directly preserves.By synchronizer communication each other, realize the real-time synchronization of data between index server.Meta data server is stored the metadata of whole file system, any operation that file metadata is produced to modification, and meta data server all can be recorded in transaction journal.Storage server under the commander of meta data server, carry out piece establishment, delete and copy.In Figure 14, for the data transmission between each server in the embodiment of the present application seven flows to, below be described:
Be below the data transmission flow direction explanation between each server in Figure 14:
1. in the time of user's upload file, can upload to operations server to file.
2. operations server can be carried out logic partitioning to file, and calculates the hash value of each piece, and file hash value is sent to index server.
3. index server can judge which file hash value is to exist, and then return message is to operations server.
4. after operations server receives information, then send the storing documents request to meta data server.
5. after meta data server receives message, carry out relevant treatment, then return to corresponding information to operations server, this information content comprises which platform storage server is each blocks of files should be stored in, and the copy of each piece should be stored in which station server etc.
6. after operations server receives message, the relevant block of file is stored on corresponding storage server, now exactly file is carried out to physical division.
Deleted file the time, at first by server, to meta data server, send the deleted file request.
8. directly to storage server, send the operation of deleted file after the meta data server request of receiving.
9. storage server receives the rear directly deleted file of operation.
Hence one can see that, and operations server sends the associative operation request to metadata, and meta data server tells how operations server will be done, next just direct and storage server contact of operations server.More special for deletion action, operations server sends deletion action to meta data server, and then meta data server sends the operation of deleted file to storage server.
It should be noted that, each embodiment in this instructions all adopts the mode of going forward one by one to describe, and what each embodiment stressed is and the difference of other embodiment that between each embodiment, identical similar part is mutually referring to getting final product.
Finally, also it should be noted that, in this article, relational terms such as the first and second grades only is used for an entity or operation are separated with another entity or operational zone, and not necessarily requires or imply between these entities or operation the relation of any this reality or sequentially of existing.And, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thereby make the process, method, article or the equipment that comprise a series of key elements not only comprise those key elements, but also comprise other key elements of clearly not listing, or also be included as the intrinsic key element of this process, method, article or equipment.In the situation that not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment that comprises described key element and also have other identical element.
Above a kind of data processing method, the Apparatus and system that the application is provided is described in detail, applied specific case herein the application's principle and embodiment are set forth, the explanation of above embodiment is just for helping to understand the application's method and core concept thereof; Simultaneously, for one of ordinary skill in the art, the thought according to the application, all will change in specific embodiments and applications, and in sum, this description should not be construed as the restriction to the application.
Claims (12)
1. a data processing method, is characterized in that, sets up in advance current index, and described current index comprises at least one index value, and each described index value is corresponding with a blocks of files of having stored respectively, and described method comprises:
Obtain file destination to be stored, described file destination comprises at least one blocks of files;
Obtain respectively the target index value of each described blocks of files;
To each described target index value, the inquiry index value consistent with described target index value in described current index;
When in described current index, inquiring consistent index value, the corresponding blocks of files of described target index value is abandoned, otherwise, write in described current index by described target index value and store blocks of files corresponding to described target index value.
2. method according to claim 1, is characterized in that, describedly obtains file destination to be stored, comprising:
Receive the file storage resource request, described file storage resource request comprises file data to be stored;
Described file data is divided in logic, obtained including the file destination of at least one blocks of files.
3. method according to claim 1, is characterized in that, the described target index value that obtains each described blocks of files comprises:
The Hash hash calculated value that obtains respectively each described blocks of files is target index value separately as it.
4. method according to claim 1, is characterized in that, blocks of files corresponding to the described target index value of described storage comprises:
Create the metadata of the blocks of files that described target index value is corresponding;
Obtain the memory address of described target index value respective file piece;
With described memory address, blocks of files corresponding to described target index value stored as DSB data store block.
5. method according to claim 1, is characterized in that, in described current index, sets up the secondary index table is arranged, wherein, described to each described target index value, and the inquiry index value consistent with described target index value in current index comprises:
To each described target index value, the corresponding index key value of the described target index value of inquiry in described secondary index table;
When inquiring the index key value corresponding with described target index value in described secondary index table, in described current index with in the corresponding sub-list of described index key value, inquire about the index value consistent with described target index value, if while in described sub-list, inquiring consistent index value, carry out and described the corresponding blocks of files of described target index value is abandoned, otherwise, carry out describedly writing in described current index by described target index value and storing blocks of files corresponding to described target index value;
When not inquiring the index key value corresponding with described target index value in described secondary index table, carry out describedly writing in described current index by described target index value and storing blocks of files corresponding to described target index value.
6. a data processing equipment, is characterized in that, sets up in advance current index, and described current index comprises at least one index value, and each described index value is corresponding with a blocks of files of having stored respectively, and described device comprises:
The file acquisition unit, for obtaining file destination to be stored, described file destination comprises at least one blocks of files;
The index value acquiring unit, for obtaining respectively the target index value of each described blocks of files;
The index value query unit for to each described target index value, is inquired about the index value consistent with described target index value in current index, when in described current index, inquiring consistent index value, trigger the first processing unit, otherwise, the second processing unit triggered;
The first processing unit, for abandoning blocks of files corresponding to described target index value;
The second processing unit, for writing to described target index value described current index and storing blocks of files corresponding to described target index value.
7. device according to claim 6, is characterized in that, described file acquisition unit comprises:
The request receiving subelement, for receiving the file storage resource request, described file storage resource request comprises file data to be stored;
The logical partitioning subelement, for described file data is divided in logic, obtain including the file destination of at least one blocks of files.
8. device according to claim 6 is characterized in that:
Described index value acquiring unit, specifically for the Hash hash calculated value that obtains respectively each described blocks of files as it target index value separately.
9. device according to claim 6, is characterized in that, described the second processing unit comprises:
Index value writes subelement, for described target index value is write to described current index;
The blocks of files storing sub-units, for storing blocks of files corresponding to described target index value;
Concrete, described blocks of files storing sub-units comprises:
The metadata store module, for creating the metadata of the blocks of files that described target index value is corresponding;
Address acquisition module, for obtaining the memory address of described target index value respective file piece;
The blocks of files memory module, for described memory address, stored blocks of files corresponding to described target index value as DSB data store block.
10. device according to claim 6, is characterized in that, in described current index, sets up the secondary index table is arranged, and wherein, described index value query unit comprises:
The first inquiry subelement, be used for each described target index value, the corresponding index key value of the described target index value of inquiry in described secondary index table, when inquiring the index key value corresponding with described target index value in described secondary index table, trigger the second inquiry subelement, otherwise, trigger described the second processing unit;
The second inquiry subelement, for sub-list corresponding with described index key value in described current index, inquire about the index value consistent with described target index value, if while in described sub-list, inquiring consistent index value, trigger described the first processing unit, otherwise, trigger described the second processing unit.
A 11. data handling system, it is characterized in that, comprise at least one index server, at least one is the operations server corresponding with each described index server respectively, meta data server, at least one storage server, and data processing equipment as described as any one in claim 6 to 10, described data processing equipment is arranged on its corresponding index server of described operations server, and described current index is placed in described index server, described storage server is in order to deposit stored blocks of files, described meta data server is in order to deposit in described storage server the metadata of storage file piece.
12. system according to claim 11, is characterized in that, also comprises the synchronizer be connected with each described index server respectively, wherein:
Described synchronizer, carry out data synchronous operation for the current index of each the described index server to coupled.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013104329367A CN103455631A (en) | 2013-09-22 | 2013-09-22 | Method, device and system for processing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013104329367A CN103455631A (en) | 2013-09-22 | 2013-09-22 | Method, device and system for processing data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103455631A true CN103455631A (en) | 2013-12-18 |
Family
ID=49737994
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013104329367A Pending CN103455631A (en) | 2013-09-22 | 2013-09-22 | Method, device and system for processing data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103455631A (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104104986A (en) * | 2014-07-29 | 2014-10-15 | 小米科技有限责任公司 | Audio frequency and subtitle synchronizing method and device |
| CN105025053A (en) * | 2014-04-24 | 2015-11-04 | 苏宁云商集团股份有限公司 | Distributed file upload method based on cloud storage technology and system |
| CN105242881A (en) * | 2015-10-12 | 2016-01-13 | 创新科软件技术（深圳）有限公司 | Distributed storage system and data read-write method for same |
| CN105677746A (en) * | 2015-12-29 | 2016-06-15 | 上海爱数信息技术股份有限公司 | Database transaction operation based duplicate files merging system and method |
| CN105893433A (en) * | 2015-12-09 | 2016-08-24 | 乐视云计算有限公司 | File storage method, file access method, file storage device and file access device |
| CN106951192A (en) * | 2017-03-25 | 2017-07-14 | 广州硕点电子科技有限公司 | A kind of date storage method, apparatus and system |
| CN107544916A (en) * | 2016-06-29 | 2018-01-05 | 迈普通信技术股份有限公司 | A kind of caching method and storage device |
| CN108241640A (en) * | 2016-12-23 | 2018-07-03 | 航天星图科技（北京）有限公司 | A kind of distributed file memory method |
| WO2018165961A1 (en) * | 2017-03-17 | 2018-09-20 | 深圳市秀趣品牌文化传播有限公司 | E-commerce data processing system and method for increasing speed of accessing e-commerce platform |
| CN109062987A (en) * | 2018-06-29 | 2018-12-21 | 联动优势科技有限公司 | A kind of document handling method and device |
| CN109215169A (en) * | 2017-07-04 | 2019-01-15 | 百度在线网络技术（北京）有限公司 | Storage method, device and the equipment of travelling data |
| CN109284290A (en) * | 2018-09-20 | 2019-01-29 | 佛山科学技术学院 | A kind of method for reading data based on distributed storage space |
| CN109684293A (en) * | 2018-12-14 | 2019-04-26 | 中国人民银行清算总中心 | Mixed file storage method and device |
| CN109977074A (en) * | 2019-03-11 | 2019-07-05 | 北京东方国信科技股份有限公司 | A kind of lob data processing method and processing device based on HDFS |
| CN110569398A (en) * | 2019-09-09 | 2019-12-13 | 广州虎牙科技有限公司 | data processing method and device, computer equipment and readable storage medium |
| CN110837575A (en) * | 2019-10-31 | 2020-02-25 | 广州华多网络科技有限公司 | Method and device for generating transmission characteristic information of video image |
| CN111447265A (en) * | 2020-03-24 | 2020-07-24 | 深圳市网心科技有限公司 | File storage method, file downloading method, file processing method and related components |
| CN111580881A (en) * | 2020-04-30 | 2020-08-25 | 支付宝(杭州)信息技术有限公司 | File loading method and device and electronic equipment |
| CN111596859A (en) * | 2019-02-20 | 2020-08-28 | 慧荣科技股份有限公司 | Data storage device and data processing method |
| CN113704240A (en) * | 2021-09-23 | 2021-11-26 | 世纪龙信息网络有限责任公司 | Data deduplication method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080270436A1 (en) * | 2007-04-27 | 2008-10-30 | Fineberg Samuel A | Storing chunks within a file system |
| US20110276780A1 (en) * | 2010-05-05 | 2011-11-10 | Microsoft Corporation | Fast and Low-RAM-Footprint Indexing for Data Deduplication |
| CN102810107A (en) * | 2011-06-01 | 2012-12-05 | 英业达股份有限公司 | Processing method for repeating data |
| CN102866955A (en) * | 2012-09-14 | 2013-01-09 | 记忆科技(深圳)有限公司 | Flash data management method and system |
| CN103154950A (en) * | 2012-05-04 | 2013-06-12 | 华为技术有限公司 | Repeated data deleting method and device |
-
2013
- 2013-09-22 CN CN2013104329367A patent/CN103455631A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080270436A1 (en) * | 2007-04-27 | 2008-10-30 | Fineberg Samuel A | Storing chunks within a file system |
| US20110276780A1 (en) * | 2010-05-05 | 2011-11-10 | Microsoft Corporation | Fast and Low-RAM-Footprint Indexing for Data Deduplication |
| CN102810107A (en) * | 2011-06-01 | 2012-12-05 | 英业达股份有限公司 | Processing method for repeating data |
| CN103154950A (en) * | 2012-05-04 | 2013-06-12 | 华为技术有限公司 | Repeated data deleting method and device |
| CN102866955A (en) * | 2012-09-14 | 2013-01-09 | 记忆科技(深圳)有限公司 | Flash data management method and system |
Non-Patent Citations (2)
| Title |
|---|
| 喻波 等: "一种基于共享前缀的两级索引结构", 《计算机工程与科学》 * |
| 李超 等: "基于二级索引的重复数据删除系统中性能相关参数的量化分析研究", 《计算机研究与发展》 * |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105025053A (en) * | 2014-04-24 | 2015-11-04 | 苏宁云商集团股份有限公司 | Distributed file upload method based on cloud storage technology and system |
| CN104104986A (en) * | 2014-07-29 | 2014-10-15 | 小米科技有限责任公司 | Audio frequency and subtitle synchronizing method and device |
| CN104104986B (en) * | 2014-07-29 | 2017-08-22 | 小米科技有限责任公司 | The synchronous method and device of audio and captions |
| CN105242881B (en) * | 2015-10-12 | 2018-02-16 | 创新科软件技术（深圳）有限公司 | Distributed memory system and its data read-write method |
| CN105242881A (en) * | 2015-10-12 | 2016-01-13 | 创新科软件技术（深圳）有限公司 | Distributed storage system and data read-write method for same |
| CN105893433A (en) * | 2015-12-09 | 2016-08-24 | 乐视云计算有限公司 | File storage method, file access method, file storage device and file access device |
| CN105677746A (en) * | 2015-12-29 | 2016-06-15 | 上海爱数信息技术股份有限公司 | Database transaction operation based duplicate files merging system and method |
| CN107544916B (en) * | 2016-06-29 | 2020-04-24 | 迈普通信技术股份有限公司 | Caching method and storage device |
| CN107544916A (en) * | 2016-06-29 | 2018-01-05 | 迈普通信技术股份有限公司 | A kind of caching method and storage device |
| CN108241640A (en) * | 2016-12-23 | 2018-07-03 | 航天星图科技（北京）有限公司 | A kind of distributed file memory method |
| WO2018165961A1 (en) * | 2017-03-17 | 2018-09-20 | 深圳市秀趣品牌文化传播有限公司 | E-commerce data processing system and method for increasing speed of accessing e-commerce platform |
| CN106951192A (en) * | 2017-03-25 | 2017-07-14 | 广州硕点电子科技有限公司 | A kind of date storage method, apparatus and system |
| CN109215169A (en) * | 2017-07-04 | 2019-01-15 | 百度在线网络技术（北京）有限公司 | Storage method, device and the equipment of travelling data |
| CN109215169B (en) * | 2017-07-04 | 2021-10-29 | 百度在线网络技术（北京）有限公司 | Driving data storage method, device and equipment |
| CN109062987A (en) * | 2018-06-29 | 2018-12-21 | 联动优势科技有限公司 | A kind of document handling method and device |
| CN109284290B (en) * | 2018-09-20 | 2022-04-26 | 佛山科学技术学院 | Data reading method based on distributed storage space |
| CN109284290A (en) * | 2018-09-20 | 2019-01-29 | 佛山科学技术学院 | A kind of method for reading data based on distributed storage space |
| CN109684293A (en) * | 2018-12-14 | 2019-04-26 | 中国人民银行清算总中心 | Mixed file storage method and device |
| CN111596859A (en) * | 2019-02-20 | 2020-08-28 | 慧荣科技股份有限公司 | Data storage device and data processing method |
| CN109977074A (en) * | 2019-03-11 | 2019-07-05 | 北京东方国信科技股份有限公司 | A kind of lob data processing method and processing device based on HDFS |
| CN109977074B (en) * | 2019-03-11 | 2021-04-27 | 北京东方国信科技股份有限公司 | HDFS-based LOB data processing method and device |
| CN110569398A (en) * | 2019-09-09 | 2019-12-13 | 广州虎牙科技有限公司 | data processing method and device, computer equipment and readable storage medium |
| CN110837575A (en) * | 2019-10-31 | 2020-02-25 | 广州华多网络科技有限公司 | Method and device for generating transmission characteristic information of video image |
| CN111447265A (en) * | 2020-03-24 | 2020-07-24 | 深圳市网心科技有限公司 | File storage method, file downloading method, file processing method and related components |
| CN111580881A (en) * | 2020-04-30 | 2020-08-25 | 支付宝(杭州)信息技术有限公司 | File loading method and device and electronic equipment |
| CN111580881B (en) * | 2020-04-30 | 2023-06-16 | 支付宝(杭州)信息技术有限公司 | File loading method and device and electronic equipment |
| CN113704240A (en) * | 2021-09-23 | 2021-11-26 | 世纪龙信息网络有限责任公司 | Data deduplication method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103455631A (en) | Method, device and system for processing data | |
| KR102240557B1 (en) | Method, device and system for storing data | |
| US10002148B2 (en) | Memory-aware joins based in a database cluster | |
| CN104077423B (en) | Consistent hash based structural data storage, inquiry and migration method | |
| JP6479020B2 (en) | Hierarchical chunking of objects in a distributed storage system | |
| US8683112B2 (en) | Asynchronous distributed object uploading for replicated content addressable storage clusters | |
| CN104794123B (en) | A kind of method and device building NoSQL database indexes for semi-structured data | |
| CN106066896B (en) | Application-aware big data deduplication storage system and method | |
| CN103473277B (en) | The Snapshot Method and device of file system | |
| CN103067461B (en) | A kind of metadata management system of file and metadata management method | |
| CN105933376A (en) | Data manipulation method, server and storage system | |
| CN104881466B (en) | The processing of data fragmentation and the delet method of garbage files and device | |
| CN102169507A (en) | Distributed real-time search engine | |
| US10310904B2 (en) | Distributed technique for allocating long-lived jobs among worker processes | |
| CN105701096A (en) | Index generation method, data inquiry method, index generation device, data inquiry device and system | |
| CN106201771A (en) | Data-storage system and data read-write method | |
| CN104820717A (en) | Massive small file storage and management method and system | |
| CN103631924A (en) | Application method and system for distributive database platform | |
| CN105005611A (en) | File management system and file management method | |
| CN106294352A (en) | A kind of document handling method, device and file system | |
| CN105808622A (en) | File storage method and device | |
| CN106095871A (en) | A kind of method and device setting up data base directory structure | |
| CN104184812A (en) | Multi-point data transmission method based on private cloud | |
| CN104182487A (en) | Unified storage method supporting various storage modes | |
| CN103823846A (en) | Method for storing and querying big data on basis of graph theories |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20131218 |
|
| RJ01 | Rejection of invention patent application after publication |