CN105095294B - The method and device of isomery copy is managed in a kind of distributed memory system - Google Patents
The method and device of isomery copy is managed in a kind of distributed memory system Download PDFInfo
- Publication number
- CN105095294B CN105095294B CN201410206795.1A CN201410206795A CN105095294B CN 105095294 B CN105095294 B CN 105095294B CN 201410206795 A CN201410206795 A CN 201410206795A CN 105095294 B CN105095294 B CN 105095294B
- Authority
- CN
- China
- Prior art keywords
- copy
- file
- isomery
- request parameter
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
Abstract
The present invention provides the method and devices that isomery copy is managed in a kind of distributed memory system, and wherein method includes: to obtain the write request parameter of the copy for storage file；The location information of each storage server in distributed memory system is obtained from meta data server according to write request parameter；The copy of file is converted to the isomery copy of the file of multiple and different formats according to write request parameter according to preassigned format；According to the location information of the storage server obtained from meta data server, the isomery copy of the file for the multiple and different formats being converted to is respectively stored in specified storage server.One file can be stored multiple and different isomery copies by embodiment through the invention respectively in multiple storage servers, when reading the isomery copy of file, it can according to need and read corresponding isomery copy, the working efficiency that user handles data can be effectively improved.
Description
Technical field
The present invention relates to isomery copy is managed in technical field of data storage more particularly to a kind of distributed memory system
Method and device.
Background technique
Big data has application in every field such as communication, internet, finance, medical treatment, military project, science.Big data analysis
Compared to traditional data warehouse applications, have the characteristics that data volume is big, query analysis is complicated.And it is directed to distributed file system
For, Yao Guanli's is mass data naturally.Just with the flow analysis data instance of telecommunications, can all there are new data daily, often
A area also has different data, for a long time such as 10 years data, will be mass data.If operator wants root
Go out rule according to these data statistics, specifies more reasonable billing scheme, then, these huge data are stored in distributed document
In system, how to improve reading efficiency is an a matter of great account feelings.
In the related technology, these huge tables are stored in a manner of some big files, are with addition when storing at the beginning
The mode write is stored in, and when needing to read, then sequentially read and.If when write-in being write by capable mode, sequentially read
It is exactly to be read out with capable format when coming., will be more troublesome but if business is wanted to analyze the data of each column, it needs
It reads out and reintegrates, data volume is bigger, and workload is also bigger.
Summary of the invention
In view of above-mentioned technical problem, overcome above-mentioned technical problem or at least be partially solved the present invention provides one kind on
State in the distributed memory system of technical problem manage isomery copy method and device, by by a file in multiple storages
Multiple and different isomery copies is stored in server respectively, when reading the isomery copy of file, can according to need reading pair
The isomery copy answered can effectively improve the working efficiency that user handles data.
According to one aspect of the present invention, a kind of method that isomery copy is managed in distributed memory system, packet are provided
It includes:
Obtain the write request parameter of the copy for storage file；
The location information of each storage server is obtained from meta data server according to the write request parameter；
The copy of the file is converted to according to preassigned format according to the write request parameter multiple and different
The isomery copy of the file of format；
It is multiple by what is be converted to according to the location information of the storage server obtained from the meta data server
The isomery copy of the file of different-format is respectively stored in specified storage server.
Optionally, the copy of the file is being converted to according to preassigned format according to the write request parameter
Before the step of isomery copy of the file of multiple and different formats, the method also includes:
Judge whether to enable the isomery copy of the file according to the write request parameter；
If enabling the isomery copy of the file, enters and pressed the copy of the file according to the write request parameter
The step of being converted to the isomery copy of the file of multiple and different formats according to preassigned format.
Optionally, the copy by the file is converted to the institute of multiple and different formats according to preassigned format
The step of stating the isomery copy of file are as follows:
The copy of the file is deposited according to the storage of row mode, the storage of column mode or block mode according to the write request parameter
The format of storage is converted into the isomery copy of the file of multiple and different formats, and caches the multiple and different formats being converted to
The isomery copy of the file.
Optionally, the method also includes:
Obtain the read request parameter for reading the copy of file；
The location information of the storage server to be read is obtained from the meta data server according to the read request parameter；
Judge whether to enable the isomery copy of the file according to read request parameter；
If enabling the isomery copy of the file, the storage service to be read is obtained according to from the meta data server
The location information of device reads the isomery copy of the file from specified storage server.
Optionally, the write request parameter includes: the copy of file handle, document misregistration amount, file size and file
Format, wherein the format of the copy of the file includes: to store according to the storage of row mode, according to column mode or according to block mode
Storage；
The read request parameter includes: the reading of the copy of file handle, document misregistration amount, file size and the file
Mode, the read mode include reading copy according to row mode, copy being read according to column mode or is read according to block mode
Copy.
According to another aspect of the invention, the dress that isomery copy is managed in a kind of distributed memory system is additionally provided
It sets, comprising:
Write request obtains module, for obtaining the write request parameter for being used for the copy of storage file；
First position obtains module, for obtaining each storage from meta data server according to the write request parameter
The location information of server；
Format converting module, for according to the write request parameter by the copy of the file according to preassigned format
It is converted to the isomery copy of the file of multiple and different formats；
Copy memory module, for being believed according to the position of the storage server obtained from the meta data server
Breath, the isomery copy of the file for the multiple and different formats being converted to is respectively stored in specified storage server.
Optionally, described device further include:
First judgment module, for judging whether to enable the isomery copy of the file according to the write request parameter；Such as
Fruit enables the isomery copy of the file, then triggers the format converting module；.
Optionally, the format converting module be further used for according to the write request parameter by the copy of the file by
The isomery pair of the file of multiple and different formats is converted into according to the format that the storage of row mode, the storage of column mode or block mode store
This, and cache the isomery copy of the file for the multiple and different formats being converted to.
Optionally, described device further include:
Read request obtains module, for obtaining the read request parameter of the copy for reading file；
The second position obtains module, for being deposited according to the read request parameter from what meta data server acquisition to be read
Store up the location information of server；
Second judgment module, for judging whether to enable the isomery copy of the file according to the read request parameter；
Copy read module, if the isomery copy for enabling the file, according to from the meta data server
The location information for obtaining the storage server to be read, reads the isomery copy of the file from specified storage server.
Optionally, the write request parameter includes: the copy of file handle, document misregistration amount, file size and file
Format, wherein the format of the copy of the file includes: to store according to the storage of row mode, according to column mode or according to block mode
Storage；
The read request parameter includes: the reading of the copy of file handle, document misregistration amount, file size and the file
Mode, the read mode include reading copy according to row mode, copy being read according to column mode or is read according to block mode
Copy.
The beneficial effects of the present invention are: in an embodiment of the present invention, since client-side program can be by the copy of file
The isomery copy of the file of multiple and different formats is converted to according to preassigned format, and multiple and different by what is be converted to
The isomery copy of the file of format is respectively stored in specified storage server, by the way that a file is deposited multiple isomery pairs
This is mutually to convert between each isomery copy, and multiple isomery copies play the role of redundancy, to improve file system
Reliability.And the format of the isomery copy of the file due to storing in each storage server is different, allows user's root
The isomery copy of file is read according to needs (such as the needs analyzed according to data) selection, such as when user needs by row analysis text
When the copy of part, so that it may which the isomery copy of the file by the storage of row mode is read in selection；When user needs by column Study document
Copy when, so that it may selection read by column mode storage file isomery copy；When user needs by block analysis file
When copy (some multi-dimensional spreadsheet data), so that it may which the isomery copy of the file by the storage of block mode is read in selection.Due to user
It can according to need the isomery copy that file is read in selection, the working efficiency that user handles data can be effectively improved, it is especially suitable
For the systematicness data manipulation of magnanimity and the management of large database concept.
Detailed description of the invention
Fig. 1 shows storage files in the method for managing isomery copy in distributed memory system in the embodiment of the present invention
One of the flow chart of isomery copy；
Fig. 2 indicates the schematic diagram of the copy redundancy structure in the embodiment of the present invention in distributed file system；
Fig. 3 indicates the schematic diagram of the isomery copy of the file in the embodiment of the present invention by the storage of row mode；
Fig. 4 indicates the schematic diagram of the isomery copy of the file in the embodiment of the present invention by the storage of column mode；
Fig. 5 indicates the schematic diagram of the isomery copy of the file in the embodiment of the present invention by the storage of block mode；
Fig. 6 indicates to manage storage file in the method for isomery copy in distributed memory system in the embodiment of the present invention
The two of the flow chart of isomery copy；
Fig. 7 indicates to manage reading file in the method for isomery copy in distributed memory system in the embodiment of the present invention
The flow chart of isomery copy；
Fig. 8 shows repair isomery pair in the method for managing isomery copy in distributed memory system in the embodiment of the present invention
This flow chart；And
Fig. 9 indicates the block diagram for managing the device of isomery copy in the embodiment of the present invention in distributed memory system.
Specific embodiment
A kind of method that isomery copy is managed in distributed memory system is disclosed according to one aspect of the present invention, first
Obtain the write request parameter of the copy for storage file；Then it is each deposited according to write request parameter from meta data server acquisition
Store up the location information of server；Then the copy of file is converted to according to preassigned format according to write request parameter more
The isomery copy of the file of a different-format；The last location information according to the storage server obtained from meta data server,
The isomery copy of the file for the multiple and different formats being converted to is respectively stored in specified storage server.
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
It is fully disclosed to those skilled in the art.
As shown in Figure 1, being stored to be managed in distributed memory system in the embodiment of the present invention in the method for isomery copy
One of the flow chart of the isomery copy of file, the executing subject of each step can be client-side program, this method in this method
Include:
Step S101, the write request parameter of the copy for storage file is obtained.
In an embodiment of the present invention, the copy of file and file content having the same, Replication technology is a kind of data
Data item is replicated more parts and is individually placed on multiple nodes (storage server) of distributed system, to improve by administrative mechanism
The reliability and access efficiency of system.
Specifically, the client program calls file system writes data-interface in step s101, to obtain for depositing
Store up file copy write request parameter, wherein the write request parameter include: file handle, document misregistration amount, file size and
The format etc. of the copy of file, wherein the format of the copy of file includes: according to row mode storage (reference can be made to Fig. 3), according to column
Mode stores (reference can be made to Fig. 4) or stores (reference can be made to Fig. 4) according to block mode.
Step S103, each storage service in distributed memory system is obtained from meta data server according to write request parameter
The location information of device.
Under normal conditions, in order to ensure the reliability of file system, the copy of the file of different-format can be stored in
In different storage servers, i.e., each storage server stores a kind of copy of the file of format, such as in write request parameter
It further include the format of the copy of file, client-side program can be inquired according to the write request parameter by meta data server at this time
The location information of multiple storage servers corresponding with the format of the copy of file, and multiple storage servers that inquiry is obtained
Location information as file copy storage location.
As shown in Fig. 2, for the schematic diagram of the copy redundancy structure in distributed file system in the embodiment of the present invention,
Middle distributed file system includes: client-side program 201, meta data server 203 and connect with client-side program 201
Multiple storage servers 205 wherein can recorde each storage server in distributed file system in meta data server 203
205 location information, storage server 205 is for the isomery copy of storage file or the copy of file.
Step S105, the copy of file is converted to according to preassigned format according to write request parameter multiple and different
The isomery copy of the file of format.
In an embodiment of the present invention, the isomery copy of file, which refers to, carries out converting for data format to the copy of file
The copy arrived, the content in the isomery copy of file is identical as the content in the copy of file and file, wherein each isomery
Copy can be adapted for different application scenarios (such as by the copy of row mode storage file, by the pair of column mode storage file
Originally, by copy of block mode storage file etc.), it can also be equally mutually redundant between each isomery copy.
Optionally, in one embodiment of the invention, in step s105, according to write request parameter by the copy of file
The isomery pair of the file of multiple and different formats is converted into according to the format of the storage of row mode, the storage of column mode or the storage of block mode
This, and cache the isomery copy of the file for the multiple and different formats being converted to.
As shown in figure 3, in the embodiment of the present invention by the schematic diagram of the copy of the file of row mode storage, wherein file
Copy 2 in content stored in a manner of table, which includes multiple rows 21 and multiple column 23, wherein row 21 include a row,
B row, c row and d row, column 23 are including 1 column, 2 column, 3 column and 4 column, the content that wherein a row records: 50,28,352 and 120, b row note
The content of record: 21,99,66 and 112, the content of c row record: 32,52,123 and 13, the content of d row record: 65,23,87 and
344.Optionally, the copy 2 of file as shown in Figure 3 is converted into according to the format that row mode stores such as figure in step s105
The isomery copy 3 of file shown in 3, and the isomery copy 3 for the file being converted to is cached, what the isomery copy 3 of file recorded
Content: 50,28,352,120,21,99,66,112,32,52,123,13,65,23,87 and 344, it is of course possible to understanding,
The particular content recorded in the copy 2 of file is not limited in an embodiment of the present invention.
As shown in figure 4, in the embodiment of the present invention by the schematic diagram of the copy of the file of column mode storage, wherein file
Copy 2 in content stored in a manner of table, table includes multiple rows 21 and multiple column 23, and row 21 includes a row, b row, c
Row and d row, column 23 are arranged including 1 column, 2 column, 3 and 4 column, wherein 1 arranges the content of record: 50,21,32 and 65,2 column record interior
Hold: 28, the content of 99,52 and 23,3 column records: the 352, content of 66,123 and 87,4 column records: 120,112,13 and 344.It can
The copy 2 of file as shown in Figure 4 is converted into as shown in Figure 4 by selection of land according to the format that column mode stores in step s105
File isomery copy 4, and cache the isomery copy 4 of file being converted to, recorded in the isomery copy 4 of file interior
Hold: 50,21,32,65,28,99,52,23,352,66,123,87,120,112,13 and 344, it is of course possible to understanding,
The particular content recorded in the copy 2 of file is not limited in the embodiment of the present invention.
As shown in figure 5, indicating the schematic diagram of the copy of the file in the embodiment of the present invention by the storage of block mode, Chinese
Content in the copy 2 of part is stored in a manner of table, which includes multiple rows 21 and multiple column 23, and wherein row 21 includes a
Row, b row, c row and d row, column 23 include 1 column, 2 column, 3 column and 4 column, are divided to obtain to multiple rows and multiple column in figure multiple
Block, the content that wherein block 25 records: 50,28,21 and 99, the content that block 27 records: 352,120,66 and 112, what block 29 recorded
Content: 123,13,87 and 344, the content that block 31 records: 32,52,65 and 23.It optionally, in step s105 will be such as Fig. 5 institute
The copy 2 of the file shown is converted into the isomery copy 5 of file as shown in Figure 5 according to the format that block mode stores, and caches and turn
The isomery copy 5 for the file got in return, the content recorded in the isomery copy 5 of file: 50,28,21,99,352,120,66,
112,32,52,65,23,123,13,87 and 344, it is of course possible to which understanding does not limit text in an embodiment of the present invention
The particular content recorded in the copy 2 of part.Of course, it should be understood that only listing the side of three kinds of formats conversion in Fig. 3~Fig. 5
Formula does not limit the mode of format conversion in an embodiment of the present invention.In the specific implementation, user can be as the case may be
To adjust the mode of format conversion.
Optionally, in an embodiment of the present invention, after executing the step S101, S103, Huo Zhejin can be entered step
Enter step S105, or enter step S103 and step S105 simultaneously, i.e., does not limit above-mentioned step in an embodiment of the present invention
Sequencing between rapid S103 and step S105.
Step S107, more by what is be converted to according to the location information of the storage server obtained from meta data server
The isomery copy of the file of a different-format is respectively stored in specified storage server.
With continued reference to Fig. 2, client-side program 201 can be according to the storage server obtained from meta data server 203
The isomery copy of the file for the multiple and different formats being converted to is respectively stored in specified storage server by location information
On 205, i.e., the isomery copy of the file of different-format is stored in each storage server 205.
In an embodiment of the present invention, since client-side program can turn the copy of file according to preassigned format
Get the isomery copy of the file of multiple and different formats in return, and by the isomery copy of the file for the multiple and different formats being converted to
It is respectively stored in specified storage server, by the way that a file is deposited multiple isomery copies, and each isomery copy is interior
Appearance is the same, and multiple isomery copies play the role of redundancy, to improve the reliability of file system.And of the invention
User can according to need the copy that file is read in selection in embodiment, such as when user needs the isomery pair by row Study document
This when, so that it may which the copy of the file by the storage of row mode is read in selection；When user needs the isomery copy by column Study document
When, so that it may when the copy by the file of column mode storage is read in selection；When user needs the isomery copy by block analysis file
When (some multi-dimensional spreadsheet data), so that it may which the isomery copy of the file by the storage of block mode is read in selection.Since user can be with
The isomery copy of file is read in selection as needed, can effectively improve the working efficiency that user handles data.
As shown in fig. 6, being stored to be managed in distributed memory system in the embodiment of the present invention in the method for isomery copy
The two of the flow chart of the copy of file, the difference with one of the flow chart of copy of storage file shown in Fig. 1 is, in Fig. 6
In before step S105, method further include: step S109, client-side program judge whether enable file isomery copy；Such as
Fruit enables the isomery copy of file, then enters step S105；If not enabling the isomery copy of file, enter step
S111, in step S111, client-side program, will according to the location information for obtaining each storage server from meta data server
The copy of file is respectively stored in specified storage server.Optionally, the embodiment of the present invention client-side program can be with
According to the parameter of user's input to determine whether enabling the isomery copy of file, it is of course possible to understanding, in reality of the invention
Apply the actual conditions for not limiting in example and judging whether to enable the isomery copy of file.
As shown in fig. 7, being read to be managed in distributed memory system in the embodiment of the present invention in the method for isomery copy
The flow chart of the isomery copy of file, when storing documentary copy or file in the storage server in distributed memory system
Isomery copy when, this method further include:
Step S113, the read request parameter for reading the copy of file is obtained.
Optionally, in an embodiment of the present invention, read request parameter includes: that file handle, document misregistration amount, file are long
It spends, the read mode of the copy of file, wherein read mode includes that row mode reads copy, column mode reads copy or block mould
Formula reads copy.It include that row mode reads pair such as when user needs the copy by row Study document, in the read request parameter
This；It include that column mode reads copy when user needs the copy by column Study document, in the read request parameter；When user needs
By block analysis file copy when, include that block mode reads copy in the read request parameter.Due to including reading in read request parameter
Modulus formula, so that user can according to need the copy that file is read in selection.
Step S115, the location information of the storage server to be read is obtained from meta data server according to read request parameter.
Optionally, the pair for having the file stored in storage server and the storage server is recorded in meta data server
The corresponding relationship of this format, such as the copy of file stored in storage server A is the isomery pair stored according to row mode
This；The copy of the file stored in storage server B is the isomery copy stored according to column mode；It is stored in storage server C
The copy of file be the isomery copy stored according to block mode.
Step S117, judge whether the isomery copy of enabling file.
If enabling the isomery copy of the file, enter step S119, in step S119, client-side program according to
The location information that the storage server to be read is obtained from meta data server reads the different of file from specified storage server
Structure copy.Such as client-side program will read the isomery copy stored according to row mode, client-side program can be according to metadata
Record has the corresponding relationship of the format of the copy of file in storage server in server, and inquiry obtains the position of storage server A
Confidence breath reads the isomery copy stored according to row mode then in step S119 from storage server A.
If not enabling the isomery copy of the file, S121 is entered step, in step S121, client-side program
According to the location information for obtaining the storage server to be read from meta data server, from appointing in specified multiple storage servers
The copy of file is read in storage server of anticipating.By the format of the copy of the file stored in multiple storage servers
It does not convert, the copy format of the file stored in multiple storage servers is identical, therefore can be from multiple storage servers
Any one storage server in read file copy.
In an embodiment of the present invention in order to improve the reliability that isomery copy stores, can be stored in storage server
Isomery copy when go wrong, repair process is carried out to the isomery copy.As shown in figure 8, being the embodiment of the present invention
The flow chart for repairing isomery copy in the method for isomery copy is managed in middle distributed memory system, is included the following steps:
Step S801, server initiates copy repair process.
Optionally, in an embodiment of the present invention, there are two the trigger conditions of isomery copy progress repair process:
One of trigger condition, client-side program request read to have found specified copy when copy (for example being row mode)
It loses (may be disk be broken or corresponding storage server closes), server initiates copy repair place at this time
Reason.
Trigger condition two, periodic detection Disk State inside file system, discovery has disk failures or removal, reaching
To after ageing time, corresponding copy repair process is also initiated.
Step S803, server judges whether to enable isomery copy, if it is, entering step S805；If
It is no, then enter step S809.
If enabling isomery copy, S805 is entered step, in step S805, server takes from other storages
Corresponding isomery copy is read in business device, subsequently into step S807.
Step S807, server is by the isomery copy of the file read from other storage servers, conversion cost
The corresponding isomery copy of storage server, and new isomery copy is stored on the disk in specified storage server.
Optionally, the data recovering algorithms in step S807 about isomery copy, are described as follows:
(1) row -> column, the conversion of column -> row, it is only necessary to know line number (or columns), two-dimensional matrix is subjected to transposition operation.
Such as Fig. 3~Fig. 5, data have specific location information in the table, such as can indicate data by line number and columns
Location information, therefore, when carrying out row -> column or the conversion of column -> row, according to the line number or columns of data, by Two-Dimensional Moment
Battle array carry out transposition operation.
(2) row -> block, the conversion of block -> row.When row -> block conversion, according to ranks number and block number, by corresponding row data
It takes out, blocking data.Such as Fig. 3~Fig. 5, data have a specific location information in the table, for example, can by line number,
Columns and block number indicate that the location information of data therefore, can be according to line number, columns and block number when row -> block conversion, will be right
The row data answered are taken out, blocking data.
(3) column -> block, block -> column conversion regime is similar with (2), no longer applies state herein.
Step S809 can be by the copy direct copying of other storage servers extremely if not enabling isomery copy
In this storage server.
The device that isomery copy is managed in a kind of distributed memory system is also disclosed according to another aspect of the invention.
As shown in figure 9, the device 900 for managing isomery copy in distributed memory system includes:
Write request obtains module 901, for obtaining the write request parameter of the copy for storage file, the wherein write request
Parameter includes: file handle, document misregistration amount and file size；
First position obtains module 903, for obtaining distributed storage system from meta data server according to write request parameter
The location information of each storage server in system.As shown in Fig. 2, for the pair in distributed file system in the embodiment of the present invention
The schematic diagram of this redundancy structure wherein can store the location information of multiple storage servers in meta data server 203.
Format converting module 905, for according to the write request parameter by the copy of file according to preassigned format
It is converted to the isomery copy of the file of multiple and different formats.In an embodiment of the present invention, the isomery copy of file is
Refer to the copy being converted to that data format is carried out to the copy of file, content that the isomery copy of file is recorded and file
Content in copy and file is identical, and each isomery copy is suitable for different application scenarios and (such as stores text by row mode
The copy of part, by the copy of column mode storage file, by copy of block mode storage file etc.), can also equally be mutually redundant.
For details, reference can be made to Fig. 3~Fig. 5.
Copy memory module 907, for will convert according to the location information for obtaining storage server from meta data server
The isomery copy of the file of obtained multiple and different formats is respectively stored in specified storage server.With continued reference to figure
2, client-side program 201 can will be converted to according to the location information of the storage server obtained from meta data server 203
The isomery copies of file of multiple and different formats be respectively stored in specified storage server 205.
In an embodiment of the present invention, since client-side program can turn the copy of file according to preassigned format
Get the isomery copy of the file of multiple and different formats in return, and by the isomery copy of the file for the multiple and different formats being converted to
It is respectively stored in specified storage server, it, can phase between isomery copy by the way that a file is deposited multiple isomery copies
Mutually conversion, so that multiple isomery copies play the role of redundancy, to improve the reliability of file system.And in reality of the invention
Applying user in example can according to need the copy that file is read in selection, such as when user needs the copy by row Study document,
It may be selected by reading the isomery copy of the file by the storage of row mode；When user needs the copy by column Study document, just
It can choose the isomery copy for reading the file stored by column mode；When user needs the isomery copy (one by block analysis file
A little multi-dimensional spreadsheet data) when, so that it may the isomery copy of the file by the storage of block mode is read in selection.Since user can basis
The isomery copy for selecting to read file is needed, the working efficiency that user handles data can be effectively improved.
Optionally, in another embodiment of the present invention, described device 900 further include:
First judgment module 909, for judging whether to enable the isomery copy of the file according to write request parameter；If
The copy of heterogeneous file is enabled, then triggering form conversion module 905；If not enabling the isomery copy of the file, touch
Copy memory module 907 is sent out according to the location information for obtaining each storage server from meta data server, by the file
Copy is respectively stored in specified storage server.
Optionally, in another embodiment of the present invention, format converting module 905 is further used for being joined according to write request
The copy of the file is converted into multiple and different lattice according to the format of the storage of row mode, the storage of column mode or the storage of block mode by number
The isomery copy of the file of formula, and cache the isomery copy of the file for the multiple and different formats being converted to.
Optionally, in another embodiment of the present invention, described device 900 further include:
Read request obtains module 911, for obtaining the read request parameter of the copy for reading file；
The second position obtains module 913, for be read according to the read request parameter from meta data server acquisition
Storage server location information；
Second judgment module 915, for judging whether to enable the isomery copy of the file according to read request parameter；
Copy read module 917, if the isomery copy for enabling the file, according to from the Metadata Service
Device obtains the location information of the storage server to be read, and the isomery copy of the file is read from specified storage server；
And if the isomery copy for not enabling the file, basis obtain the storage to be read from the meta data server
The location information of server reads the copy of any one file from specified multiple storage servers.
Optionally, in another embodiment of the present invention, the write request parameter includes: file handle, document misregistration
Amount, file size and file copy format, wherein the format of the copy of the file include: according to the storage of row mode, by
It stores according to column mode or is stored according to block mode；The read request parameter includes: that file handle, document misregistration amount, file are long
Degree and the file copy read mode, the read mode include row mode read copy, column mode read copy or
Person's block mode reads copy.
Above-described is the preferred embodiment of the present invention, it should be pointed out that the ordinary person of the art is come
It says, can also make several improvements and retouch under the premise of not departing from principle of the present invention, these improvements and modifications also exist
In protection scope of the present invention.
Claims (10)
1. managing the method for isomery copy in a kind of distributed memory system characterized by comprising
Obtain the write request parameter of the copy for storage file；
The position of each storage server in distributed memory system is obtained from meta data server according to the write request parameter
Information；
The copy of the file is converted to multiple and different formats according to preassigned format according to the write request parameter
The file isomery copy, the isomery copy of file refers to the pair being converted to that data format is carried out to the copy of file
This, the content in the isomery copy of file is identical as the content in the copy of file and file, and each isomery copy is suitable for
Different application scenarios；
It is multiple and different by what is be converted to according to the location information of the storage server obtained from the meta data server
The isomery copy of the file of format is respectively stored in specified storage server.
2. the method according to claim 1, wherein according to the write request parameter by the copy of the file
Before the step of being converted to the isomery copy of the file of multiple and different formats according to preassigned format, the method
Further include:
Judge whether to enable the isomery copy of the file according to the write request parameter；
If enabling the isomery copy of the file, enter the copy of the file according to the write request parameter according to pre-
The step of isomery copy for the file that first specified format is converted to multiple and different formats.
3. method according to claim 1 or 2, which is characterized in that it is described according to the write request parameter by the file
The copy file that is converted to multiple and different formats according to preassigned format isomery copy the step of are as follows:
According to the write request parameter by the copy of the file according to the storage of row mode, the storage of column mode or the storage of block mode
Format is converted into the isomery copy of the file of multiple and different formats, and caches the described of the multiple and different formats being converted to
The isomery copy of file.
4. the method according to claim 1, wherein the method also includes:
Obtain the read request parameter for reading the copy of file；
The location information of the storage server to be read is obtained from the meta data server according to the read request parameter；
Judge whether to enable the isomery copy of the file according to the read request parameter；
If enabling the isomery copy of the file, the storage server to be read is obtained according to from the meta data server
Location information reads the isomery copy of the file from specified storage server.
5. according to the method described in claim 4, it is characterized in that, the write request parameter includes: file handle, document misregistration
Amount, file size and file copy format, wherein the format of the copy of the file include: according to the storage of row mode, by
It stores according to column mode or is stored according to block mode；
The read request parameter includes: the reading mould of the copy of file handle, document misregistration amount, file size and the file
Formula, the read mode include reading copy according to row mode, copy being read according to column mode or is read according to block mode secondary
This.
6. managing the device of isomery copy in a kind of distributed memory system characterized by comprising
Write request obtains module, for obtaining the write request parameter for being used for the copy of storage file；
First position obtains module, for being obtained in distributed memory system according to the write request parameter from meta data server
The location information of each storage server；
Format converting module, for being converted the copy of the file according to preassigned format according to the write request parameter
The isomery copy of the file of multiple and different formats is obtained, the isomery copy of file, which refers to, carries out data lattice to the copy of file
The copy of formula being converted to, the content in the isomery copy of file is identical as the content in the copy of file and file, often
A isomery copy is suitable for different application scenarios；
Copy memory module will for the location information according to the storage server obtained from the meta data server
The isomery copy of the file for the multiple and different formats being converted to is respectively stored in specified storage server.
7. device according to claim 6, which is characterized in that described device further include:
First judgment module, for judging whether to enable the isomery copy of the file according to the write request parameter；If opened
With the isomery copy of the file, then the format converting module is triggered.
8. device according to claim 6 or 7, which is characterized in that the format converting module is further used for according to institute
It states write request parameter and is converted into the copy of the file according to the format of the storage of row mode, the storage of column mode or the storage of block mode
The isomery copy of the file of multiple and different formats, and cache the isomery of the file for the multiple and different formats being converted to
Copy.
9. device according to claim 6, which is characterized in that described device further include:
Read request obtains module, for obtaining the read request parameter of the copy for reading file；
The second position obtain module, for according to the read request parameter from the meta data server acquisition to be read described in deposit
Store up the location information of server；
Second judgment module, for judging whether to enable the isomery copy of the file according to the read request parameter；
Copy read module obtains if the isomery copy for enabling the file according to from the meta data server
The location information of the storage server to be read reads the isomery copy of the file from specified storage server.
10. device according to claim 9, which is characterized in that the write request parameter includes: that file handle, file are inclined
The format of the copy of shifting amount, file size and file, wherein the format of the copy of the file include: according to row mode storage,
It stores according to column mode or is stored according to block mode；
The read request parameter includes: the reading mould of the copy of file handle, document misregistration amount, file size and the file
Formula, the read mode include reading copy according to row mode, copy being read according to column mode or is read according to block mode secondary
This.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410206795.1A CN105095294B (en) | 2014-05-15 | 2014-05-15 | The method and device of isomery copy is managed in a kind of distributed memory system |
| PCT/CN2014/086658 WO2015172478A1 (en) | 2014-05-15 | 2014-09-16 | Method and apparatus for heterogeneous replica management in distributed storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410206795.1A CN105095294B (en) | 2014-05-15 | 2014-05-15 | The method and device of isomery copy is managed in a kind of distributed memory system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105095294A CN105095294A (en) | 2015-11-25 |
| CN105095294B true CN105095294B (en) | 2019-08-09 |
Family
ID=54479238
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410206795.1A Active CN105095294B (en) | 2014-05-15 | 2014-05-15 | The method and device of isomery copy is managed in a kind of distributed memory system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN105095294B (en) |
| WO (1) | WO2015172478A1 (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107180031B (en) * | 2016-03-09 | 2021-04-09 | 华为技术有限公司 | Distributed storage method and device, and data processing method and device |
| CN107451138A (en) * | 2016-05-30 | 2017-12-08 | 中兴通讯股份有限公司 | A kind of distributed file system storage method and system |
| CN107515863A (en) * | 2016-06-15 | 2017-12-26 | 上海宽带技术及应用工程研究中心 | The method and system that a kind of SDN clusters based on distributed data base are realized |
| CN106202396A (en) * | 2016-07-08 | 2016-12-07 | 乐视控股（北京）有限公司 | Object storage method and object storage system |
| CN107295070B (en) * | 2017-05-31 | 2019-10-29 | 上海交通大学 | The distributed coding caching laying method and system of file size isomery |
| CN109428861A (en) * | 2017-08-29 | 2019-03-05 | 阿里巴巴集团控股有限公司 | Network communication method and equipment |
| CN107807793B (en) * | 2017-10-27 | 2019-11-08 | 清华大学 | The storage of data copy isomery and access method in distributed computer storage system |
| CN108304471A (en) * | 2017-12-28 | 2018-07-20 | 中国银联股份有限公司 | A kind of Heterogeneous data storage method and Heterogeneous data storage device |
| CN108334565A (en) * | 2018-01-15 | 2018-07-27 | 贵州易鲸捷信息技术有限公司 | A kind of data mixing storage organization, data store query method, terminal and medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6601072B1 (en) * | 2000-06-21 | 2003-07-29 | International Business Machines Corporation | Method and system for distribution of application data to distributed databases of dissimilar formats |
| CN102004743B (en) * | 2009-09-02 | 2013-08-14 | 中国银联股份有限公司 | System and method for copying data among heterogeneous databases |
| US20120215763A1 (en) * | 2011-02-18 | 2012-08-23 | Microsoft Corporation | Dynamic distributed query execution over heterogeneous sources |
| CN102314375A (en) * | 2011-03-18 | 2012-01-11 | 北京神州数码思特奇信息技术股份有限公司 | Heterogeneous database storage unified interface and database access method |
-
2014
- 2014-05-15 CN CN201410206795.1A patent/CN105095294B/en active Active
- 2014-09-16 WO PCT/CN2014/086658 patent/WO2015172478A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| CN105095294A (en) | 2015-11-25 |
| WO2015172478A1 (en) | 2015-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105095294B (en) | The method and device of isomery copy is managed in a kind of distributed memory system | |
| JP5656563B2 (en) | Document management system, document management system control method, and program | |
| JP6697392B2 (en) | Transparent discovery of semi-structured data schema | |
| JP6118515B2 (en) | System for associative memory update | |
| CN104618482B (en) | Access method, server, conventional memory device, the system of cloud data | |
| CN103890709B (en) | Key value database based on caching maps and replicates | |
| US9158843B1 (en) | Addressing mechanism for data at world wide scale | |
| US9020802B1 (en) | Worldwide distributed architecture model and management | |
| CN104301360B (en) | A kind of method of logdata record, log server and system | |
| CN103714123B (en) | Enterprise's cloud memory partitioning object data de-duplication and restructuring version control method | |
| US7788303B2 (en) | Systems and methods for distributed system scanning | |
| US9280381B1 (en) | Execution framework for a distributed file system | |
| CN104408111B (en) | A kind of method and device of deleting duplicated data | |
| CN109997126A (en) | Event-driven is extracted, transformation, loads (ETL) processing | |
| JP2005122702A5 (en) | ||
| CN105068758B (en) | Towards the Distributed File System Data I/O optimization methods of parallel data acquisition | |
| CN106294352B (en) | A kind of document handling method, device and file system | |
| FR3031604A1 (en) | APPARATUS AND METHODS FOR SYNCHRONIZATION OF DATA | |
| EP2780834A1 (en) | Data processing service | |
| CN104184812B (en) | A kind of multipoint data transmission method based on private clound | |
| CA2834864A1 (en) | Database system and method | |
| WO2011146452A1 (en) | Data storage and processing service | |
| CN105095247B (en) | symbol data analysis method and system | |
| CN107122361A (en) | Data mover system and method | |
| CN103795811A (en) | Information storage and data statistical management method based on meta data storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |