CN106375404B - Data storage control method, date storage method, data capture method and device - Google Patents

Data storage control method, date storage method, data capture method and device Download PDF

Info

Publication number
CN106375404B
CN106375404B CN201610777273.6A CN201610777273A CN106375404B CN 106375404 B CN106375404 B CN 106375404B CN 201610777273 A CN201610777273 A CN 201610777273A CN 106375404 B CN106375404 B CN 106375404B
Authority
CN
China
Prior art keywords
data
group
request
client
data storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610777273.6A
Other languages
Chinese (zh)
Other versions
CN106375404A (en
Inventor
曹立勋
施威特
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201610777273.6A priority Critical patent/CN106375404B/en
Publication of CN106375404A publication Critical patent/CN106375404A/en
Application granted granted Critical
Publication of CN106375404B publication Critical patent/CN106375404B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/14—Protection against unauthorised use of memory or access to memory
    • G06F12/1458—Protection against unauthorised use of memory or access to memory by checking the subject access rights

Abstract

The invention discloses a kind of data storage control method, data storage, data capture method and devices.In the present invention, in data storage procedure, after host node in EC group receives the data storage authority request that client is sent, authority request batch operation timing mark is stored for the data of client, member node of the client into EC group sends the second data storage request, and operation timing mark and the corresponding fragment of the data to be stored are carried in data storage request.In data acquisition, client is according to the first data acquisition request received, the member node in the EC group and EC grouping at place is determined for the data of request, data are obtained from the member node in the EC group determined, it identifies and is compared from all member nodes acquisition operation timing in EC group, if the operation timing mark of the member node in EC group is identical, the data got are returned.The present invention can reduce the expense of host node.

Description

Data storage control method, date storage method, data capture method and device
Technical field
The present invention relates to technical field of data storage more particularly to a kind of data storage control methods, data storage, data Acquisition methods and device.
Background technique
Distributed memory system is frequently with distributed hashtable (Distributed Hash Table, abbreviation DHT) algorithm Storage resource in cluster is divided.The hash space of entire distributed type assemblies is from 0~232- 1, this integer space by etc. It is divided into N number of subregion, the cryptographic Hash range that each subregion includes is equal.Each data is issued to distribution with Key mark and deposits Storage system is stored, and each Key obtains a cryptographic Hash by Hash calculation, which can be mapped to certain of hash space On a subregion.
As shown in Figure 1, the hash space that 4 physical nodes provide is divided into 20 subregions, from P0~P19, physics section Point is as shown in the figure with the corresponding relationship of dummy node.The value of Data Identification key is k1, which is mapped after Hash calculation To P0 subregion.What A~T was represented is dummy node, and the data in each subregion are stored on corresponding dummy node.For example, reflecting The data for being mapped to P0 subregion are stored on A node, are mapped in the data storage to B node of P1 subregion.
Erasure Code mode can be used in distributed memory system and carry out data redundancy storage, an Erasure Code group (i.e. EC group) corresponds to a subregion in cluster.It include host node and common section in member node in one EC group Point, host node can control multiple client or system front and back request Concurrency.In writing data flow, client will Data object is sent to host node, and data object is cut into data fragmentation by the EC algorithm that host node is used according to system, calculates Fragment is verified, is then deposited other nodes that the data fragmentation of generation and verification fragment are sent in EC group by host node Storage.As shown in Fig. 2, the data ABCDEFGHI for being identified as key=k1 is mapped to P0 subregion, the master in the corresponding EC group of P0 subregion Node OSD1 is split into 3 fragments ABC, DEF, GHI, calculates verification according to EC algorithm (such as RS algorithm) Fragment YXY, QGC, and this 3 data fragmentations and 2 verification fragments are stored respectively in all member nodes in the EC group In (OSD1~OSD5).In time data stream journey, read data request is sent to host node by client, and host node is into EC group Other nodes obtain the data fragmentation needed, and return to client and read successful data.
In above-mentioned time data stream journey, fragment and distribution, above-mentioned reading data are carried out to the data that client is sent by host node In process, data are obtained from host node to EC group membership, cause data traffic to concentrate on host node, host node expense is excessive.
Summary of the invention
The embodiment of the invention provides a kind of data storage control method and devices.
Data storage control method provided in an embodiment of the present invention, applied to point using Erasure Code memory mechanism Cloth storage system, the memory space of the distributed memory system are divided into multiple subregions, the corresponding EC of a subregion Group, an interior EC group includes host node, this method comprises:
Host node in EC group receives the data that client is sent and stores authority request；
The host node stores authority request according to the data, is that the data of the client store authority request distribution Operation timing mark, the operation timing mark send data for the member node into the EC group by the client and deposit Storage request；
The host node is responded to client returned data storage permission, and the operation of distribution is carried in the response Timing mark.
Optionally, after the data storage authority request that the host node reception client in the EC group is sent, according to institute State data storage authority request before, further includes: for the data storage authority request corresponding to Data Identification, the master Node determines whether that the data storage operations of identical data mark do not complete also；If so, the number being then currently received It is hung up according to storage authority request, and after the completion of the data storage operations not completed also, releases the data storage permission Request, which releases, to be hung up.
Optionally, after the data storage authority request that the host node reception client in the EC group is sent, according to institute State data storage authority request before, further includes: the host node determines whether the corresponding subregion of the EC group can be used；If can not With then response of the host node to client returned data storage authority distribution failure.
Optionally, after the host node is responded to client returned data storage permission, further includes: the main section Point receives the data storage operations completion notice that the client is sent, and the data storage operations completion notice is the client After end identifies the transmission data storage request of the member node into the EC group according to the operation timing, confirm in the EC group Member node complete data storage operations after send.
Optionally, the operation timing is identified as operation timing number, and the operation timing numbers the sequencing according to distribution It is incremented by.
Host node device provided in an embodiment of the present invention is deposited applied to using the distribution of Erasure Code memory mechanism Storage system, the memory space of the distributed memory system are divided into multiple subregions, the corresponding EC group of a subregion, and one It include host node in EC group characterized by comprising
First receiving module, the data for receiving client transmission store authority request；
Distribution module is that the data of the client store authority request for storing authority request according to the data Batch operation timing mark, the operation timing mark send number for the member node into the EC group by the client It is requested according to storage；
Sending module carries distribution in the response for responding to client returned data storage permission Operation timing mark.
Optionally, the distribution module is also used to: after receiving the data storage authority request that client is sent, being directed to Data Identification corresponding to the data storage authority request, it is determined whether the data storage operations for having identical data to identify are also not It completes；If so, then the data being currently received storage authority request is hung up, and in the also unfinished data storage After the completion of operation, releases the data storage authority request and release hang-up.
Optionally, the distribution module is also used to: after receiving the data storage authority request that client is sent, basis Before the data storage authority request, determine whether the corresponding subregion of the EC group can be used；If unavailable, to the client Hold the response of returned data storage authority distribution failure.
Optionally, further includes: the second receiving module, for responding it to client returned data storage permission Afterwards, the data storage operations completion notice that the client is sent is received, the data storage operations completion notice is the visitor After family end identifies the transmission data storage request of the member node into the EC group according to the operation timing, the EC group is confirmed Interior member node completes what data storage operations were sent later.
Optionally, further include data recovery module, be used for: receiving the data acquisition request that client device is sent, it is described The mark and data that the data of request are carried in data acquisition request restore instruction；Restore to indicate according to the data Carry out data recovery to the member node in the EC group of place；After data are successfully recovered, the data of acquisition request acquisition；It returns Data are successfully recovered response, and the data are successfully recovered in response and carry the data got.
Optionally, the operation timing is identified as operation timing sequence number, and the operation timing, sequence number was according to the successive of distribution Sequentially.
Device provided in an embodiment of the present invention for communication, applied to the distribution using Erasure Code memory mechanism Formula storage system, the memory space of the distributed memory system are divided into multiple subregions, and a subregion corresponds to an EC group, It include host node in one EC group, which includes: that the device includes: transceiver, processor and memory；
The memory, for storing computer program instructions；
The processor is coupled to the memory, for reading the computer program instructions of the memory storage, and Execute above-mentioned data storage control method.
In the above embodiment of the present invention, the host node in EC group receives the data that client is sent and stores authority request Afterwards, authority request batch operation timing mark is stored for the data of the client, and stores permission to client returned data It responds, the operation timing mark of distribution is carried in the response.Since operation timing mark is used for by client into EC group Member node send data storage request, thus on the one hand data storage is managed by host node is same, another aspect Allow client directly to store data in the member node in the EC group, compared with prior art, avoids data It is sent to the process of host node, so as to reduce the expense of host node.
The embodiment of the invention provides a kind of date storage method and devices.
Date storage method provided in an embodiment of the present invention, applied to the distribution using Erasure Code memory mechanism Storage system, the memory space of the distributed memory system are divided into multiple subregions, the corresponding EC group of a subregion, and one It include host node in a EC group, this method comprises:
For client according to the first data storage request received, the data for request storage determine corresponding EC group；
Host node of the client into the EC group sends data and stores authority request, receives the host node and returns Data storage permission response, when carrying the operation of data acquisition authority request distribution of the promising client in the response Sequence mark；
Member node of the client into the EC group sends the second data storage request, the data storage request In carry operation timing mark and the corresponding fragment of the data to be stored.
Optionally, the fragment carried in second data storage request includes data fragmentation or verification fragment；The visitor Member node of the family end into the EC group is sent before the second data storage request, further includes: the client to it is described to The data of storage carry out fragment, and verification fragment is calculated according to obtained data fragmentation.
Optionally, the client is that data to be stored determine corresponding EC group, comprising: the client is according to request The mark of the data of storage determines subregion corresponding with the mark, and determines corresponding EC group according to the subregion determined.
Optionally, it after member node of the client into the EC group sends the second data storage request, also wraps Include: it is complete that the client receives the data storage operations that the member node in the EC group is returned according to the second data storage request At notice；After the client receives the data storage operations completion notice of the return of all member nodes in the EC group, Host node into the EC group sends data storage operations completion notice.
Another embodiment of the present invention provides client device, applied to using Erasure Code memory mechanism distribution Formula storage system, the memory space of the distributed memory system are divided into multiple subregions, and a subregion corresponds to an EC group, It include host node in one EC group, comprising:
First determining module, for being determining pair of data of request storage according to the first data storage request received The EC group answered；
Authority acquiring module sends data for the host node into the EC group and stores authority request, receives the master The data storage permission that node returns responds, and the data storage authority request distribution of the promising client is carried in the response Operation timing mark；
Data storage request module sends the second data storage request for the member node into the EC group, described The operation timing mark and the corresponding fragment of the data to be stored are carried in data storage request.
Optionally, the fragment carried in second data storage request includes data fragmentation or verification fragment；The visitor Family end equipment further include: data processing module, for sending the second data storage request in the member node into the EC group Before, fragment is carried out to the data to be stored, and verification fragment is calculated according to obtained data fragmentation.
Optionally, first determining module is specifically used for: according to the mark of the data of request storage, the determining and mark Corresponding subregion, and corresponding EC group is determined according to the subregion determined.
Optionally, the data storage request module is also used to: the member node in Xiang Suoshu EC group sends the second data After storage request, it is complete to receive the data storage operations that the member node in the EC group is returned according to the second data storage request At notice, after receiving the data storage operations completion notice that all member nodes in the EC group return, Xiang Suoshu EC group Interior host node sends data storage operations completion notice.
Another embodiment of the present invention provides device for communication, applied to using Erasure Code memory mechanism Distributed memory system, the memory space of the distributed memory system are divided into multiple subregions, and a subregion is one corresponding EC group, an interior EC group includes host node, which includes: transceiver, processor and memory；
The memory, for storing computer program instructions；
The processor is coupled to the memory, for reading the computer program instructions of the memory storage, and Execute above-mentioned date storage method.
In the above embodiment of the present invention, client is according to the first data storage request received, for request storage Data determine corresponding EC group, and the host node into the EC group sends data and stores authority request, receive the number that host node returns According to storage permission response, the operation timing mark of the data acquisition authority request distribution of the promising client is carried in response； Member node of the client into EC group sends the second data storage request, and operation timing mark is carried in data storage request Knowledge and the corresponding fragment of the data to be stored.As can be seen that being used for by client to EC group since the operation timing identifies Interior member node sends data storage request, to be on the one hand managed by host node is same to data storage, another party Face allows client directly to store data in the member node in the EC group, compared with prior art, avoids number According to the process for being sent to host node, so as to reduce the expense of host node.
The embodiment of the invention provides a kind of data capture method and devices.
Data capture method provided in an embodiment of the present invention, comprising:
Client according to the first data acquisition request received, be EC group where the data of request determine and Member node in EC grouping；
The client obtains data from the member node in the EC group determined；
The client obtains operation timing mark from all member nodes in the EC group, and is compared, if institute The operation timing mark for stating the member node in EC group is identical, then returns to response corresponding with first data acquisition request, The data got are carried in the response.
Optionally, if the operation timing mark of the member node in the EC group is not identical, further include: the client Host node into the EC group sends data acquisition request, and the data of request are carried in the data acquisition request Mark and data restore instruction, and the data recovery request is for triggering the host node to the member node in the EC group of place The recovery of carry out data and after data are successfully recovered return institute's request data；The client receives host node and returns The data returned are successfully recovered response, and the data are successfully recovered in response and carry the data got；The client returns Response corresponding with first data acquisition request carries the data got in the response.
Optionally, host node of the client into the EC group sends data recovery request, comprising: the client According in the operation timing of the member node in the maximum quantity of recoverable fragment and EC group mark, the time is distributed In the quantity of rear and identical member node, judge whether to meet recovery condition；If meeting recovery condition, the client to Host node in the EC group sends data recovery request；If being unsatisfactory for recovery condition, the method also includes: the client End returns to data acquisition request failure response corresponding with first data acquisition request.
Another embodiment of the present invention provides client device, applied to using Erasure Code memory mechanism distribution Formula storage system, the memory space of the distributed memory system are divided into multiple subregions, and a subregion corresponds to an EC group, It include host node in one EC group, comprising: it is characterised by comprising:
Second determining module, for determining institute for the data of request according to the first data acquisition request received EC group and the EC grouping in member node；
Module is obtained, for obtaining data from the member node in the EC group determined, and from the institute in the EC group There is member node to obtain operation timing mark；
Respond module compares for the operation timing mark to all member nodes in the EC group got Compared with if the operation timing mark of the member node in the EC group is identical, return is corresponding with first data acquisition request Response, carry the data got in the response.
Optionally, the respond module is also used to: if the operation timing mark of the member node in the EC group is not identical, Then the host node into the EC group sends data acquisition request, and the data of request are carried in the data acquisition request Mark and data restore instruction, the data recovery request be used for trigger the host node in the EC group of place member save The carry out data recovery of point and the data that institute's request is returned after data are successfully recovered；And it receives host node and returns Data be successfully recovered response, the data are successfully recovered in response and carry the data got；And it returns and described the The corresponding response of one data acquisition request carries the data got in the response.
Another embodiment of the present invention provides device for communication, applied to using Erasure Code memory mechanism Distributed memory system, the memory space of the distributed memory system are divided into multiple subregions, and a subregion is one corresponding EC group, an interior EC group includes host node, which includes: that the device includes: transceiver, processor and memory；
The memory, for storing computer program instructions；
The processor is coupled to the memory, for reading the computer program instructions of the memory storage, and Execute above-mentioned data capture method.
In the above embodiment of the present invention, client is request according to the first data acquisition request received Data determine the member node in the EC group and EC grouping at place, obtain number from the member node in the EC group determined According to；The client was identified and is compared from all member nodes acquisition operation timing in EC group, if the member in EC group The operation timing mark of node is identical, then returns to response corresponding with first data acquisition request, carry in the response There are the data got.As can be seen that client directly can obtain data from the member node in EC group, and simultaneously can root According to the comparison identified to the operation timing, guarantee the requirement of availability of data, compared with prior art, reduces opening for host node Pin.
Detailed description of the invention
Fig. 1 is DHT algorithm schematic diagram in the prior art；
Fig. 2 is to write data flow diagram in the prior art；
Fig. 3 is data Stored Procedure schematic diagram provided in an embodiment of the present invention；
Fig. 4 is an example schematic of data Stored Procedure provided in an embodiment of the present invention；
Fig. 5 is data acquisition flow diagram provided in an embodiment of the present invention；
Fig. 6 is an example schematic of data acquisition process provided in an embodiment of the present invention；
Fig. 7 is the structural schematic diagram of host node device provided in an embodiment of the present invention；
The structural schematic diagram of the respectively client device provided in an embodiment of the present invention of Fig. 8,9；
Figure 10, Figure 11, Figure 12 are respectively the structural schematic diagram of communication device provided in an embodiment of the present invention.
Specific embodiment
The embodiment of the invention provides a kind of storage of data and acquisition methods, can be applied to store using Erasure Code The distributed memory system of mechanism, the memory space of the distributed memory system are divided into multiple subregions, and a subregion is corresponding One EC group, an interior EC group includes host node and ordinary node.
In the embodiment of the present invention, data can not be transmitted to full dose the host node in EC group, data fragmentation and distribution by Client carries out, and the host node in EC group controls data operating right, to reduce host node bandwidth consumption.
The host node of each EC group safeguards the operation timing mark of the EC group.Operation timing, mark was for identifying a data Storage operation.It is realized to simplify technology, sequence number can be used in operation timing mark, i.e. the operation timing numbers.Operation timing compiles Number can according to the sequencing of distribution be incremented by.
When the host node in EC group receive client transmission have modification generic operation (such as data storage operations) from visitor When family end issues, the host node in EC group distributes an operation timing number to client, is compiled the operation timing by client Number and request data be issued to each member node in the EC group.Operation timing number is similar to versions of data, can be one A shaping numerical value, each modification class IO (input/output) of EC group can be such that operation timing number increases, when client will request After being issued to each member node of EC group, the operation timing number that each member node of EC group is carried according to request executes in order Request, to guarantee all modifications generic operation sequence consensus that all member nodes of the EC group execute, so that each EC group membership Data on node are consistent.
For example, host node of the client first into EC group, which is sent, obtains timing volume when client has a write request Number message (for example the message can be described as data storage authority request), host node in EC group according to subregion current state and Whether there is request collision to determine whether to number to the request batch operation timing and notifies client.Client obtains timing number While, data are split into the data fragmentation of needs by the EC algorithm used according to system, and calculate verification fragment, are divided After area's operating right, each member node into EC group sends data fragmentation to client respectively or verification fragment carries out data Persistence.
The embodiment of the present invention is described in detail with reference to the accompanying drawing.Following embodiment is identified as behaviour with the operation timing Make to be described for timing number.
It is data Stored Procedure schematic diagram provided in an embodiment of the present invention, which, which describes client, number referring to Fig. 3 According to storage authority acquiring process when needing to store and the data storage procedure after getting storage permission.As schemed Show, which may include following steps:
Step 301: client is directed to data to be stored, and determines corresponding EC group, and the host node hair into the EC group Data are sent to store authority request.
In the step, client can be after receiving data storage request, and the data for request storage determine corresponding EC Group.By taking monitoring system as an example, which may be from video capture device (such as camera).The data storage is asked It seeks the data storage request that may be based on user or other application and generates, the embodiment of the present invention is without limitation.
In distributed memory system, data can be used<key, value>data structure, wherein key is data mark Know, value is the content of the data.In the step, the mark (key) that client treats the data (value) of storage is breathed out Obtained cryptographic Hash is mapped to a subregion in the memory space of distributed memory system, corresponding to the subregion by uncommon operation EC group be the data to be stored storage location.It can carry in the data storage authority request that client is sent wait store The key of data carries out the cryptographic Hash that Hash operation obtains to the key.
Step 302: after the host node receives the data storage authority request of client transmission, for the data of the client Store authority request batch operation timing number.
In the step, host node, can be first to for concurrent data storage request before batch operation timing number Mutual exclusion is carried out, to avoid conflict.Specifically, host node can first determine whether for the data to be stored mark (such as key or The cryptographic Hash of key), whether a upper data storage request is carrying out or waiting for (can be according to the data being currently received The cryptographic Hash of the key or the key of the data to be stored carried in request are stored to judge: whether having for identical key or is somebody's turn to do The data storage request of the cryptographic Hash of key), if so, the data storage request being currently received is hung up, one is waited on this It after the completion of data storage request, releases and hangs up for the data storage request that is suspended, distribute timing number (such as at this for it On the basis of the corresponding operation timing number of a upper data storage request plus 1).Certainly, it is stored for the data of identical data The mutual exclusion processing of request can also be carried out in client-side.When it is implemented, the data that host node can will be identified for identical data Storage request is handled according to chronological order, and then can and be advanced for the data storage request of different data mark Row processing.
Further, host node can determine whether the corresponding subregion of EC group can be used before batch operation timing number, If unavailable, response of the host node to the storage authority distribution failure of client returned data.The state of the corresponding subregion of EC group Can include: available mode, down state, data synchronous regime etc..The physical node that the state of subregion can include according to subregion State and the data consistency strategy change that provides of system external, when there are when physics node failure, system is by root in subregion Determine whether subregion can externally provide service according to data consistency strategy, to determine subregion state.For example, if existing in subregion Physical node failure and when can not restore, the state of the subregion is unavailable.
Step 303: the operation timing number of distribution is carried on data storage permission response and returns to the visitor by the host node Family end.
Step 304: after client receives the data storage permission response of host node return, member's section into the EC group Point sends the second data storage request, wherein the operation timing for carrying the data acquisition authority request distribution of the promising client compiles Number and data to be stored corresponding to fragment.
Wherein, fragment is carried out for data to be stored, and verification fragment is calculated according to obtained data fragmentation Operation carried out in client, client can carry out the operation of data fragmentation after receiving data storage request.
Member node in EC group carries out data storage processing according to the data storage request received, and can be further After completing data storage operations, operation completion notification is stored to the client returned data.Further, client is receiving After the data storage operations completion notice that all member nodes in EC group return, the host node into the EC group sends data Store operation completion notification.
By taking Fig. 1 and distributed memory system shown in Fig. 2 as an example, asked as shown in figure 4, client receives the first data storage It asks, wherein<key is carried, value>, wherein key=k1, value are expressed as ABCDEFGHI.Client splits value At 3 fragments ABC, DEF, GHI, verification fragment YXY, QGC are calculated, and Hash calculation is carried out to the k1, it is true according to cryptographic Hash Fixed corresponding subregion is P0.Host node OSD1 of the client into the corresponding EC group of P0 subregion sends data and stores authority request (step 401) carries the cryptographic Hash of k1 in the request.Host node OSD1 receive to data store authority request after, first Write operation mutual exclusion processing is carried out, it is also untreated complete currently without other data storage authority request if being directed to the cryptographic Hash of k1 At being then data storage authority request batch operation timing number, and return to the client by data memory response and (walk It is rapid 402).
After client receives data storage permission response, each member node into the EC group sends data storage respectively Request, wherein the data (step 403) for carrying operation timing number and being sent to respective nodes, for example, data ABC is sent Node OSD1 is given, data DEF is sent to OSD2, data GHI is sent to OSD3, and so on.Optionally, if fragment number Amount is less than the number of nodes in EC group, then client is also required to numbering the operation timing into all members section being sent in EC group Point.
By above description as can be seen that being managed by host node is same to data storage, client can directly by Data are stored in the member node in the EC group, compared with prior art, avoid the process for sending the data to host node, from And the expense of host node can be reduced.
It is data acquisition flow diagram provided in an embodiment of the present invention referring to Fig. 5, which describes client request Obtain the process of data.As shown, the process may include following steps:
Step 501: client is where the data of request determine according to the first data acquisition request received Member node in EC group and EC grouping.
Wherein, Data Identification key and data storage location relevant information (ratio are carried in the first data acquisition request Such as deviant of data storage).In the step, client to the Data Identification key carried in the first data acquisition request into Row Hash operation determines that corresponding EC is grouped according to the cryptographic Hash that operation obtains, according to what is carried in the first data acquisition request The storage location relevant information of data determines node of the data fragmentation of institute's request where in the EC group.For example, being asked The mark key for obtaining data is asked to fall in subregion P1, the node of carrying subregion P1 is A, B, C, D, E, F, and client calculates this The initial address of key falls in the C node of subregion P1 plus the deviant of the data of institute's request, it is determined that institute's request Node where data fragmentation is C node.
Step 502: client obtains data from the member node in the EC group determined.
In the step, the transmittable data acquisition request of client gives the member node in the EC group, carries in the request The mark and location dependent information of the data of request.
Step 503: client obtains operation timing mark from all member nodes in EC group, and is compared, if EC The operation timing mark of member node in group is identical, then is transferred to step 504.
In the step, client can all member nodes into EC group send operation timing number acquisition request.Having When body is implemented, for storing the member node of data of request, client can be by a request message from corresponding Node obtains data and obtains the operation timing number in respective nodes.When member node in EC group receives acquisition operation After the request of sequence number, the client will be returned to sequential operation number current on node.
The operation timing number of member nodes reflections all in EC group is compared by client, only when EC group it is all at The timing number of member's node is consistent, just shows that the data of this EC group are available.
Step 504: client returns to corresponding with the first data acquisition request response, carries and gets in the response Data.
If the comparison result in step 503 is that the operation timing number of the member node in EC group is inconsistent, show this The data of EC group are unavailable.In some embodiments, client can abandon the data got, terminate this data acquisition stream Journey, client returned data obtain the response of failure；In further embodiments, client can also trigger the master in the EC group Node carries out data recovery process.
In some embodiments, client triggering host node carries out the process of data recovery can include: client is to EC group Interior host node sends data acquisition request, and the instruction that the data of institute's request are carried out with data recovery is carried in the request The mark of the data of information and institute's request, may also include storage location relevant information.Host node is according to the request, touching The member node sent out in the EC group carries out data recovery to corresponding data, and is asked after data are successfully recovered to client return Seek the data of acquisition.Client returns to sound corresponding with first data acquisition request after receiving the data that host node returns It answers, the data got is carried in the response.
In other embodiments, client can also first trigger host node and carry out data recovery, revert in data After function, then initiate data acquisition request.Specifically, client triggering host node carries out the process of data recovery can include: client It holds the host node into EC group to send data recovery request, the mark for being requested the data restored is carried in the request, may be used also Including storage location relevant information.For host node according to the request, the member node triggered in the EC group counts corresponding data According to recovery, and the response after data are successfully recovered to client returned data recovery member.Client receives host node and returns The response that is successfully recovered of data after, initiate data acquisition request according to mode described in Fig. 5, and obtain from respective nodes Data.
Optionally in above-mentioned data recovery procedure, client can be it is first determined whether meet recovery condition, if meeting recovery Condition, then the host node into EC group sends data recovery request, if being unsatisfactory for recovery condition, client is returned and the first number According to the corresponding data acquisition request failure response of acquisition request.Wherein, client can be according to the maximum quantity of recoverable fragment In the operation timing of (quantity is related to the memory mechanism of use) and the member node in EC group mark, the distribution time exists Afterwards and the quantity of identical member node, judge whether to meet recovery condition.M verification collapsible mould is added for K data fragmentation The EC mechanism of formula, at least needs K fragment that could restore data.For example, for using RAID5 memory mechanism, if Operation timing numbers maximum and identical membership (can restore data greater than the fragment quantity that can restore data in such cases Fragment quantity be equal to 2), then meet data restore condition, be otherwise unsatisfactory for data restore condition.
By taking Fig. 1 and distributed memory system shown in Fig. 2 as an example, as shown in fig. 6, client receives data acquisition request, Wherein carry Data Identification key and data offset, wherein key=k1.Client carries out Hash calculation, root to the k1 It determines that corresponding subregion is P0 according to cryptographic Hash, corresponding EC group is determined according to subregion P0, and asked according to data offset determination The node OSD1 and node OSD2 for asking the data of acquisition to be stored in the EC group.Member node OSD1 of the client into the EC group Data acquisition request is sent with node OSD2, the cryptographic Hash of k1 is carried in the request, the node OSD3 into the EC group to section Point OSD5 sends the request (step 601a) for obtaining operation timing number.Node OSD1 and node OSD2 are according to the request received Data reading processing is carried out, and returns to the operation timing number on the data read and node, node OSD3 to node OSD5 According to the operation timing number (step 601b) on the request return node received.Client receives the behaviour that each node returns After making timing number, confirm that the operation timing number of all nodes is consistent, then returned data obtains response, wherein carrying acquisition The data arrived, so far this data acquisition is completed.
Through above description as can be seen that in data acquisition, client can be directly from member's section in EC group Point obtains data, and can guarantee the requirement of availability of data according to the comparison identified to the operation timing simultaneously, with the prior art It compares, reduces the expense of host node.
By the EC technical solution that the above embodiment of the present invention can be seen that the prior art compared with prior art In, for the EC mode of K+M, it is K that client, which sends size of data to host node, and host node distributes data point to each member of EC group Piece size is K+M-1, and system total flow is K+K+M-1.And the above embodiment of the present invention is used, host node only is needing to carry out Con current control is carried out to request under the scene of mutual exclusion control, by client executing, system always flows for the calculating and distribution of data fragmentation Amount is K+M, i.e. the flow summation of client distribution data fragmentation to each node of EC group, it can be seen that under K+M mode, the present invention Embodiment can save the system bandwidth of K-1.
Based on the same technical idea, the embodiment of the invention also provides a kind of host node device and client devices.
It is the structural schematic diagram of host node device provided in an embodiment of the present invention, the equipment application is in use referring to Fig. 7 The distributed memory system of Erasure Code memory mechanism is, it can be achieved that the process that previous embodiment describes.As shown, this sets It is standby can include: the first receiving module 701, distribution module 702, sending module 703 further may also include the second reception mould Block 704 and data recovery module 705, in which:
First receiving module 701, the data for receiving client transmission store authority request；
Distribution module 702 is asked for storing authority request according to the data for the data storage permission of the client Batch operation timing is asked to identify, the operation timing mark is sent by the client for the member node into the EC group Data storage request；
Sending module 703 carries distribution in the response for responding to client returned data storage permission Operation timing mark.
Optionally, distribution module 702 can also be used in: after receiving the data storage authority request that client is sent, needle To Data Identification corresponding to data storage authority request, it is determined whether the data storage operations for having identical data to identify are also It does not complete；If so, then hanging up the data being currently received storage authority request, and deposited in the also unfinished data After the completion of storage operation, releases the data storage authority request and release hang-up.
Optionally, distribution module 702 can also be used in: after receiving the data storage authority request that client is sent, root Before storing authority request according to the data, determine whether the corresponding subregion of the EC group can be used；If unavailable, to the visitor Family end returned data stores the response of authority distribution failure.
Optionally, the second receiving module 704, for connecing after being responded to client returned data storage permission The data storage operations completion notice that the client is sent is received, the data storage operations completion notice is the client root According to the operation timing identify the member node into the EC group send data storage request after, confirm in the EC group at Member's node completes what data storage operations were sent later.
Optionally, data recovery module 705 are used for: receiving the data acquisition request that client device is sent, the data The mark and data that the data of request are carried in acquisition request restore instruction；Restore instruction to institute according to the data The carry out data recovery of member node in EC group；After data are successfully recovered, the data of acquisition request acquisition；Returned data It is successfully recovered response, the data are successfully recovered in response and carry the data got.
It is the structural schematic diagram of client device provided in an embodiment of the present invention, the equipment application is in use referring to Fig. 8 The distributed memory system of Erasure Code memory mechanism is, it can be achieved that the data Stored Procedure that previous embodiment describes.Such as figure It is shown, the equipment can include: the first determining module 801, authority acquiring module 802, data storage request module 803, further Ground may also include data processing module 804, in which:
First determining module 801, for according to the first data storage request received, the data for request storage to be determined Corresponding EC group；
Authority acquiring module 802 sends data for the host node into the EC group and stores authority request, described in reception The data storage permission that host node returns responds, and the data storage authority request point of the promising client is carried in the response The operation timing mark matched；
Data storage request module 803 sends the second data storage request, institute for the member node into the EC group It states and carries the operation timing mark and the corresponding fragment of the data to be stored in data storage request.
Optionally, the fragment carried in the second data storage request includes data fragmentation or verification fragment.Data processing mould Block 804 carries out the data to be stored before the member node into the EC group sends the second data storage request Fragment, and verification fragment is calculated according to obtained data fragmentation.
Optionally, the first determining module 801 is specifically used for: according to the mark of the data of request storage, the determining and mark Corresponding subregion, and corresponding EC group is determined according to the subregion determined.
Optionally, data storage request module 803 is also used to: the member node in Xiang Suoshu EC group sends the second data and deposits After storage request, the member node received in the EC group is completed according to the data storage operations that the second data storage request returns Notice, after receiving the data storage operations completion notice that all member nodes in the EC group return, in Xiang Suoshu EC group Host node send data storage operations completion notice.
It is client device provided in an embodiment of the present invention referring to Fig. 9, which is deposited in using Erasure Code The distributed memory system of storage mechanism is, it can be achieved that the data acquisition process that previous embodiment describes.As shown, the equipment can wrap Include: the second determining module 901 obtains module 902, respond module 903, in which:
Second determining module 901, for being that the data of request are determined according to the first data acquisition request received Member node in the EC group at place and EC grouping；
Module 902 is obtained, for obtaining data from the member node in the EC group determined, and out of described EC group All member nodes obtain operation timing mark；
Respond module 903 is identified for the operation timing to all member nodes in the EC group got and is carried out Compare, if the operation timing mark of the member node in the EC group is identical, returns and first data acquisition request pair The response answered carries the data got in the response.
Optionally, respond module 903 is also used to: if the operation timing mark of the member node in the EC group is not identical, Then the host node into the EC group sends data acquisition request, and the data of request are carried in the data acquisition request Mark and data restore instruction, the data recovery request be used for trigger the host node in the EC group of place member save The carry out data recovery of point and the data that institute's request is returned after data are successfully recovered；And it receives host node and returns Data be successfully recovered response, the data are successfully recovered in response and carry the data got；And it returns and described the The corresponding response of one data acquisition request carries the data got in the response.
Above-mentioned Fig. 8 and functional module shown in Fig. 9 may be alternatively located in same client device, to make the client device Realize the data storage and data acquisition process of the above embodiment of the present invention description.
Referring to Figure 10, be communication device provided in an embodiment of the present invention, it can be achieved that in previous embodiment host node process. As shown in Figure 10, device can include: transceiver 1001, processor 1002 and memory 1003.
Processor 1002 is used to control the operation of the device, including carried out by transceiver 1001 data transmission (including It receives and/or sends)；Memory 1003 may include read-only memory and random access memory, be used for processor 1002 Instruction and data is provided.The a part of of memory 1003 can also include non-volatile row random access memory (NVRAM).The dress The various components set are coupled by bus system, and wherein bus system 1004 further includes in addition to including data/address bus Power bus, control bus and status signal bus in addition.But for the sake of clear explanation, various buses are all designated as always in figure Linear system system 1004.
The process that the embodiment of the present application discloses can be applied in processor 1002, or be realized by processor 1002.? During realization, each step of the process which realizes can by the integrated logic circuit of the hardware in processor 1002 or The instruction of person's software form is completed.Processor 1002 can be general processor, digital signal processor, specific integrated circuit, Field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, It may be implemented or execute disclosed each method, step and the logic diagram in the embodiment of the present application.General processor can be Microprocessor or any conventional processor etc..The step of method in conjunction with disclosed in the embodiment of the present application, can directly embody Execute completion for hardware processor, or in processor hardware and software module combination execute completion.Software module can be with Positioned at random access memory, flash memory, read-only memory, programmable read only memory or electrically erasable programmable memory, deposit In the storage medium of this fields such as device maturation.The storage medium is located at memory 1003, and processor 1002 reads memory 1003 In information, in conjunction with its hardware complete the embodiment of the present invention instruction process the step of.
Specifically, processor 1002 can be configured to perform the process of host node side described in above-described embodiment.Processor Above-mentioned process performed by 1002, reference can be made to the description of previous embodiment, this will not be detailed here.
Referring to Figure 11, be communication device provided in an embodiment of the present invention, it can be achieved that in previous embodiment client process. As shown in figure 11, device can include: transceiver 1101, processor 1102 and memory 1103.
Processor 1102 is used to control the operation of the device, including carried out by transceiver 1101 data transmission (including It receives and/or sends)；Memory 1103 may include read-only memory and random access memory, be used for processor 1102 Instruction and data is provided.The a part of of memory 1103 can also include non-volatile row random access memory (NVRAM).The dress The various components set are coupled by bus system, and wherein bus system 1104 further includes in addition to including data/address bus Power bus, control bus and status signal bus in addition.But for the sake of clear explanation, various buses are all designated as always in figure Linear system system 1104.
The process that the embodiment of the present application discloses can be applied in processor 1102, or be realized by processor 1102.? During realization, each step of the process which realizes can by the integrated logic circuit of the hardware in processor 1102 or The instruction of person's software form is completed.Processor 1102 can be general processor, digital signal processor, specific integrated circuit, Field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, It may be implemented or execute disclosed each method, step and the logic diagram in the embodiment of the present application.General processor can be Microprocessor or any conventional processor etc..The step of method in conjunction with disclosed in the embodiment of the present application, can directly embody Execute completion for hardware processor, or in processor hardware and software module combination execute completion.Software module can be with Positioned at random access memory, flash memory, read-only memory, programmable read only memory or electrically erasable programmable memory, deposit In the storage medium of this fields such as device maturation.The storage medium is located at memory 1103, and processor 1102 reads memory 1103 In information, in conjunction with its hardware complete the embodiment of the present invention instruction process the step of.
Specifically, processor 1102 can be configured to perform the data Stored Procedure of client described in above-described embodiment. Above-mentioned process performed by processor 1102, reference can be made to the description of previous embodiment, this will not be detailed here.
Referring to Figure 12, be communication device provided in an embodiment of the present invention, it can be achieved that in previous embodiment client process. As shown in figure 12, device can include: transceiver 1201, processor 1202 and memory 1203.
Processor 1202 is used to control the operation of the device, including carried out by transceiver 1201 data transmission (including It receives and/or sends)；Memory 1203 may include read-only memory and random access memory, be used for processor 1202 Instruction and data is provided.The a part of of memory 1203 can also include non-volatile row random access memory (NVRAM).The dress The various components set are coupled by bus system, and wherein bus system 1204 further includes in addition to including data/address bus Power bus, control bus and status signal bus in addition.But for the sake of clear explanation, various buses are all designated as always in figure Linear system system 1204.
The process that the embodiment of the present application discloses can be applied in processor 1202, or be realized by processor 1202.? During realization, each step of the process which realizes can by the integrated logic circuit of the hardware in processor 1202 or The instruction of person's software form is completed.Processor 1202 can be general processor, digital signal processor, specific integrated circuit, Field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, It may be implemented or execute disclosed each method, step and the logic diagram in the embodiment of the present application.General processor can be Microprocessor or any conventional processor etc..The step of method in conjunction with disclosed in the embodiment of the present application, can directly embody Execute completion for hardware processor, or in processor hardware and software module combination execute completion.Software module can be with Positioned at random access memory, flash memory, read-only memory, programmable read only memory or electrically erasable programmable memory, deposit In the storage medium of this fields such as device maturation.The storage medium is located at memory 1203, and processor 1202 reads memory 1203 In information, in conjunction with its hardware complete the embodiment of the present invention instruction process the step of.
Specifically, processor 1202 can be configured to perform the data acquisition process of client described in above-described embodiment. Above-mentioned process performed by processor 1202, reference can be made to the description of previous embodiment, this will not be detailed here.
In short, being not intended to limit of the invention the foregoing is merely the preferred embodiment of technical solution of the present invention Protection scope.All within the spirits and principles of the present invention, any modification, equivalent replacement, improvement and so on should be included in Within protection scope of the present invention.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more, The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces The form of product.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs The processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices is instructed, so that A stream in flow chart can be achieved by the instruction that the computer or the processor of other programmable data processing devices execute The function of being specified in journey or multiple processes and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates, Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or The instruction executed on other programmable devices is provided for realizing in one process or multiple processes and/or block diagrams of flow chart One box or multiple boxes in specify function the step of.
Although alternative embodiment of the invention has been described, created once a person skilled in the art knows basic Property concept, then additional changes and modifications may be made to these embodiments.So the following claims are intended to be interpreted as include can It selects embodiment and falls into all change and modification of the scope of the invention.
Those skilled in the art various changes and modifications can be made to the invention without departing from spirit of the invention and Range.In this way, if these modifications and changes of the present invention is within the scope of the claims of the present invention and its equivalent technology, Then the present invention is also intended to include these modifications and variations.

Claims (25)

1. a kind of data storage control method, applied to the distributed memory system using Erasure Code memory mechanism, institute The memory space for stating distributed memory system is divided into multiple subregions, the corresponding EC group of a subregion, the interior packet of an EC group Containing host node, which is characterized in that this method comprises:
Host node in EC group receives the data that client is sent and stores authority request；
The host node stores authority request according to the data, is that the data of the client store authority request batch operation Timing mark, the operation timing mark, which sends data storage for the member node into the EC group by the client, asks It asks；The operation timing is identified as operation timing number, and the operation timing numbers to be incremented by according to the sequencing of distribution；
The host node is responded to client returned data storage permission, and the operation timing of distribution is carried in the response Mark.
2. the method as described in claim 1, which is characterized in that the host node in the EC group receives the data that client is sent The data storage authority request distribution that authority request is the client is stored after storage authority request, according to the data to grasp Before making timing mark, further includes:
For Data Identification corresponding to data storage authority request, the host node determines whether that identical data identifies Data storage operations also do not complete；
If so, then the data being currently received storage authority request is hung up, and in the also unfinished data storage After the completion of operation, releases the data storage authority request and release hang-up.
3. the method as described in claim 1, which is characterized in that the host node in the EC group receives the data that client is sent The data storage authority request distribution that authority request is the client is stored after storage authority request, according to the data to grasp Before making timing mark, further includes:
The host node determines whether the corresponding subregion of the EC group can be used；
If unavailable, response of the host node to client returned data storage authority distribution failure.
4. the method as described in claim 1, which is characterized in that the host node stores permission to the client returned data After response, further includes:
The host node receives the data storage operations completion notice that the client is sent, and the data storage operations are completed logical After knowing that the client identifies the member node transmission data storage request into the EC group according to the operation timing, really The member node recognized in the EC group completes what data storage operations were sent later.
5. a kind of date storage method, applied to the distributed memory system using Erasure Code memory mechanism, described point The memory space of cloth storage system is divided into multiple subregions, and the corresponding EC group of a subregion, an EC group is interior to include master Node, which is characterized in that this method comprises:
For client according to the first data storage request received, the data for request storage determine corresponding EC group；
Host node of the client into the EC group sends data and stores authority request, receives the number that the host node returns According to storage permission response, the operation timing mark of the data acquisition authority request distribution of the promising client is carried in the response Know；The operation timing is identified as operation timing number, and the operation timing numbers to be incremented by according to the sequencing of distribution；
Member node of the client into the EC group sends the second data storage request, takes in the data storage request With operation timing mark and the corresponding fragment of the data to be stored.
6. method as claimed in claim 5, which is characterized in that the fragment carried in second data storage request includes number According to fragment or verification fragment；
Member node of the client into the EC group is sent before the second data storage request, further includes:
The client carries out fragment to the data to be stored, and verification point is calculated according to obtained data fragmentation Piece.
7. method as claimed in claim 5, which is characterized in that the client is that data to be stored determine corresponding EC Group, comprising:
The client determines corresponding with mark subregion according to the marks of the data of request storage, and according to determining Subregion determines corresponding EC group.
8. method as claimed in claim 5, which is characterized in that member node of the client into the EC group sends the After two data storage requests, further includes:
The client receives the data storage operations that the member node in the EC group is returned according to the second data storage request Completion notice；
After the client receives the data storage operations completion notice of the return of all member nodes in the EC group, to institute The host node stated in EC group sends data storage operations completion notice.
9. a kind of data capture method characterized by comprising
Client is the EC group and the EC where the data of request determine according to the first data acquisition request received Member node in group；
The client obtains data from the member node in the EC group determined；
The client obtains operation timing mark from all member nodes in the EC group, and is compared, if the EC The operation timing mark of member node in group is identical, then returns to response corresponding with first data acquisition request, described The data got are carried in response；The operation timing is identified as operation timing number, and the operation timing numbers according to distribution Sequencing be incremented by.
10. method as claimed in claim 9, which is characterized in that if the operation timing mark of the member node in the EC group It is not identical, then further include:
Host node of the client into the EC group sends data acquisition request, carries and asks in the data acquisition request The mark of the data of acquisition and data is asked to restore instruction, the data recovery request is for triggering the host node to place EC The carry out data recovery of member node in group and the data that institute's request is returned after data are successfully recovered；
The client receives the data that host node returns and is successfully recovered response, and the data are successfully recovered to carry in response and obtain The data got；
The client returns to response corresponding with first data acquisition request, carries the number got in the response According to.
11. method as claimed in claim 10, which is characterized in that host node of the client into the EC group sends number According to recovery request, comprising:
The client is marked according to the operation timing of the member node in the maximum quantity of recoverable fragment and the EC group In knowledge, the time is distributed in the quantity of rear and identical member node, judges whether to meet recovery condition；
If meeting recovery condition, host node of the client into the EC group sends data recovery request；
If being unsatisfactory for recovery condition, the method also includes: the client returns and first data acquisition request pair The data acquisition request failure response answered.
12. a kind of client device, applied to the distributed memory system using Erasure Code memory mechanism, the distribution The memory space of formula storage system is divided into multiple subregions, and the corresponding EC group of a subregion, an interior EC group includes main section Point characterized by comprising
First determining module, for being determined for the data that request stores corresponding according to the first data storage request received EC group；
Authority acquiring module sends data for the host node into the EC group and stores authority request, receives the host node The data storage permission of return responds, and the behaviour of the data storage authority request distribution of the promising client is carried in the response Make timing mark；The operation timing is identified as operation timing number, and the operation timing numbers to be incremented by according to the sequencing of distribution；
Data storage request module sends the second data storage request, the data for the member node into the EC group The operation timing mark and the corresponding fragment of the data to be stored are carried in storage request.
13. equipment as claimed in claim 12, which is characterized in that the fragment carried in second data storage request includes Data fragmentation or verification fragment；
The client device further include:
Data processing module, for into the EC group member node send the second data storage request before, to described Data to be stored carry out fragment, and verification fragment is calculated according to obtained data fragmentation.
14. equipment as claimed in claim 12, which is characterized in that first determining module is specifically used for: being deposited according to request The mark of the data of storage determines subregion corresponding with the mark, and determines corresponding EC group according to the subregion determined.
15. equipment as claimed in claim 12, which is characterized in that the data storage request module is also used to: Xiang Suoshu EC After member node in group sends the second data storage request, the member node received in the EC group is deposited according to the second data The data storage operations completion notice that storage request returns receives the data storage that all member nodes in the EC group return After operation completion notification, the host node in Xiang Suoshu EC group sends data storage operations completion notice.
16. a kind of client device, applied to the distributed memory system using Erasure Code memory mechanism, the distribution The memory space of formula storage system is divided into multiple subregions, and the corresponding EC group of a subregion, an interior EC group includes main section Point characterized by comprising it is characterised by comprising:
Second determining module, for being where the data of request determine according to the first data acquisition request received Member node in EC group and the EC group；
Obtain module, for from the EC group determined member node obtain data, and from the EC group it is all at Member's node obtains operation timing mark；The operation timing is identified as operation timing number, and the operation timing numbers according to distribution Sequencing is incremented by；
Respond module is compared for the operation timing mark to all member nodes in the EC group got, if The operation timing mark of member node in the EC group is identical, then returns to sound corresponding with first data acquisition request It answers, the data got is carried in the response.
17. equipment as claimed in claim 16, which is characterized in that the respond module is also used to: if in the EC group at The operation timing mark of member's node is not identical, then the host node into the EC group sends data acquisition request, and the data obtain The mark for the data for carrying request in request and data is taken to restore instruction, the data recovery request is for triggering institute It states host node and returns to the carry out data recovery of the member node in the EC group of place and after data are successfully recovered and request to obtain The data taken；And receive the data that host node returns and be successfully recovered response, the data are successfully recovered to carry in response and obtain The data got；And corresponding with first data acquisition request response is returned, it carries in the response and to get Data.
18. a kind of host node device, applied to the distributed memory system using Erasure Code memory mechanism, the distribution The memory space of formula storage system is divided into multiple subregions, and the corresponding EC group of a subregion, an interior EC group includes main section Point characterized by comprising
First receiving module, the data for receiving client transmission store authority request；
Distribution module is that the data of the client store authority request distribution for storing authority request according to the data Operation timing mark, the operation timing mark send data for the member node into the EC group by the client and deposit Storage request；
Sending module carries the operation of distribution for responding to client returned data storage permission in the response Timing mark；The operation timing is identified as operation timing number, and the operation timing numbers to be incremented by according to the sequencing of distribution.
19. equipment as claimed in claim 18, which is characterized in that the distribution module is also used to:
After receiving the data storage authority request that client is sent, for number corresponding to data storage authority request According to mark, it is determined whether the data storage operations for having identical data to identify do not complete also；If so, described in being then currently received Data store authority request and hang up, and after the completion of the data storage operations not completed also, release the data storage power Limit request, which releases, to be hung up.
20. equipment as claimed in claim 18, which is characterized in that the distribution module is also used to:
Authority request is stored after receiving the data storage authority request that client is sent, according to the data as the client Before the data storage authority request batch operation timing mark at end, determine whether the corresponding subregion of the EC group can be used；If no It can use, then to the response of client returned data storage authority distribution failure.
21. equipment as claimed in claim 18, which is characterized in that further include:
Second receiving module, for receiving the client hair after responding to client returned data storage permission The data storage operations completion notice sent, the data storage operations completion notice are the clients according to the operation timing After identifying the member node transmission data storage request into the EC group, confirm that the member node in the EC group completes data It is sent after storage operation.
22. equipment as claimed in claim 18, which is characterized in that further include data recovery module, be used for:
The data acquisition request that client device is sent is received, the data of request are carried in the data acquisition request Mark and data restore instruction；
Restore carry out data recovery of the instruction to the member node in the EC group of place according to the data；
After data are successfully recovered, the data of acquisition request acquisition；
Returned data is successfully recovered response, and the data are successfully recovered in response and carry the data got.
23. a kind of client device, applied to the distributed memory system using Erasure Code memory mechanism, the distribution The memory space of formula storage system is divided into multiple subregions, and the corresponding EC group of a subregion, an interior EC group includes main section Point, which is characterized in that the client device includes: transceiver, processor and memory；
The memory, for storing computer program instructions；
The processor is coupled to the memory, for reading the computer program instructions of the memory storage, and executes Method as described in any one of claim 5 to 8.
24. a kind of client device, applied to the distributed memory system using Erasure Code memory mechanism, the distribution The memory space of formula storage system is divided into multiple subregions, and the corresponding EC group of a subregion, an interior EC group includes main section Point, which is characterized in that the client device includes: transceiver, processor and memory；
The memory, for storing computer program instructions；
The processor is coupled to the memory, for reading the computer program instructions of the memory storage, and executes Method as described in any one of claim 9 to 11.
25. a kind of host node device, applied to the distributed memory system using Erasure Code memory mechanism, the distribution The memory space of formula storage system is divided into multiple subregions, and the corresponding EC group of a subregion, an interior EC group includes main section Point, which is characterized in that the host node device includes: transceiver, processor and memory；
The memory, for storing computer program instructions；
The processor is coupled to the memory, for reading the computer program instructions of the memory storage, and executes Method as described in any one of Claims 1-4.
CN201610777273.6A 2016-08-30 2016-08-30 Data storage control method, date storage method, data capture method and device Active CN106375404B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610777273.6A CN106375404B (en) 2016-08-30 2016-08-30 Data storage control method, date storage method, data capture method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610777273.6A CN106375404B (en) 2016-08-30 2016-08-30 Data storage control method, date storage method, data capture method and device

Publications (2)

Publication Number Publication Date
CN106375404A CN106375404A (en) 2017-02-01
CN106375404B true CN106375404B (en) 2019-10-25

Family

ID=57900987

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610777273.6A Active CN106375404B (en) 2016-08-30 2016-08-30 Data storage control method, date storage method, data capture method and device

Country Status (1)

Country Link
CN (1) CN106375404B (en)

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110311880B (en) * 2018-03-20 2021-08-06 中移(苏州)软件技术有限公司 File uploading method, device and system
CN108959549A (en) * 2018-06-29 2018-12-07 北京奇虎科技有限公司 Method for writing data, calculates equipment and computer storage medium at device
CN109165136B (en) * 2018-07-25 2021-09-28 平安科技（深圳）有限公司 Terminal operation data monitoring method, terminal device and medium
CN109254732A (en) * 2018-08-31 2019-01-22 郑州云海信息技术有限公司 A kind of reading/writing method of storing data, device and relevant device
CN109271105A (en) * 2018-08-31 2019-01-25 郑州云海信息技术有限公司 Date storage method and distributed storage cluster applied to distributed storage cluster
CN110968453A (en) * 2018-09-28 2020-04-07 杭州海康威视系统技术有限公司 Data storage method and device
CN110968750A (en) * 2018-09-30 2020-04-07 北京国双科技有限公司 Data storage method and system
CN109686415B (en) * 2018-12-21 2021-07-16 苏州爱医斯坦智能科技有限公司 Data transmission method, management platform, electronic device and storage medium
CN109885424A (en) * 2019-01-16 2019-06-14 平安科技（深圳）有限公司 A kind of data back up method, device and computer equipment
CN110287197B (en) * 2019-06-28 2022-02-08 微梦创科网络科技（中国）有限公司 Data storage method, migration method and device
CN112748851A (en) * 2019-10-30 2021-05-04 北京白山耘科技有限公司 Data reading method, device and system
CN110955388A (en) * 2019-10-31 2020-04-03 北京浪潮数据技术有限公司 Storage server, data storage method and device thereof and storage medium
CN111273859B (en) * 2020-01-14 2023-09-15 北京百度网讯科技有限公司 Method, device, equipment and medium for changing replication group members in distribution mode
CN111277634B (en) * 2020-01-14 2021-12-21 北京金山云网络技术有限公司 Data ID distribution method, device, system and server
CN111541762B (en) * 2020-04-20 2023-08-01 广州酷狗计算机科技有限公司 Data processing method, management server, device and storage medium
CN112328663A (en) * 2020-11-24 2021-02-05 深圳市鹰硕技术有限公司 Data discovery method and system applied to big data

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102667738A (en) * 2009-10-09 2012-09-12 提琴存储器公司 Memory system with multiple striping of RAID groups and method for performing the same
CN103262024A (en) * 2010-05-09 2013-08-21 思杰系统有限公司 Methods and systems for forcing an application to store data in a secure storage location
CN104106057A (en) * 2011-12-13 2014-10-15 英特尔公司 Method and system for providing instant responses to sleep state transitions with non-volatile random access memory
CN104902009A (en) * 2015-04-27 2015-09-09 浙江大学 Erasable encoding and chained type backup-based distributed storage system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102667738A (en) * 2009-10-09 2012-09-12 提琴存储器公司 Memory system with multiple striping of RAID groups and method for performing the same
CN103262024A (en) * 2010-05-09 2013-08-21 思杰系统有限公司 Methods and systems for forcing an application to store data in a secure storage location
CN104106057A (en) * 2011-12-13 2014-10-15 英特尔公司 Method and system for providing instant responses to sleep state transitions with non-volatile random access memory
CN104902009A (en) * 2015-04-27 2015-09-09 浙江大学 Erasable encoding and chained type backup-based distributed storage system

Also Published As

Publication number Publication date
CN106375404A (en) 2017-02-01

Similar Documents

Publication Publication Date Title
CN106375404B (en) Data storage control method, date storage method, data capture method and device
JP2022501752A (en) How to assign electronic bill identifiers, how to generate electronic bills, their devices and systems, as well as storage media and computer programs.
CN108664660A (en) Distributed implementation method, apparatus, equipment and the storage medium of time series database
CN107402722B (en) Data migration method and storage device
CN104636286B (en) For the method and apparatus of data access
CN105677673B (en) Method for processing business, apparatus and system
JP2016515745A (en) Methods and devices for updating clients
WO2020133963A1 (en) Blockchain-based data storage method, related device and storage medium
CN105677675B (en) Method for processing business and device
CN106385377B (en) Information processing method and system
CN109600302A (en) A kind of method, apparatus orderly communicated, storage medium and electronic equipment
CN110633168A (en) Data backup method and system for distributed storage system
CN104793981B (en) A kind of online snapshot management method and device of cluster virtual machine
US10740159B2 (en) Synchronization object prioritization systems and methods
CN105357042A (en) High-availability cluster system, master node and slave node
CN109542627A (en) Node switching method, device, supervisor, node device and distributed system
CN105471955A (en) Writing method of distributed file system, client device and distributed file system
WO2020133962A1 (en) Blockchain-based data storage method, related device and storage medium
KR20200118798A (en) Electronic devices, systems and methods
CN105933352B (en) Method of data synchronization, client and system between client-based server
KR101674294B1 (en) Apparatus for operating data structure capable of random access and state access and operating method thereof
CN110704198A (en) Data operation method, device, storage medium and processor
EP3346671B1 (en) Service processing method and equipment
CN111339089B (en) Data storage and acquisition method and device applied to blockchain
US10567507B2 (en) Message processing method and apparatus, and message processing system

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant