CN106484311B - A kind of data processing method and device - Google Patents
A kind of data processing method and device Download PDFInfo
- Publication number
- CN106484311B CN106484311B CN201510547258.8A CN201510547258A CN106484311B CN 106484311 B CN106484311 B CN 106484311B CN 201510547258 A CN201510547258 A CN 201510547258A CN 106484311 B CN106484311 B CN 106484311B
- Authority
- CN
- China
- Prior art keywords
- node
- data
- write
- preassigned
- standby
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data processing method and devices, for solving the problems, such as that data writing mode cannot be guaranteed to be successfully written data on the desired node of user in the prior art.The data processing method includes: that calculate node receives the write request that client is sent, and the write request includes to request the data of write-in；The calculate node determines the primary storage node and M standby node for storing the data for requesting write-in；The data for requesting write-in are written into the primary storage node for the calculate node；The calculate node receives the write-in success message that the primary storage node returns；The calculate node is respectively written into the data for requesting write-in to the M standby node；The calculate node determination receives the write-in success message that each node returns respectively in N number of preassigned standby node；The calculate node writes success message to client return.
Description
Technical field
The present invention relates to field of data storage, in particular to a kind of data processing method and device.
Background technique
Distributed system, which refers to, to be distributed to computing resource and storage resource on different nodes, by network by each node
The system connected.Distributed system can be accomplished a task by managing and controlling multiple nodes, but externally performance
For a complete system of independence, user's perception is less than distributed system task in internal execution details.
Fig. 1 is the schematic diagram of distributed system, wherein client (Client) is used to receive user demand (such as write request
And read request), calculate node (English: Distributed Object Process；Referred to as: DOP) it is responsible for distribution routing, and deposits
Store up node (English: Key-Value Store；Referred to as: KVS) it is responsible for single-deck management and actual storage, in addition, further including in system
Management node (English: Distributed Object Management；Referred to as: DOM), it is responsible for carrying out multiple DOP and KVS
Management.
Distributed system saves data using more replication policies, i.e., same data is stored on multiple KVS, to avoid
Loss of data caused by KVS failure.Meanwhile system will also ensure the consistency of multiple copies of same data, i.e., with multiple KVS
When storing same data, it is ensured that the data stored on multiple KVS are consistent.
NRW mechanism is a kind of strategy for taking into account more copy consistencies and data read-write efficiency, wherein N indicates that data are had
Some number of copies, R indicate the required minimum number of copies read of completion read operation, and W indicates to complete the required write-in of write operation
Minimum number of copies, the i.e. required minimum node number participated in of a write operation.For example, in NRW322 mechanism, number in system
According to there are 3 different copies, when carrying out write operation, need that at least 2 copies is waited to complete the write operation system just meeting
The state to run succeeded is returned, and when carrying out read operation, then need to read at least two copies, in two copy datas one
Read operation success is determined when cause.
In addition, distributed system also uses primary copy locking to solve concurrency conflict when write operation, that is, from storing data
Multiple memory nodes in select a memory node as primary storage node, remaining memory node is known as slack storage node.
When data are written, what calculate node needed first to compete primary storage node writes lock permission, to primary storage after lock permission is write in acquisition
Data are written in node, and data are written to standby node again after being written successfully, are write as function in the standby node return for having setting quantity
When, calculate node writes success message to client return, and remaining standby node continues to write data behaviour using asynchronous mechanism
Make.
Referring to Fig. 2, by taking NRW322 mechanism as an example, the process that primary storage node writes data into memory node includes following step
Rapid: step 101, client sends write request to calculate node；Step 102, the lock of writing that calculate node obtains primary storage node is weighed
Limit, and data are written to primary storage node；Step 103, primary storage node returns to write-in success message to calculate node；Step
104, data are written to standby node in calculate node；Step 105, any standby node (might as well be set as standby node 2) is to calculating
Node returns to write-in success message；Step 106, calculate node returns to write-in success message to client.
In actual conditions, user may expect successfully to preserve data on the standby node of part, for example, reading data is fast
Data are preserved on the high node of rate, quickly to read, alternatively, data are preserved on the high node of storage stability, to keep away
Exempt from loss of data, or, data are preserved on the node of user office.But in above-mentioned technical proposal, calculate
Node writes success message to client return when writing success message what the standby node for receiving setting quantity returned, but regardless of tool
Body is which standby node return to write success message by, accordingly, it is possible to which data are not yet written in the desired standby node of user
When, calculate node writes success message to client return, and the desired standby node of user fails due to failure later
Data are successfully written, data is caused to be not stored on the desired node of user.
Summary of the invention
The embodiment of the present invention provides a kind of data processing method and device, for solving data writing mode in the prior art
It cannot be guaranteed that the problem of being successfully written data on the desired node of user.
In a first aspect, the embodiment of the present invention provides a kind of data processing method, comprising:
Calculate node receives the write request that client is sent, and the write request includes to request the data of write-in；
The calculate node determines the primary storage node and M standby node for storing the data for requesting write-in,
Wherein, the M standby node includes N number of preassigned standby node, and M is the positive integer greater than 1, and N is just whole less than M
Number；
The data for requesting write-in are written into the primary storage node for the calculate node；
The calculate node receives the write-in success message that the primary storage node returns；
The calculate node is respectively written into the data for requesting write-in to the M standby node；
The calculate node determination receives what each node in N number of preassigned standby node returned respectively
Success message is written；
The calculate node writes success message to client return.
With reference to first aspect, in the first possible implementation of the first aspect, described N number of preassigned spare
Node is the node that storage performance parameter meets preset requirement in the M standby node.
With reference to first aspect or the first possible implementation of first aspect, second in first aspect are possible
In implementation, after the calculate node writes success message to client return, further includes:
The calculate node receives the read request that the client is sent, and the read request packet is containing the mark for requesting reading data
Know；
The calculate node determines that the storage mark is corresponding and requests to read the described N number of preassigned spare of data
Node；And
Store in any node in N number of preassigned standby node corresponding with the mark is read to be requested
Data are read, and the data of reading are returned into the client.
The possible implementation of second with reference to first aspect, in the third possible implementation of first aspect
In, before the data of reading are returned to the client by the calculate node, further includes:
The calculate node determines that store in N number of preassigned standby node corresponding with the mark is asked
Ask reading data all the same.
The third possible implementation of the possible implementation of second with reference to first aspect or first aspect,
In 4th kind of possible implementation of first aspect, the calculate node is stored with mapping table, and the mapping table preserves number
According to the corresponding relationship of mark and data storage location, the data storage location includes the described N number of predesignated of storing data
Standby node；
The calculate node determines that the storage mark is corresponding and requests to read the described N number of preassigned spare of data
Node, comprising:
The calculate node determines that the storage mark is corresponding and requests to read according to the mark and the mapping table
N number of preassigned standby node of data.
Second aspect, the embodiment of the present invention provide a kind of data processing equipment, comprising:
Receiving module, for receiving the write request of client transmission, the write request includes to request the data of write-in；
Determining module, for determining primary storage node and M standby node for storing the data for requesting write-in,
Wherein, the M standby node includes N number of preassigned standby node, and M is the positive integer greater than 1, and N is just whole less than M
Number；
Writing module, for the data for requesting write-in to be written into the primary storage node；
The receiving module is also used to: receiving the write-in success message that the primary storage node returns；
The write module is also used to: being received the write-in that the primary storage node returns in the receiving module and is successfully disappeared
After breath, Xiang Suoshu M standby node is respectively written into the data for requesting write-in；
The determining module is also used to: being determined and is received each node point in N number of preassigned standby node
The write-in success message not returned；
Described device further include: sending module, it is described N number of preassigned for being received in determining module determination
After the write-in success message that each node returns respectively in standby node, success message is write in the return of Xiang Suoshu client.
In conjunction with second aspect, in the first possible implementation of the second aspect, the determining module is used for: being determined
It is the preassigned standby node that storage performance parameter, which meets N number of node of preset requirement, in the M standby node.
In conjunction with the possible implementation of the first of second aspect or second aspect, second in second aspect is possible
In implementation, the receiving module is also used to: receiving the read request that the client is sent, the read request packet, which contains, is requested
Read the mark of data；
The determining module is also used to: determining the corresponding N number of preparatory finger for requesting to read data of the storage mark
Fixed standby node；
Described device further include: read module, for reading in N number of preassigned standby node in any node
The corresponding with the mark of storage requests to read data；
The sending module is also used to: the data of reading are returned to the client.
In conjunction with second of possible implementation of second aspect, in the third possible implementation of second aspect
In, the determining module is also used to: determining the institute corresponding with the mark stored in N number of preassigned standby node
It is all the same that data are read in request；
The sending module is used for: being determined in the determining module and is stored in N number of preassigned standby node
It is corresponding with the mark request read data it is all the same after, the data of reading are returned into the client.
In conjunction with second of possible implementation of second aspect or the third possible implementation of second aspect,
In 4th kind of possible implementation of second aspect, described device is stored with mapping table, and the mapping table preserves data mark
Know the corresponding relationship with data storage location, the data storage location includes the described N number of preassigned standby of storing data
Use node；The determining module is used for: according to the mark and the mapping table, being determined that the storage mark is corresponding and is asked
Seek the N number of preassigned standby node for reading data.
The one or more technical solutions provided in the embodiment of the present invention, have at least the following technical effects or advantages:
In the embodiment of the present invention, calculate node is when being written data into memory node according to write request, to primary storage
After data success is written in node, be only all written in N number of preassigned standby node after data success just to client
End returns and writes success message, wherein N number of preassigned standby node (such as can be deposited for the desired standby node of user
Store up the node of better performances).Therefore, above-mentioned technical proposal can guarantee to be successfully written number on the desired standby node of user
According to meeting the needs of user, improve the stability of reading data.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the invention, for this
For the those of ordinary skill in field, without any creative labor, it can also be obtained according to these attached drawings
His attached drawing.
Fig. 1 is the schematic diagram of distributed system；
Fig. 2 is the flow diagram that data are write in background technique；
Fig. 3 is the flow diagram of data processing method in the embodiment of the present invention；
Fig. 4 is another flow diagram of data processing method in the embodiment of the present invention；
Fig. 5 is the another flow diagram of data processing method in the embodiment of the present invention；
Fig. 6 is the schematic diagram that data are write in the embodiment of the present invention；
Fig. 7 is the schematic diagram that data are read in the embodiment of the present invention；
Fig. 8 is the structural schematic block diagram of data processing equipment in the embodiment of the present invention；
Fig. 9 is the structural schematic block diagram of data processing equipment in the embodiment of the present invention.
Specific embodiment
Technical solution of the present invention is described in detail below by attached drawing and specific embodiment, it should be understood that the present invention
Specific features in embodiment and embodiment are the detailed description to technical solution of the present invention, rather than to the technology of the present invention
The restriction of scheme, in the absence of conflict, the technical characteristic in the embodiment of the present invention and embodiment can be combined with each other.
Fig. 3 is the flow diagram of data processing method provided in an embodiment of the present invention, which includes the following steps:
Step 101: calculate node receives the write request that client is sent, and write request includes to request the data of write-in；
Step 102: calculate node determines primary storage node and M spare sections for storing the data for requesting write-in
Point, wherein M standby node includes N number of preassigned standby node, and M is the positive integer greater than 1, and N is just whole less than M
Number；
Step 103: the data for requesting write-in are written into primary storage node for calculate node；
Step 104: calculate node receives the write-in success message that primary storage node returns；
Step 105: calculate node is respectively written into the data for requesting write-in to M standby node；
Step 106: calculate node determination receives what each node in N number of preassigned standby node returned respectively
Success message is written；
Step 107: calculate node writes success message to client return.
Specifically, client generates write request according to the request request that data are written in user, and will write and ask in step 101
It asks and is sent to calculate node, and calculate node receives the write request, and the data for requesting write-in can be obtained from write request.It is real
In the situation of border, which can be sent to any calculate node by client, can also be determined to send according to preset rules and write
The target computing nodes of request, for example, write request is sent to the current idle maximum calculate node of computing capability, or
Person, be sent to the shortest calculate node, etc. that oneself is delayed, this will not be detailed here.
In step 102, calculate node determines the target storage node of write-in data, target storage node that is, for storing
Request the primary storage node and M standby node of the data of write-in.Specifically method of determination includes:
Mode 1, before the data for requesting write-in are data-storage system when not stored new data, calculate node from
Primary storage node and M standby node are distributed in memory node for the data；Alternatively, calculate node is deposited to management node transmission
Node distribution request is stored up, the distribution of memory node is carried out by management node, determines above-mentioned primary storage node and M spare sections
Point.Wherein, the method for salary distribution of memory node please refers to related art method in the prior art, and this will not be detailed here.
Mode 2, when storing the storing data in data-storage system before requesting the data of write-in to be, client
Holding also includes the mark for being requested write-in data in the write request sent, and maintenance has data mapping tables in calculate node, the table
In be stored with the corresponding relationship of Data Identification and data storage location, calculate node is inquired in write request in the mapping table includes
Request write-in data mark, that is, can determine storage request be written data primary storage node and M standby node.
In the embodiment of the present invention, aforesaid way 1 or mode 2 no matter are used, includes N number of preparatory in M standby node
Specified standby node, wherein N is less than M, which can be to be requested write-in data to carry out for the first time
Standby node distribute when carry out it is specified, N number of preassigned node can according to user's characteristic of interest (e.g., store
The reading rate of node, the stability of memory node, service life of memory node, the speed of response of memory node, memory node
Burden, etc.) it is determined, that is, select to meet the requirements in terms of user's characteristic of interest from M standby node (or arrange
Name it is forward) N number of node as preassigned standby node.
Then, step 103 is executed, the data for requesting write-in are written into the primary storage node determined for calculate node,
After being written successfully, primary storage node will return to write-in success message to calculate node, and calculate node executes step 104, receive master
The write-in success message that memory node returns, determines and data successfully is written in primary storage node.
Then, calculate node executes step 105, and the data for requesting write-in are respectively written into M standby node.M standby
With any node in node in write-in data success, write-in success message will be returned to calculate node.
Then, calculate node will receive after the write-in success message that standby node returns, and judge that this is written to
What whether function message was returned by preassigned standby node.Only N number of preassigned standby node is received in determination
In after the write-in success message that returns respectively of each node, calculate node, which just determines, writes data success, writes to client return
Success message.
In other words, as long as the write-in success message that N number of preassigned standby node returns is received, but regardless of remaining section
Whether point has been write as function, and calculate node will return to client and write success message.Or even if in M standby node
The write-in success message for having there is a standby node of setting quantity (N) to return, as long as having a preassigned standby node still
Write-in success message is not returned, and calculate node will not return to client and write success message, but wait preassigned standby
After all returning to write-in success message with node, success message just is write to client return.
In above-mentioned technical proposal, calculate node is when being written data into memory node according to write request, to primary storage
After data success is written in node, be only all written in N number of preassigned standby node after data success just to client
End returns and writes success message, wherein N number of preassigned standby node can be the desired standby node of user.Therefore, on
Stating technical solution can guarantee to be successfully written data on the desired standby node of user, meet the needs of user.
Optionally, in the embodiment of the present invention, N number of preassigned standby node is storage performance ginseng in M standby node
Number meets the node of preset requirement.Wherein, storage performance parameter may include: to read writing rate, job stability, using the longevity
The parameters such as life, operational capability, memory space.Preset requirement can be according to the application environment of data-storage system and user
Specific requirements are set.
By determining that N number of storage performance parameter meets default want from M standby node previously according to storage performance parameter
The node asked is as preassigned node, so that N number of preassigned node return is write as function i.e. to client when writing data
End returns and writes success message, in the case where guaranteeing data write efficiency, ensure that the stability and subsequent right that data store
The rate that data are read out,
Optionally, in the embodiment of the present invention, after step 105, before step 106, can also include the following steps: if
Calculate node is not received by all preassign in the setting duration after data are written to M standby node write request
Node return write-in success message, then calculate node to it is no return write-in success message preassigned node again
The data of write request write-in.
It, can be main when preassigned standby node fails to return to write-in success message in time in above-mentioned technical proposal
Data are written in it to trend again, avoid writing data and take long time.
Optionally, it in the embodiment of the present invention, before step 103, further comprises the steps of: calculate node and obtains primary storage node
Write lock permission.Lock control is write by primary storage node, more calculate nodes is avoided to be directed to the concurrent operating conflict of same data.
Optionally, in the embodiment of the present invention, client send write request further include request be written data mark with
And version number, before step 103, further comprise the steps of: version number that calculate node determines that write request includes be greater than it is currently stored
Request the corresponding version number of write-in data.By Version Control, avoid valid data capped.
Further, present inventor has found, in the prior art when reading data, in order to ensure the consistency of data, needs
It reads the data on primary storage node and sets the data of the standby node of quantity, after verify data consistency, ability
The data that request is read are returned to client.
For the example for continuing to use NRW322, calculate node needs to send reading to primary storage node and a standby node
According to request, the data that the two returns then is received, and judge whether two data are consistent, when consistent, return and read to client
The data taken.
It is above-mentioned not only to need to read primary storage node in the prior art, but also need to read setting quantity (for example, writing data
When, it receives the write-in success message that N number of preassigned standby node returns and writes success message to client return, then read
Need to read N number of preassigned standby node storage when data request reading data) being requested of storing of standby node
Data are read, reading data amount is larger, and bandwidth occupancy is larger, time-consuming also longer.
In view of the above-mentioned problems, data processing method provided in an embodiment of the present invention further includes following steps referring to Fig. 4:
Step 108: calculate node receives the read request that client is sent, and read request packet is containing the mark for requesting reading data；
Step 109: calculate node determines that the mark for including in storage read request is corresponding and requests to read the N number of preparatory of data
Specified standby node；
Step 110: calculate node read stored in any node in N number of preassigned standby node with the mark pair
That answers requests reading data；
Step 111: the data of reading are returned to client by calculate node.
Specifically, client requests to generate read request according to user, and read request is sent to calculating section in step 108
Point, comprising requesting the marks of reading data in the read request.Wherein, writing process and current reading before requesting to read data
Take process that can be responsible for by different calculate nodes, or same calculate node is responsible for, and the embodiment of the present invention not limits.
In step 109, calculate node determines the corresponding preassigned standby node of mark for including in storage read request
Mode include:
It is requested first, calculate node is sent to management node, includes the mark in request, request management node is according to its dimension
The data management information of shield determines the preassigned standby node of the corresponding data of the mark, and calculate node receives management node
The information of return therefrom determines the N number of preassigned standby node for requesting to read data.
Second, calculate node itself is stored with mapping table, which preserves Data Identification and data storage location
Corresponding relationship, data storage location include N number of preassigned standby node of storing data.Calculate node according to mark and
Mapping table can determine the corresponding N number of preassigned standby node for requesting to read data of storage mark.
In step 110, calculate node any node from N number of preassigned standby node reads corresponding with the mark
The data are returned to client by data.
Compared with prior art, in above-mentioned technical proposal, calculate node be not necessarily to not only to read data on primary storage node but also
Read setting standby node on data, but directly read stored on any preassigned standby node request read
According to.Handle so feasible the reason is that, being write in writing in data procedures before in N number of preassigned standby node
Success message is write in success Shi Caixiang client return, so, the requested data stored on preassigned standby node is most
The data being newly written, returning to the data i.e. to client can guarantee the correctness of reading data.
Through the above technical solutions, volume of transmitted data when reading data can be reduced, reduce data occupied bandwidth, and
It reduces and reads data time-consuming, improve data reading performance using redundancy.
Optionally, further include following steps referring to Fig. 5 before step 111 as another embodiment:
Step 112: calculate node determines that store in N number of preassigned standby node corresponding with mark requests to read
Data are all the same.
Specifically, calculate node sends the request for reading the mark corresponding data to N number of preassigned standby node, connect
It receives the data that N number of node returns to return, and whether the data for judging that N number of node returns are consistent, step 111 is executed when consistent, to
Client returns to the data read.If the data that N number of node returns are inconsistent, report an error to client, using preset mistake
The system of missing flight is handled, and the embodiment of the present invention is not described in detail herein.
By comparing the consistency of the data of N number of preassigned standby node, the consistent of data can be preferably ensured
Property, meanwhile, compared with prior art, calculate node still can read the data of primary storage node less, reduce the reading of data
Amount improves the efficiency for reading data.
It should be noted that primary storage node and preassigned standby node, spare section in the embodiment of the present invention
Point etc., for different data, corresponding primary storage node and is pre-established spare for specific data
Node can be different.
Below by taking use technical solution of the embodiment of the present invention is to the improvement project of NRW322 technology as an example, invention is given in detail
It states.
Specifically, storage system passes through primary storage node, preassigned standby node 1,2 storing data A of standby node.
Fig. 6 is the schematic diagram that the process of data A is written, and is included the following steps:
Step 201, client sends the request of write-in data A to calculate node；
Step 202, data A is written to primary storage node in calculate node；
Step 203, primary storage node returns to write-in success message to calculate node；
Step 204, data A is written to standby node in calculate node；
Step 205, the write-in success message that standby node 2 is returned to calculate node；
Step 206, calculate node determines that received write-in success message is not from the standby node 1 pre-established；
Step 207, the write-in success message that standby node 1 is returned to calculate node；
Step 208, calculate node determines that received write-in success message from the standby node 1 pre-established, executes step
Rapid 209；
Step 209, calculate node returns to write-in success message to client.
Corresponding, Fig. 7 is the schematic diagram for reading the process of data A, is included the following steps:
Step 210, client sends the request for reading data A to calculate node；
Step 211, calculate node sends the request for reading data A to preassigned standby node 1；
Step 212, standby node 1 is to calculate node returned data A；
Step 213, calculate node is to client returned data A.
When carrying out writing data through the above way, it can guarantee write-in data success in memory node that user specifies, and
When reading data through the above way, reading data efficiency can be improved, reduce bandwidth occupancy.
Based on the same technical idea, the embodiment of the invention also provides a kind of data processing equipments 300 referring to Fig. 8 is
The structural schematic block diagram of data processing equipment 300, comprising:
Receiving module 301, for receiving the write request of client transmission, write request includes to request the data of write-in；
Determining module 302, for determining primary storage node and M spare sections for storing the data for requesting write-in
Point, wherein M standby node includes N number of preassigned standby node, and M is the positive integer greater than 1, and N is just whole less than M
Number；
Writing module 303, for the data for requesting write-in to be written into primary storage node；
Receiving module 301 is also used to: receiving the write-in success message of primary storage node return；
Writing module 303 is also used to: after the write-in success message that receiving module receives the return of primary storage node, to M
A standby node is respectively written into the data for requesting write-in；
Determining module 302 is also used to: determining that receive each node in N number of preassigned standby node returns respectively
Write-in success message；
Device 300 further include: sending module 304, for receiving N number of preassigned spare section in determining module determination
After the write-in success message that each node returns respectively in point, success message is write to client return.
Optionally, in the embodiment of the present invention, determining module 302 is used for: determining that storage performance parameter is full in M standby node
N number of node of sufficient preset requirement is preassigned standby node.
Optionally, in the embodiment of the present invention, receiving module 301 is also used to: receiving the read request that client is sent, read request
Mark comprising requesting reading data；
Determining module 302 is also used to: determining the corresponding N number of preassigned spare section for requesting to read data of storage mark
Point；
Device 300 further include: read module 305 is deposited in any node for reading in N number of preassigned standby node
The corresponding with mark of storage requests to read data；
Sending module 304 is also used to: the data of reading are returned to client.
Optionally, determining module 302 is also used to: determine stored in N number of preassigned standby node it is corresponding with mark
Request read data it is all the same；
Sending module 304 is used for: determining module determine stored in N number of preassigned standby node it is corresponding with mark
Request read data it is all the same after, the data of reading are returned into client.
Optionally, in the embodiment of the present invention, device 300 is stored with mapping table, and mapping table preserves Data Identification and data
The corresponding relationship of storage location, data storage location include N number of preassigned standby node of storing data；Determining module
302 are used for: according to mark and mapping table, determining that storage mark is corresponding and request to read the N number of preassigned spare of data
Node.
Data processing equipment 300 in the present embodiment and aforementioned data processing method be based on the same inventive concept under two
A aspect is in front described in detail the implementation process of method, so those skilled in the art can be according to aforementioned
The structure and implementation process of data processing equipment 300 being well understood in the present embodiment are described, in order to illustrate the succinct of book,
This is just repeated no more.
Based on identical inventive concept, the embodiment of the present invention also provides data processing equipment 400, and Fig. 9 sets for data processing
Standby 400 structural schematic block diagram, comprising: bus 401, and it is connected respectively to the processing unit 402 of bus 401, storage unit
403, transceiver 404.
Wherein, transceiver 404 is used to carry out information transmission with client, and for carrying out information biography with memory node
It is defeated.
Storage unit 403 is for storing instruction.
Processing unit 402 is used to execute the instruction of the storage of storage unit 403, to execute such as when executing described instruction
Lower operation: after the write request that transceiver 404 receives that client is sent, the main memory for storing the data for requesting write-in is determined
Store up node and M standby node, wherein write request includes to request the data of write-in, and M standby node includes N number of preparatory
Specified standby node, M are the positive integer greater than 1, and N is the positive integer less than M；Through transceiver 404 into primary storage node
The data of write-in are requested in write-in；The write-in success message that primary storage node returns is received by transceiver 404；Pass through transceiver
404 are respectively written into the data for requesting write-in to M standby node；It determines and receives in N number of preassigned standby node often
The write-in success message that one node returns respectively；Success message is write to client return by transceiver 404.
Optionally, in the present embodiment, N number of preassigned standby node is that storage performance parameter is full in M standby node
The node of sufficient preset requirement.
Optionally, in the present embodiment, processing unit 402 is also used to: being asked receiving the reading that client is sent in transceiver 404
After asking, the corresponding N number of preassigned standby node for requesting to read data of storage mark is determined, wherein the read request packet contains
Request the mark of reading data；Read by transceiver 404 stored in any node in N number of preassigned standby node with
Identify it is corresponding request to read data, and the data of reading are returned to by client by transceiver 404.
Optionally, in the present embodiment, processing unit 402 is being used for: the data of reading being returned to visitor by transceiver 404
It before the end of family, is also used to: it is homogeneous to determine that store in N number of preassigned standby node corresponding with mark requests to read data
Together.
Optionally, in the present embodiment, data processing equipment 400 is stored with mapping table, mapping table preserve Data Identification with
The corresponding relationship of data storage location, data storage location include N number of preassigned standby node of storing data.Processing is single
Member 402 is used for: determining the corresponding N number of preassigned standby node for requesting to read data of storage mark, comprising: according to mark
And mapping table, determine the corresponding N number of preassigned standby node for requesting to read data of storage mark.
Data processing equipment 400 in the present embodiment and aforementioned data processing method be based on the same inventive concept under two
A aspect is in front described in detail the implementation process of method, so those skilled in the art can be according to aforementioned
The structure and implementation process of data processing equipment 400 being well understood in the present embodiment are described, in order to illustrate the succinct of book,
This is just repeated no more.
It should be noted that bus 401 can be industry standard architecture (English: Industry Standard
Architecture；Referred to as: ISA) bus, external equipment interconnection (English: Peripheral Component；It is referred to as: PCI) total
Line or extended industry-standard architecture (English: Extended Industry Standard Architecture；Referred to as:
EISA) bus etc..Bus 401 can specifically be divided into address bus, data/address bus, control bus etc..For convenient for indicating, in Fig. 9
It is only indicated with a line, it is not intended that an only bus or a type of bus.
Processing unit 402 can be a processor, be also possible to the general designation of multiple processing elements.For example, processing unit
402 can be central processing unit (English: Central Processing Unit；Referred to as: CPU), it is also possible to specific integrated electricity
Road (English: Application Specific Intergrated Circuit；Referred to as: ASIC), or it is arranged to reality
One or more integrated circuits of the embodiment of the present invention are applied, such as: one or more microprocessors (English: digital
singnal processor；Referred to as: DSP), or, one or more field programmable gate array (English: Field
Programmable Gate Array；Referred to as: FPGA).
Storage unit 403 can be a storage device, be also possible to the general designation of multiple memory elements, and can for storing
Parameter, data required for execution program code, access network equipment or terminal operating etc..And memory may include random access memory
(English: Random Access Memory；It referred to as: RAM), also may include nonvolatile memory (non-volatile
Memory), such as magnetic disk storage, flash memory (Flash) etc..
The one or more technical solutions provided in the embodiment of the present invention, have at least the following technical effects or advantages:
In the embodiment of the present invention, calculate node is when being written data into memory node according to write request, to primary storage
After data success is written in node, be only all written in N number of preassigned standby node after data success just to client
End returns and writes success message, wherein N number of preassigned standby node can be the desired standby node of user.Therefore, on
Stating technical solution can guarantee to be successfully written data on the desired standby node of user, meet the needs of user.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to include these modifications and variations.
Claims (12)
1. a kind of data processing method characterized by comprising
Calculate node receives the write request that client is sent, and the write request includes to request the data of write-in；
The calculate node determines the primary storage node and M standby node for storing the data for requesting write-in, wherein
The M standby node includes N number of preassigned standby node, and M is the positive integer greater than 1, and N is the positive integer less than M；
The data for requesting write-in are written into the primary storage node for the calculate node；
The calculate node receives the write-in success message that the primary storage node returns；
The calculate node is respectively written into the data for requesting write-in to the M standby node；
The calculate node determination receives the write-in that each node returns respectively in N number of preassigned standby node
Success message；
The calculate node writes success message to client return.
2. the method as described in claim 1, which is characterized in that N number of preassigned standby node is the M spare
Storage performance parameter meets the node of preset requirement in node.
3. method according to claim 1 or 2, which is characterized in that write as in the calculate node to client return
After function message, further includes:
The calculate node receives the read request that the client is sent, and the read request packet is containing the mark for requesting reading data；
The calculate node determines the corresponding N number of preassigned spare section for requesting to read data of the storage mark
Point；And
It reads store in any node in N number of preassigned standby node corresponding with the mark and requests reading
According to, and the data of reading are returned into the client.
4. method as claimed in claim 3, which is characterized in that the data of reading are returned to institute in the calculate node
Before stating client, further includes:
The calculate node determines that store in N number of preassigned standby node corresponding with the mark requests to read
Data are all the same.
5. method as claimed in claim 3, which is characterized in that the calculate node is stored with mapping table, and the mapping table is protected
There is the corresponding relationship of Data Identification and data storage location, the data storage location includes the described N number of pre- of storing data
First specified standby node；
The calculate node determines the corresponding N number of preassigned spare section for requesting to read data of the storage mark
Point, comprising:
The calculate node determines that the storage mark is corresponding and requests to read data according to the mark and the mapping table
N number of preassigned standby node.
6. method as claimed in claim 4, which is characterized in that the calculate node is stored with mapping table, and the mapping table is protected
There is the corresponding relationship of Data Identification and data storage location, the data storage location includes the described N number of pre- of storing data
First specified standby node；
The calculate node determines the corresponding N number of preassigned spare section for requesting to read data of the storage mark
Point, comprising:
The calculate node determines that the storage mark is corresponding and requests to read data according to the mark and the mapping table
N number of preassigned standby node.
7. a kind of data processing equipment characterized by comprising
Receiving module, for receiving the write request of client transmission, the write request includes to request the data of write-in；
Determining module, for determining primary storage node and M standby node for storing the data for requesting write-in,
In, the M standby node includes N number of preassigned standby node, and M is the positive integer greater than 1, and N is just whole less than M
Number；
Writing module, for the data for requesting write-in to be written into the primary storage node；
The receiving module is also used to: receiving the write-in success message that the primary storage node returns；
The write module is also used to: receiving the write-in success message that the primary storage node returns in the receiving module
Afterwards, Xiang Suoshu M standby node is respectively written into the data for requesting write-in；
The determining module is also used to: determining that receive each node in N number of preassigned standby node returns respectively
The write-in success message returned；
Described device further include: sending module, it is described N number of preassigned spare for being received in determining module determination
After the write-in success message that each node returns respectively in node, success message is write in the return of Xiang Suoshu client.
8. device as claimed in claim 7, which is characterized in that the determining module is used for: determining in the M standby node
N number of node that storage performance parameter meets preset requirement is the preassigned standby node.
9. device as claimed in claim 7 or 8, it is characterised in that:
The receiving module is also used to: receiving the read request that the client is sent, the read request packet, which contains, requests to read data
Mark；
The determining module is also used to: being determined that the storage mark is corresponding and is requested to read the described N number of preassigned of data
Standby node；
Described device further include: read module stores in any node for reading in N number of preassigned standby node
It is corresponding with the mark request read data；
The sending module is also used to: the data of reading are returned to the client.
10. device as claimed in claim 9, which is characterized in that the determining module is also used to: determining N number of preparatory finger
Stored in fixed standby node it is corresponding with the mark request read data it is all the same；
The sending module is used for: the determining module determine stored in N number of preassigned standby node with institute
State mark it is corresponding request reading data it is all the same after, the data of reading are returned into the client.
11. device as claimed in claim 9, which is characterized in that described device is stored with mapping table, and the mapping table is preserved
The corresponding relationship of Data Identification and data storage location, the data storage location include N number of preparatory finger of storing data
Fixed standby node；The determining module is used for: according to the mark and the mapping table, determining that storing the mark corresponds to
Request read data N number of preassigned standby node.
12. device as claimed in claim 10, which is characterized in that described device is stored with mapping table, and the mapping table saves
There is the corresponding relationship of Data Identification and data storage location, the data storage location includes the described N number of preparatory of storing data
Specified standby node；The determining module is used for: according to the mark and the mapping table, determining the storage mark pair
That answers requests to read N number of preassigned standby node of data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510547258.8A CN106484311B (en) | 2015-08-31 | 2015-08-31 | A kind of data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510547258.8A CN106484311B (en) | 2015-08-31 | 2015-08-31 | A kind of data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106484311A CN106484311A (en) | 2017-03-08 |
| CN106484311B true CN106484311B (en) | 2019-07-19 |
Family
ID=58236081
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510547258.8A Active CN106484311B (en) | 2015-08-31 | 2015-08-31 | A kind of data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106484311B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107037987A (en) * | 2017-04-01 | 2017-08-11 | 北京华云网际科技有限公司 | To the treating method and apparatus rolled up in distributed memory system |
| CN107797945A (en) * | 2017-10-31 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of storage system and its date storage method, device, system and equipment |
| CN107908370B (en) * | 2017-11-30 | 2021-07-06 | 新华三技术有限公司 | Data storage method and device |
| CN110196680B (en) * | 2018-03-27 | 2021-10-26 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| CN108462756B (en) * | 2018-03-29 | 2020-11-06 | 新华三技术有限公司 | Data writing method and device |
| CN110955382A (en) * | 2018-09-26 | 2020-04-03 | 华为技术有限公司 | Method and device for writing data in distributed system |
| JP7313458B2 (en) * | 2019-09-18 | 2023-07-24 | 華為技術有限公司 | Storage system, storage node and data storage method |
| CN111309805B (en) * | 2019-12-13 | 2023-10-20 | 华为技术有限公司 | Data reading and writing method and device for database |
| CN113296684A (en) * | 2020-05-12 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Data writing method and device in distributed system and distributed system |
| CN113301084A (en) * | 2020-06-30 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN111857603B (en) * | 2020-07-31 | 2022-12-02 | 重庆紫光华山智安科技有限公司 | Data processing method and related device |
| CN112347335B (en) * | 2020-10-30 | 2024-06-04 | 银盛支付服务股份有限公司 | Method, system, computer equipment and storage medium for efficient retrieval of elastic search |
| CN113360287B (en) * | 2021-06-21 | 2022-09-23 | 上海哔哩哔哩科技有限公司 | Data processing method and device |
| CN114979180B (en) * | 2022-05-24 | 2024-05-17 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1336068A (en) * | 1998-10-20 | 2002-02-13 | 安德鲁·杜根 | An intelligent network |
| CN101895472A (en) * | 2010-07-16 | 2010-11-24 | 华为技术有限公司 | Traffic flow load sharing method and processing method and corresponding device and system |
| CN102546782A (en) * | 2011-12-28 | 2012-07-04 | 北京奇虎科技有限公司 | Distribution system and data operation method thereof |
| CN102594849A (en) * | 2011-01-06 | 2012-07-18 | 阿里巴巴集团控股有限公司 | Data backup and recovery method and device, virtual machine snapshot deleting and rollback method and device |
| CN103856511A (en) * | 2012-11-30 | 2014-06-11 | 腾讯科技（深圳）有限公司 | Data packet upload method, client terminal, nodes, information server and system |
-
2015
- 2015-08-31 CN CN201510547258.8A patent/CN106484311B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1336068A (en) * | 1998-10-20 | 2002-02-13 | 安德鲁·杜根 | An intelligent network |
| CN101895472A (en) * | 2010-07-16 | 2010-11-24 | 华为技术有限公司 | Traffic flow load sharing method and processing method and corresponding device and system |
| CN102594849A (en) * | 2011-01-06 | 2012-07-18 | 阿里巴巴集团控股有限公司 | Data backup and recovery method and device, virtual machine snapshot deleting and rollback method and device |
| CN102546782A (en) * | 2011-12-28 | 2012-07-04 | 北京奇虎科技有限公司 | Distribution system and data operation method thereof |
| CN103856511A (en) * | 2012-11-30 | 2014-06-11 | 腾讯科技（深圳）有限公司 | Data packet upload method, client terminal, nodes, information server and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106484311A (en) | 2017-03-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106484311B (en) | A kind of data processing method and device | |
| CN102736945B (en) | A kind of method and system of the Multi-instance running application | |
| US20190166192A1 (en) | Method and Device for Storage Resource Allocation for Video Cloud Storage | |
| RU2651219C2 (en) | Computer, control device and data processing method | |
| CN103324533B (en) | distributed data processing method, device and system | |
| US10235047B2 (en) | Memory management method, apparatus, and system | |
| CN104731635B (en) | A kind of virtual machine access control method and virtual machine access control system | |
| JP2007026094A (en) | Execution device and application program | |
| CN110837479B (en) | Data processing method, related equipment and computer storage medium | |
| CN110119304A (en) | A kind of interruption processing method, device and server | |
| US9946721B1 (en) | Systems and methods for managing a network by generating files in a virtual file system | |
| CN109391487A (en) | A kind of configuration update method and system | |
| CN106293966B (en) | The AP layers of communication means and LFS system with SP layers in a kind of LFS system | |
| CN110019444A (en) | A kind of operation requests processing method, device, equipment and system | |
| CN105391755B (en) | Data processing method, apparatus and system in a kind of distributed system | |
| US9021492B2 (en) | Dual mode reader writer lock | |
| CN110618883A (en) | Method, device, equipment and storage medium for sharing memory linked list | |
| CN105471930B (en) | A kind of method, system and equipment reading distributed data | |
| CN105264608B (en) | Method, Memory Controller Hub and the central processing unit of data storage | |
| CN109213701A (en) | Control method, system, equipment and the storage medium of the peripheral hardware of shared I O board | |
| WO2024066380A1 (en) | Pod scheduling method and apparatus, and computer device | |
| EP3249540A1 (en) | Method for writing multiple copies into storage device, and storage device | |
| CN106557429B (en) | A kind of moving method and Node Controller of internal storage data | |
| CN105988871A (en) | Remote memory allocation method, device and system | |
| CN114675954A (en) | Task scheduling method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220907 Address after: No. 1899 Xiyuan Avenue, high tech Zone (West District), Chengdu, Sichuan 610041 Patentee after: Chengdu Huawei Technologies Co.,Ltd. Address before: 611731 Qingshui River District, Chengdu hi tech Zone, Sichuan, China Patentee before: HUAWEI DIGITAL TECHNOLOGIES (CHENG DU) Co.,Ltd. |