WO2017088572A1 - 一种处理数据的方法、装置及系统 - Google Patents
一种处理数据的方法、装置及系统 Download PDFInfo
- Publication number
- WO2017088572A1 WO2017088572A1 PCT/CN2016/099590 CN2016099590W WO2017088572A1 WO 2017088572 A1 WO2017088572 A1 WO 2017088572A1 CN 2016099590 W CN2016099590 W CN 2016099590W WO 2017088572 A1 WO2017088572 A1 WO 2017088572A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- written
- node
- nodes
- storage system
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/065—Replication mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/10—Payment architectures specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
Definitions
- the present invention relates to the field of computer technologies, and in particular, to a method, device and system for processing data.
- a distributed storage system distributes data across multiple independent devices to improve the reliability, availability, and storage efficiency of data storage.
- a distributed storage system includes a plurality of nodes that store data, and the data stored by each node is ultimately consistent. However, in the process of data writing, since the data written by each node is not synchronized, it is determined whether the data is successfully written into the distributed storage system by whether the number of nodes writing data in the memory reaches a threshold.
- the data stored by each node in the distributed storage system may be inconsistent.
- the quorum (Quorum) mechanism is used to read the data: the client sends a read request to the R nodes in the distributed storage system, NW ⁇ R ⁇ N and N, W, and R are integers, N is the number of nodes in the distributed storage system, and W is the threshold number of nodes for judging whether the data is successfully written into the distributed storage system; R nodes send the respective storages to the client according to the read request. Data; the client selects the latest data of the version number in the data sent by the R nodes.
- the client selects the latest data of the version number in the data sent by the R nodes, but the number of nodes that write the data in the memory may be less than W, that is, the data is not successfully written to the distributed storage system, causing the client to read the data error. , causing inconvenience to the user.
- the embodiment of the present invention provides a method, device, and system for processing data.
- the technical solution is as follows:
- an embodiment of the present invention provides a method for processing data, where the method includes:
- the distributed storage system comprising N nodes, N ⁇ 2 and N being an integer;
- the data sent by the R nodes are all written into the memory of at least W nodes of the N nodes, To determine whether the data is successfully written to the node number threshold of the distributed storage system, N-W ⁇ R ⁇ N and R and W are integers;
- the latest data of the version number in the data sent by the R nodes is selected.
- the data passing through the R nodes is written into the memory of at least W nodes of the N nodes, and W is a threshold number of nodes for judging whether the data is successfully written into the distributed storage system, thereby avoiding the data due to the reading. A situation in which the data read by the client is incorrectly written to the distributed storage system.
- NW ⁇ R ⁇ N the latest data of the version number of the data of the R nodes
- N is the number of nodes of the distributed storage system, thereby ensuring reading.
- the data is up-to-date, achieving strong data consistency and providing convenience for users.
- an embodiment of the present invention provides a method for processing data, where the method includes:
- W is a threshold value for determining whether the data is successfully written into the distributed storage system, NW ⁇ R ⁇ N, W ⁇ N, N ⁇ 2, and N, R, W is an integer.
- the data passing through the R nodes is written into the memory of at least W nodes of the N nodes, and W is a threshold number of nodes for judging whether the data is successfully written into the distributed storage system, thereby avoiding the data due to the reading. A situation in which the data read by the client is incorrectly written to the distributed storage system.
- NW ⁇ R ⁇ N the latest data of the version number of the data of the R nodes
- N is the number of nodes of the distributed storage system, thereby ensuring reading.
- the data is up-to-date, achieving strong data consistency and providing convenience for users.
- the method further includes:
- the response message is used to indicate that the data to be written is successfully written A distributed storage system.
- the distributed storage system When the number of nodes in the memory to write data to be written is less than W, the distributed storage system does not send a response message to the second client, and if the second client does not receive the response message at the set time, it determines that it is to be written.
- the incoming data is not successfully written to the distributed storage system, and the response message sent by the distributed storage system is received when the write request is resent and the data to be written is written until the number of nodes in the memory to write the data to be written is greater than or equal to W. To ensure that the read data is successfully written to the distributed storage system.
- the receiving the write request sent by the second client includes:
- the first node in the distributed storage system receives the write request, and the first node is a set node in the distributed storage system or any one of the distributed storage systems.
- the first node may be a set node in the distributed storage system, or may be any node in the distributed storage system. If the first node is any one of the distributed storage systems, the problem that the data to be written cannot be written due to the setting node failure can be effectively avoided. Moreover, multiple nodes in the distributed storage system can simultaneously receive write requests for data writing, and data writing efficiency is improved compared to only one setting node in the distributed storage system can receive write requests for data writing. N times, N is the number of nodes of the distributed storage system.
- the writing the data to be written into the memory of each node in the distributed storage system according to the write request includes:
- the write request is used to instruct the other node to write the data to be written into a respective disk;
- the first node When the number of the received accept messages is greater than or equal to W, the first node writes the to-be-written data into the memory, and sends a commitment message to the other node, where the commitment message is used Instructing the other node to write the data to be written into a respective memory.
- the node fails, such as power failure, the data in the memory will be lost. At this time, the memory will be restored according to the data in the disk after the failure recovery.
- data is first written to disk, ensuring that data can be written to memory regardless of node failure.
- sending a response message to the second client includes:
- the first node When the number of the received confirmation messages is greater than or equal to W, the first node sends the response message to the second client.
- the distributed storage system when the number of nodes in the memory to write data to be written is less than W, the distributed storage system does not send a response message to the second client, and the second client does not receive the response message at the set time. , determining that the data to be written is not successfully written to the distributed storage system, resending the write request and writing the data to be written until the number of nodes in the memory to write the data to be written is greater than or equal to W, receiving the distributed storage A response message sent by the system to ensure that the read data is successfully written to the distributed storage system.
- an embodiment of the present invention provides a method for processing data, where the method includes:
- the response message is sent to the second client, where W is a threshold value for determining whether the data is successfully written into the distributed storage system.
- W ⁇ N and W is a positive integer, and the response message is used to indicate that the data to be written is successfully written to the distributed storage system.
- the receiving, by the second client, the write request includes:
- the first node in the distributed storage system receives the write request, and the first node is a set node in the distributed storage system or any one of the distributed storage systems.
- the writing, in the memory of each node in the distributed storage system, according to the write request includes:
- the write request is used to instruct the other node to write the data to be written into a respective disk;
- the first node When the number of the received accept messages is greater than or equal to W, the first node writes the to-be-written data into the memory, and sends a commitment message to the other node, where the commitment message is used Instructing the other node to write the data to be written into a respective memory.
- sending a response message to the second client including:
- the first node When the number of the received confirmation messages is greater than or equal to W, the first node sends the response message to the second client.
- an embodiment of the present invention provides an apparatus for processing data, where the apparatus includes a module for implementing the method described in the foregoing first aspect, such as a read request sending module, a data receiving module, and a data selecting module.
- an embodiment of the present invention provides an apparatus for processing data, where the apparatus is used to implement a module of the method described in the foregoing second aspect, such as a read request receiving module and a data sending module.
- an embodiment of the present invention provides an apparatus for processing data, where the apparatus is used to implement a module of the method described in the foregoing third aspect, such as a write request receiving module, a data writing module, and a write response sending. Module.
- an embodiment of the present invention provides a system for processing data, where the system includes:
- a first client configured to send a read request to the distributed storage system, where the distributed storage system includes N nodes, N ⁇ 2 and N is an integer;
- the distributed storage system is configured to send data of R nodes in the distributed storage system to the first client according to the read request, where data of the R nodes are all written by the N
- W is a threshold value for determining whether the data is successfully written into the distributed storage system
- N-W ⁇ R ⁇ N, W ⁇ N, and R and W are integers
- the first client is further configured to select the latest data of the version number in the data sent by the R nodes.
- an embodiment of the present invention provides a system for processing data, where the system includes:
- a second client configured to send a write request to the distributed storage system, where the distributed storage system includes N nodes, N ⁇ 2 and N is an integer, and the write request includes data to be written;
- the distributed storage system is configured to write the to-be-written data into a memory of each node in the distributed storage system according to the write request; when the data to be written is written in a memory
- W When the number of nodes is greater than or equal to W, a response message is sent to the second client, W ⁇ N and W is an integer, and the response message is used to indicate that the data to be written is successfully written into the distributed storage system.
- the data passing through the R nodes is written into the memory of at least W nodes of the N nodes, and W is a threshold number of nodes for judging whether the data is successfully written into the distributed storage system, thereby avoiding the data due to the reading. A situation in which the data read by the client is incorrectly written to the distributed storage system.
- NW ⁇ R ⁇ N the latest data of the version number of the data of the R nodes
- N is the number of nodes of the distributed storage system, thereby ensuring reading.
- the data is up-to-date, achieving strong data consistency and providing convenience for users.
- FIG. 1 is an application scenario diagram of a method for processing data according to an embodiment of the present invention
- FIG. 2 is a schematic structural diagram of a client provided by an embodiment of the present invention.
- FIG. 3 is a schematic structural diagram of a node in a distributed storage system according to an embodiment of the present invention.
- FIG. 4 is a flowchart of a method for processing data according to an embodiment of the present invention.
- FIG. 5 is a schematic structural diagram of an apparatus for processing data according to an embodiment of the present disclosure.
- FIG. 6 is a schematic structural diagram of an apparatus for processing data according to an embodiment of the present invention.
- FIG. 7 is a schematic structural diagram of a system for processing data according to an embodiment of the present invention.
- the distributed storage system 10 includes a plurality of nodes 11 that store data, and the nodes 11 are connected to each other.
- At least one client 20 is coupled to the distributed storage system 10, and the client 20 can write data to the distributed storage system 10 or read data from the distributed storage system 10.
- the distributed storage system 10 and the client 20 can be applied to a banking transaction system, a mail system, an advertisement service system, a service inquiry system, and the like.
- the funds in the account A are transferred to the account B, and the user of the account A logs in to the account A through the client a for fund transfer, and the distributed storage system will update according to the write request of the client a.
- the funds data is written into account A and account B, respectively, and the updated fund data of account A is fed back to client a.
- the user of account A can know that the funds are successfully transferred through client a, which is the data writing process.
- the user of the account B logs in to the account B through the client b, and the distributed storage system feeds back the fund data updated by the account B before the account B to the client b according to the query request of the client b, and the account B of the account B
- the user can also know the successful transfer of funds through the client b, which is the data reading process.
- a user queries the service information stored in the distributed storage system b through the client c, and the distributed storage system b feeds back the corresponding service information to the client according to the read request of the client c. c, this is the data reading process.
- the distributed storage system 10 may be a distributed database system, a cluster configuration database (Cluster Config Database, CCDB for short), Zookeeper (distributed, open source distributed application coordination service), and the like.
- the client 20 can be a mobile terminal, a tablet, a laptop, a desktop computer, or the like.
- the client 20 can adopt the structure shown in FIG. 2, and the client 20 can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc., as shown in FIG. 2, the client 20 includes a memory 210 and a processor 220. Transmitter 230 and receiver 240.
- the client structure illustrated in FIG. 2 does not constitute a limitation to the client, may include more or fewer components than illustrated, or combine some components, or different component arrangements. among them:
- the processor 220 is the control center of the client 20, connecting various portions of the entire client 20 using various interfaces and lines, by running or executing software programs and/or modules stored in the memory 210, and by calling them stored in the memory 210.
- processor 220 can include one or more processing cores.
- the memory 210 can be used to store various data, such as various configuration parameters, as well as to store software programs and modules.
- the processor 220 executes various functional applications and data processing by running software programs and modules stored in the memory 210.
- the memory 210 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system 211, the read request sending module 212, the data receiving module 213, and the data selecting module 214, and may also store the write request sending module 215.
- the write response receiving module 216 and the like; the storage data area can store data created according to the use of the client 20, such as data read according to the read request.
- the memory 210 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read only memory. (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), read only Read Only Memory (ROM), magnetic memory, flash memory, disk or optical disk. Accordingly, memory 210 may also include a memory controller to provide processor 220 access to memory 210.
- SRAM static random access memory
- EEPROM Electrically erasable programmable Read Only Memory
- EPROM Erasable Programmable Read Only Memory
- PROM Programmable Read-Only Memory
- ROM Read Only Memory
- magnetic memory flash memory
- flash memory disk or optical disk.
- memory 210 may also include a memory controller to provide processor 220 access to memory 210.
- the transmitter 230 is configured to send, according to an instruction of the processor 220, the read request to the distributed storage system 10, so that the distributed storage system 10 reads the data according to the read request, and sends the read data to the client. 20.
- the transmitter 230 is further configured to send a write request to the distributed storage system 10.
- the receiver 240 is configured to receive data read by the distributed storage system 10 according to the read request.
- the receiver 240 is further configured to receive data into the distributed storage system 10 at least W Response message sent after the memory of the node.
- the node 10 in the distributed storage system may adopt the structure shown in FIG. 3.
- the node 10 in the distributed storage system includes a memory 110, a processor 120, a transmitter 130, and a receiver 140.
- the node structure in the distributed storage system 10 illustrated in FIG. 3 does not constitute a limitation on nodes in the distributed storage system, may include more or fewer components than illustrated, or may combine certain components, or different Parts layout.
- Processor 120 is a control center for nodes in distributed storage system 10 that connects various portions of nodes throughout distributed storage system 10 using various interfaces and lines, by running or executing software programs stored in memory 110 and/or Or modules, as well as invoking data stored in memory 110, perform various functions and processing data for nodes in distributed storage system 10 to provide overall control of nodes in distributed storage system 10.
- processor 120 may include one or more processing cores.
- the memory 110 can be used to store various data, such as various configuration parameters, as well as to store software programs and modules.
- the processor 120 executes various functional applications and data processing by running software programs and modules stored in the memory 110.
- the memory 110 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system 111, the read request receiving module 112, the data sending module 113, and may also store the write request receiving module 114 and the data writing module. 115.
- Write response transmitting module 116 and the like; the storage data area may store data created according to the use of nodes in the distributed storage system 10, such as data read according to a read request.
- the memory 110 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read only memory. (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), read only Read Only Memory (ROM), magnetic memory, flash memory, disk or optical disk. Accordingly, memory 110 may also include a memory controller to provide processor 120 access to memory 110.
- SRAM Static Random Access Memory
- EEPROM Electrically erasable programmable Read only memory
- EPROM Erasable Programmable Read Only Memory
- PROM Programmable Read-Only Memory
- ROM Read Only Memory
- magnetic memory flash memory
- flash memory disk or optical disk.
- memory 110 may also include a memory controller to provide processor 120 access to memory 110.
- the receiver 140 is configured to receive a read request sent by the client 20.
- the sender 130 is further configured to receive a write request sent by the client 20.
- the transmitter 130 is configured to send the distributed storage to the client 20 according to the instruction of the processor 120. Data for R nodes in system 10.
- the sender 130 is further configured to send, according to the instruction of the processor 120, a response message sent by the client 20 after the data is written into the memory of at least the W nodes in the distributed storage system 10.
- FIG. 4 is a schematic diagram of a method for processing data according to an embodiment of the present invention. The method is applied to the client shown in FIG. 2 and the distributed storage system shown in FIG. 3. Referring to FIG. 4, the method includes :
- Step 301 The second client sends a write request to the distributed storage system.
- the distributed storage system includes N nodes, N ⁇ 2 and N is an integer.
- a node can be a server, and each server is independent of each other.
- the write request includes data to be written. It can be understood that the write request can also include a destination address, a source address, and an identifier of the data to be written.
- the destination address is used to accurately send the write request to the distributed storage system;
- the source address is used by the distributed storage system to respond to the second client after receiving the write request, such as sending a response message;
- the identifier of the data to be written is used for Differentiating individual data, nodes in the distributed storage system can determine the storage location of the data according to the identification.
- the step 301 can include:
- the second client sends a write request to the first node in the distributed storage system.
- the configuration file when configuring the client and the distributed storage system, may be stored in the client and the distributed storage system, and the configuration file includes the number of nodes in the distributed storage system, and determines whether the data is successfully written and distributed.
- the first node may be a set node in the distributed storage system, or may be any one of the distributed storage systems. If the first node is a set node in the distributed storage system, the first node may be manually set, or may be negotiated by all nodes in the distributed storage system (the first set node changes when notified) Each node and client) is not limited by the present invention.
- the problem that the data to be written cannot be written due to the setting node failure can be effectively avoided.
- multiple nodes in the distributed storage system can simultaneously receive write requests for data writing, and data writing efficiency is improved compared to only one setting node in the distributed storage system can receive write requests for data writing. N times, N The number of nodes for a distributed storage system.
- Step 302 The distributed storage system writes the data to be written into the memory of each node in the distributed storage system according to the write request.
- the client since the client reads data from the memory of at least one node in the distributed storage system, the data to be written needs to be written into the memory of each node in the distributed storage system.
- the step 302 can include:
- the first node writes the data to be written to the disk according to the write request, and sends a write request to other nodes in the distributed storage system, and the other nodes are all nodes except the first node in the distributed storage system;
- the other nodes write the data to be written into the respective disks according to the write request, and send an accept message to the first node;
- the first node receives and counts the accept message sent by other nodes
- the first node When the number of received accept messages is greater than or equal to W, the first node writes the data to be written into the memory, and sends a commitment message to other nodes;
- the other nodes write the data to be written into their respective memories according to the commitment message.
- W ⁇ N and W is a positive integer
- N is the number of nodes of the distributed storage system
- W is a threshold number of nodes for determining whether data is successfully written into the distributed storage system.
- Step 303 When the number of nodes in the memory to be written data is greater than or equal to W, the distributed storage system sends a response message to the second client.
- the response message is used to indicate that the data to be written is successfully written to the distributed storage system.
- the distributed storage system does not send a response message to the second client, and if the second client does not receive the response message at the set time, Then, it is determined that the data to be written is not successfully written into the distributed storage system, and step 301 is restarted until the number of nodes in the memory to be written data is greater than or equal to W, and the response message sent by the distributed storage system is received, thereby Ensure that the read data is successfully written to the distributed storage system.
- the step 303 can include:
- the other node sends a confirmation message to the first node after the data to be written is written into the respective memory;
- the first node receives and counts the acknowledgement message sent by other nodes
- the first node When the number of received acknowledgment messages is greater than or equal to W, the first node sends a response message to the second client.
- steps 301-303 realize writing data into the distributed storage system.
- Step 304 The first client sends a read request to the distributed storage system.
- the first client and the second client may be different clients.
- the read request may include a destination address, a source address, and an identification of the data to be read.
- the destination address is used to accurately send the read request to the distributed storage system;
- the source address is used by the distributed storage system to respond to the first client after receiving the read request, such as data to be read;
- the nodes in the distributed storage system can determine the storage location of the data according to the identifier.
- the step 304 may include:
- the first client sends a read request to R nodes in the distributed storage system.
- N-W ⁇ R ⁇ N and R is an integer
- N is the number of nodes of the distributed storage system
- W is a threshold number of nodes for judging whether data is successfully written into the distributed storage system.
- the R nodes that receive the read request in the distributed storage system are arbitrary, so that multiple groups of R nodes can receive different read requests for data reading at the same time, and only the distributed storage system Compared with a set node receiving a read request for data reading, the query rate per second (Query Per Second, QPS for short) is raised to 1+(N-1)/R times, and N is the number of nodes of the distributed storage system.
- QPS query Per Second
- the configuration file when configuring a client and a distributed storage system, can be stored in the client and the distributed storage system, and the configuration file includes the number of nodes in the distributed storage system, and determines whether the data is successfully written and distributed.
- the cluster system the number of nodes, the address of each node, and the like, the first client directly obtains the addresses of the R nodes from the configuration file, and sends a write request according to the address.
- Step 305 The distributed storage system sends data of R nodes in the distributed storage system to the first client according to the read request.
- the data of the R nodes are all written in the memory of at least W nodes of the N nodes, NW ⁇ R ⁇ N, N is the number of nodes of the distributed storage system, and W is determining whether the data is successful. The number of nodes written to the distributed storage system.
- data of R nodes are written into the memory of at least W nodes of the N nodes, and at least one of the R nodes is written into the memory of at least W nodes of the N nodes, and R The other data in the nodes has been written into the memory of at least W of the N nodes.
- the data of the R nodes includes the data A and the data B, and the data B is the updated data of the data A.
- the data A is written into the memory of at least the W nodes of the N nodes.
- the data B is written into the memory of at least W of the N nodes.
- the step 305 can include:
- the R nodes in the distributed storage system send respective data to the first client according to the read request.
- Step 306 The first client selects the latest data of the version number in the data of the R nodes.
- the version number is a number used to indicate that the data is old or new during the data writing process.
- the data with the version number V2 is the updated data of the version number V1. That is, if the data read by the first client includes the data with the version number V2 and the data with the version number V1, the first client selects the data with the version number V2.
- steps 304-306 realize reading data from the distributed storage system.
- step 301 is performed by the client according to the read request sending module 212 and the distributed storage system in FIG. 2 according to the read request receiving module 112 in FIG. 3, and step 302 is sent by the distributed storage system according to the data in FIG.
- the module 113 and the client are executed according to the data receiving module 213 in FIG. 2, the step 303 is performed by the client according to the data selecting module 214 in FIG. 2, and the step 304 is sent by the client according to the write request in FIG.
- the storage system is executed according to the write request receiving module 114 in FIG. 3, the step 305 is performed by the distributed storage system according to the data writing module 115 in FIG. 3, and the step 306 is performed by the distributed storage system according to the write response in FIG.
- the transmitting module 116 and the client are executed in accordance with the write response receiving module 216 in FIG.
- the apparatus may be implemented as a whole or a part of a client by software, hardware, or a combination of both.
- the apparatus includes a read request sending module 401, a data receiving module 402, and a data selecting module 403.
- the read request sending module 401 is configured to send a read request to the distributed storage system, where the distributed storage system includes N nodes, N ⁇ 2 and N is an integer.
- the data receiving module 402 is used for receiving The R nodes of the N nodes receive data according to the read request, and the data sent by the R nodes are written into the memory of at least W nodes of the N nodes, so that it is determined whether the data is successfully written to the distributed
- the threshold number of nodes of the storage system, NW ⁇ R ⁇ N and R and W are integers.
- the data selection module 403 is configured to select the latest version data of the data sent by the R nodes.
- the apparatus may further include: a write request sending module 404 and a write response receiving module 405.
- the write request sending module 404 is configured to send a write request to the distributed storage system, where the write request includes data to be written.
- the write response receiving module 405 is configured to receive a response message, and the response message is used to indicate that the data to be written is successfully written to the distributed storage system.
- the apparatus may be implemented as a whole or a part of a distributed storage system by software, hardware, or a combination of both.
- the apparatus includes a read request receiving module 501 and a data sending module 502.
- the read request receiving module 501 is configured to receive a read request sent by the first client.
- the data sending module 502 is configured to send data of R nodes in the distributed storage system to the first client according to the read request, where the distributed storage system includes N nodes, and data of the R nodes are written into the N nodes.
- W is a threshold number of nodes for judging whether data is successfully written into the distributed storage system, and NW ⁇ R ⁇ N, W ⁇ N, N ⁇ 2, and N, R, and W are integers.
- the apparatus may further include: a write request receiving module 503, a data writing module 504, and a write response sending module 505.
- the write request receiving module 503 is configured to receive a write request sent by the second client, where the write request includes data to be written.
- the data writing module 504 is configured to write the data to be written into the memory of each node in the distributed storage system according to the write request.
- the write response sending module 505 is configured to send a response message to the second client when the number of nodes in the memory to write the data to be written is greater than or equal to W, and the response message is used to indicate that the data to be written is successfully written into the distributed storage. system.
- the write request receiving module 503 is configured to receive a write request by using a first node in the distributed storage system, where the first node is one of a set node or a distributed storage system in the distributed storage system. node.
- the data writing module 504 can include a disk writing unit 504a, an accept message counting unit 504b, and a memory writing unit 504c.
- the disk writing unit 504a is configured to write the data to be written into the first section according to the write request. Point the disk and send a write request to other nodes in the distributed storage system.
- the other nodes are all nodes except the first node in the distributed storage system, and the write request is used to indicate that other nodes are to be written. Data is written to the respective disk.
- the accept message statistics unit 504b is configured to receive and count the accept messages sent by other nodes, and the accept message is sent by other nodes after the data to be written is written to the respective disks.
- the memory writing unit 504c is configured to write the data to be written into the memory of the first node when the number of received receiving messages is greater than or equal to W, and send a commitment message to the other node, the commitment message is used to indicate other
- the node writes the data to be written to its own memory.
- the write response transmitting module 505 may include an acknowledgement message counting unit 505a and a write response transmitting unit 505b.
- the acknowledgment message statistic unit 505a is configured to receive and count the acknowledgment messages sent by other nodes, and the acknowledgment message is sent by the other node to the first node after the data to be written is written into the respective memory.
- the write response sending unit 505b is configured to send a response message to the second client when the number of received acknowledgement messages is greater than or equal to W.
- the system may include a first client 601 and a distributed storage system 602.
- the first client 601 is configured to send a read request to the distributed storage system 602.
- the distributed storage system 602 includes N nodes, N ⁇ 2 and N is an integer.
- the distributed storage system 602 is configured to send data of R nodes in the distributed storage system to the first client 601 according to the read request, and data of the R nodes are written into the memory of at least W nodes of the N nodes.
- W is a threshold number of nodes for judging whether data is successfully written into the distributed storage system. NW ⁇ R ⁇ N, W ⁇ N, and R and W are integers.
- the first client 603 is further configured to select the latest data of the version number in the data sent by the R nodes.
- system may further include a second client 603.
- the second client 603 is configured to send a write request to the distributed storage system 602, where the write request includes data to be written.
- the distributed storage system 602 is configured to write the data to be written into the memory of each node in the distributed storage system according to the write request; when the number of nodes in the memory to write the data to be written is greater than or equal to W Sending a response message to the second client, where the response message is used to indicate that the data to be written is successfully written to the distributed storage system.
- the apparatus for processing data provided by the foregoing embodiment only processes The division of each of the above functional modules is exemplified. In practical applications, the above function assignments may be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. .
- the apparatus for processing data and the method for processing data provided by the foregoing embodiments are in the same concept, and the specific implementation process is described in detail in the method embodiment, and details are not described herein again.
- a person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium.
- the storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Finance (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer Security & Cryptography (AREA)
Abstract
本发明公开了一种处理数据的方法、装置及系统,属于计算机技术领域。所述方法包括:向分布式存储系统发送读取请求,所述分布式存储系统包括N个节点,N≥2且N为整数;接收所述N个节点中的R个节点根据所述读取请求发送的数据,所述R个节点发送的数据均写入过所述N个节点中的至少W个节点的内存中,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N且R、W均为整数;选择所述R个节点发送的数据中版本号最新的数据。本发明保证读取的数据是最新的,实现了数据的强一致性,为用户使用提供了方便。
Description
本发明涉及计算机技术领域,特别涉及一种处理数据的方法、装置及系统。
分布式存储系统是将数据分散存储在多台独立的设备上,以提高数据存储的可靠性、可用性、以及存储效率。分布式存储系统包括多个存储数据的节点,各个节点存储的数据最终具有一致性。但是在数据写入的过程中,由于各个节点写入数据不同步,因此是以内存中写入数据的节点数量是否达到阈值判断数据是否成功写入分布式存储系统。
基于分布式存储系统中的各个节点存储的数据可能不一致,目前采用法定人数(Quorum)机制读取数据:客户端向分布式存储系统中的R个节点发送读取请求,N-W<R≤N且N、W、R为整数,N为分布式存储系统中的节点数量,W为判断数据是否成功写入分布式存储系统的节点数量阈值;R个节点根据读取请求向客户端发送各自存储的数据;客户端选择R个节点发送的数据中版本号最新的数据。
在实现本发明的过程中,发明人发现现有技术至少存在以下问题:
客户端选择R个节点发送的数据中版本号最新的数据,但是内存中写入该数据的节点数量可能小于W,即该数据并未成功写入分布式存储系统,造成客户端读取数据错误,为用户造成不便。
发明内容
为了解决现有技术造成客户端读取数据错误,为用户造成不便的问题,本发明实施例提供了一种处理数据的方法、装置及系统。所述技术方案如下:
第一方面,本发明实施例提供了一种处理数据的方法,所述方法包括:
向分布式存储系统发送读取请求,所述分布式存储系统包括N个节点,N≥2且N为整数;
接收所述N个节点中的R个节点根据所述读取请求发送的数据,所述R个节点发送的数据均写入过所述N个节点中的至少W个节点的内存中,W
为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N且R、W均为整数;
选择所述R个节点发送的数据中版本号最新的数据。
通过R个节点的数据均写入过N个节点中的至少W个节点的内存中,W为用于判断数据是否成功写入分布式存储系统的节点数量阈值,从而避免了由于读取的数据未成功写入分布式存储系统而造成客户端读取的数据错误的情况。而且通过获取分布式存储系统中的R个节点的数据,并选择R个节点的数据中版本号最新的数据,N-W<R≤N,N为分布式存储系统的节点数量,从而保证读取的数据是最新的,实现了数据的强一致性,为用户使用提供了方便。
第二方面,本发明实施例提供了一种处理数据的方法,所述方法包括:
接收第一客户端发送的读取请求;
根据所述读取请求向所述第一客户端发送分布式存储系统中的R个节点的数据,所述分布式存储系统包括N个节点,所述R个节点的数据均写入过所述N个节点中的至少W个节点的内存中,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N、W≤N、N≥2且N、R、W均为整数。
通过R个节点的数据均写入过N个节点中的至少W个节点的内存中,W为用于判断数据是否成功写入分布式存储系统的节点数量阈值,从而避免了由于读取的数据未成功写入分布式存储系统而造成客户端读取的数据错误的情况。而且通过获取分布式存储系统中的R个节点的数据,并选择R个节点的数据中版本号最新的数据,N-W<R≤N,N为分布式存储系统的节点数量,从而保证读取的数据是最新的,实现了数据的强一致性,为用户使用提供了方便。
在第二方面一种可能的实现方式中,所述方法还包括:
接收第二客户端发送的写入请求,所述写入请求包括待写入数据;
根据所述写入请求将所述待写入数据写入所述分布式存储系统中的各个节点的内存中;
当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,所述响应消息用于表示所述待写入数据成功写入所
述分布式存储系统。
当内存中写入待写入数据的节点数量小于W时,分布式存储系统不会向第二客户端发送响应消息,第二客户端若在设定时间没有收到响应消息,则判定待写入数据未成功写入分布式存储系统,重新发送写入请求和写入待写入数据直到内存中写入待写入数据的节点数量大于或等于W时接收到分布式存储系统发送的响应消息,从而确保读取的数据成功写入分布式存储系统中。
可选地,所述接收第二客户端发送的写入请求,包括:
所述分布式存储系统中的第一节点接收所述写入请求,所述第一节点为所述分布式存储系统中的一个设定节点或者所述分布式存储系统中的任意一个节点。
第一节点可以为分布式存储系统中的一个设定节点,也可以为分布式存储系统中的任意一个节点。若第一节点为分布式存储系统中的任意一个节点,则可以有效避免由于设定节点故障而无法写入待写入数据的问题。而且分布式存储系统中的多个节点可以同时接收写入请求进行数据写入,与分布式存储系统中只有一个设定节点可以接收写入请求进行数据写入相比,数据写入效率提升至N倍,N为分布式存储系统的节点数量。
优选地,所述根据所述写入请求将所述待写入数据写入所述分布式存储系统中的各个节点的内存中,包括:
所述第一节点根据所述写入请求将所述待写入数据写入磁盘中,并向所述分布式存储系统中的其它节点发送所述写入请求,所述其它节点为所述分布式存储系统中除所述第一节点之外的所有节点,所述写入请求用于指示所述其它节点将所述待写入数据写入各自的磁盘中;
所述第一节点接收并统计所述其它节点发送的接受消息,所述接受消息是所述其它节点将所述待写入数据写入各自的磁盘后发送的;
当接收到的所述接受消息的个数大于或等于W时,所述第一节点将所述待写入数据写入内存中,并向所述其它节点发送承诺消息,所述承诺消息用于指示所述其它节点将所述待写入数据写入各自的内存中。
若节点故障,如掉电,则内存中的数据会丢失,此时内存会在故障恢复后按照磁盘中的数据进行恢复。利用该特点,先将数据写入磁盘中,从而确保无论节点是否故障,都能将数据写入内存中。
更优选地,所述当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,包括:
所述第一节点接收并统计所述其它节点发送的所述确认消息,所述确认消息是所述其它节点将所述待写入数据写入各自的内存后向所述第一节点发送的;
当接收到的所述确认消息的个数大于或等于W时,所述第一节点向所述第二客户端发送所述响应消息。
如前所述,当内存中写入待写入数据的节点数量小于W时,分布式存储系统不会向第二客户端发送响应消息,第二客户端若在设定时间没有收到响应消息,则判定待写入数据未成功写入分布式存储系统,重新发送写入请求和写入待写入数据直到内存中写入待写入数据的节点数量大于或等于W时接收到分布式存储系统发送的响应消息,从而确保读取的数据成功写入分布式存储系统中。
第三方面,本发明实施例提供了一种处理数据的方法,所述方法包括:
接收第二客户端发送的写入请求,所述写入请求包括待写入数据;
根据所述写入请求将所述待写入数据写入分布式存储系统中的各个节点的内存中,所述分布式存储系统包括N个节点,N≥2且N为整数;
当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,W≤N且W为正整数,所述响应消息用于表示所述待写入数据成功写入所述分布式存储系统。
在第三方面一种可能的实现方式中,所述接收第二客户端发送的写入请求,包括:
所述分布式存储系统中的第一节点接收所述写入请求,所述第一节点为所述分布式存储系统中的一个设定节点或者所述分布式存储系统中的任意一个节点。
可选地,所述根据所述写入请求将所述待写入数据写入分布式存储系统中的各个节点的内存中,包括:
所述第一节点根据所述写入请求将所述待写入数据写入磁盘中,并向所述分布式存储系统中的其它节点发送所述写入请求,所述其它节点为所述分
布式存储系统中除所述第一节点之外的所有节点,所述写入请求用于指示所述其它节点将所述待写入数据写入各自的磁盘中;
所述第一节点接收并统计所述其它节点发送的接受消息,所述接受消息是所述其它节点将所述待写入数据写入各自的磁盘后发送的;
当接收到的所述接受消息的个数大于或等于W时,所述第一节点将所述待写入数据写入内存中,并向所述其它节点发送承诺消息,所述承诺消息用于指示所述其它节点将所述待写入数据写入各自的内存中。
优选地,所述当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,包括:
所述第一节点接收并统计所述其它节点发送的所述确认消息,所述确认消息是所述其它节点将所述待写入数据写入各自的内存后向所述第一节点发送的;
当接收到的所述确认消息的个数大于或等于W时,所述第一节点向所述第二客户端发送所述响应消息。
第四方面,本发明实施例提供了一种处理数据的装置,所述装置包括用于实现上述第一方面所述的方法的模块,例如读取请求发送模块、数据接收模块、数据选择模块。
第五方面,本发明实施例提供了一种处理数据的装置,所述装置用于实现上述第二方面所述的方法的模块,例如读取请求接收模块、数据发送模块。
第六方面,本发明实施例提供了一种处理数据的装置,所述装置用于实现上述第三方面所述的方法的模块,例如写入请求接收模块、数据写入模块、写入响应发送模块。
第七方面,本发明实施例提供了一种处理数据的系统,所述系统包括:
第一客户端,用于向分布式存储系统发送读取请求,所述分布式存储系统包括N个节点,N≥2且N为整数;
所述分布式存储系统,用于根据所述读取请求向所述第一客户端发送分布式存储系统中的R个节点的数据,所述R个节点的数据均写入过所述N
个节点中的至少W个节点的内存中,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N、W≤N且R、W均为整数;
所述第一客户端还用于,选择所述R个节点发送的数据中版本号最新的数据。
第八方面,本发明实施例提供了一种处理数据的系统,所述系统包括:
第二客户端,用于向分布式存储系统发送写入请求,所述分布式存储系统包括N个节点,N≥2且N为整数,所述写入请求包括待写入数据;
所述分布式存储系统,用于根据所述写入请求将所述待写入数据写入所述分布式存储系统中的各个节点的内存中;当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,W≤N且W为整数,所述响应消息用于表示所述待写入数据成功写入所述分布式存储系统。
本发明实施例提供的技术方案带来的有益效果是:
通过R个节点的数据均写入过N个节点中的至少W个节点的内存中,W为用于判断数据是否成功写入分布式存储系统的节点数量阈值,从而避免了由于读取的数据未成功写入分布式存储系统而造成客户端读取的数据错误的情况。而且通过获取分布式存储系统中的R个节点的数据,并选择R个节点的数据中版本号最新的数据,N-W<R≤N,N为分布式存储系统的节点数量,从而保证读取的数据是最新的,实现了数据的强一致性,为用户使用提供了方便。
为了更清楚地说明本发明实施例中的技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。
图1是本发明实施例提供的处理数据的方法的应用场景图;
图2是本发明实施例提供的客户端的结构示意图;
图3是本发明实施例提供的分布式存储系统中的节点的结构示意图;
图4是本发明实施例提供的一种处理数据的方法的流程图;
图5是本发明实施例提供的一种处理数据的装置的结构示意图;
图6是本发明实施例提供的一种处理数据的装置的结构示意图;
图7是本发明实施例提供的一种处理数据的系统的结构示意图。
为使本发明的目的、技术方案和优点更加清楚,下面将结合附图对本发明实施方式作进一步地详细描述。
下面先结合图1简单介绍一下本发明实施例提供的处理数据的方法的应用场景。如图1所示,分布式存储系统10包括多个存储数据的节点11,各个节点11之间相互连接。至少一个客户端20与分布式存储系统10连接,客户端20可以向分布式存储系统10中写入数据,也可以从分布式存储系统10中读取数据。
在实际应用中,分布式存储系统10和客户端20可以应用于银行交易系统、邮件系统、广告服务系统、业务查询系统等。
例如,在银行交易系统中将账户A中的资金转移到账户B中,账户A的用户通过客户端a登录账户A进行资金转移,分布式存储系统根据客户端a的写入请求,将更新后的资金数据分别写入账户A和账户B,并将账户A更新后的资金数据反馈到客户端a,账户A的用户通过客户端a可以获知资金成功转移,此为数据写入过程。此时账户B的用户通过客户端b登录账户B进行查询,分布式存储系统又根据客户端b的查询请求,将账户B之前由于转移资金而更新的资金数据反馈到客户端b,账户B的用户通过客户端b也可以获知资金成功转移,此为数据读取过程。
又如,在业务查询系统中,某个用户通过客户端c查询分布式存储系统b存储的业务信息,分布式存储系统b根据客户端c的读取请求,将相应的业务信息反馈给客户端c,此为数据读取过程。
具体地,分布式存储系统10可以为分布式数据库系统、集群配置数据库(Cluster Config Database,简称CCDB)、Zookeeper(分布式的、开放源码的分布式应用程序协调服务)等。客户端20可以为移动终端、平板电脑、笔记本电脑、台式电脑等。
在具体实现中,客户端20可以采用图2所示的结构,客户端20可以为手机、平板电脑、笔记本电脑、台式计算机等,如图2所示,客户端20包括存储器210、处理器220、发送器230和接收器240。本领域技术人员可以理解,图2中示出的客户端结构并不构成对客户端的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置。其中:
处理器220是客户端20的控制中心,利用各种接口和线路连接整个客户端20的各个部分,通过运行或执行存储在存储器210内的软件程序和/或模块,以及调用存储在存储器210内的数据,执行客户端20的各种功能和处理数据,从而对客户端20进行整体控制。可选的,处理器220可包括一个或多个处理核心。
存储器210可用于存储各种数据,例如各种配置参数,以及存储软件程序以及模块。处理器220通过运行存储在存储器210的软件程序以及模块,从而执行各种功能应用以及数据处理。存储器210可主要包括程序存储区和数据存储区,其中,程序存储区可存储操作系统211、读取请求发送模块212、数据接收模块213和数据选择模块214,还可存储写入请求发送模块215、写入响应接收模块216等;存储数据区可存储根据客户端20的使用所创建的数据,例如根据读取请求读取的数据等。此外,存储器210可以由任何类型的易失性或非易失性存储设备或者它们的组合实现,如静态随机存取存储器(Static Random Access Memory,简称SRAM),电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory,简称EEPROM),可擦除可编程只读存储器(Erasable Programmable Read Only Memory,简称EPROM),可编程只读存储器(Programmable Read-Only Memory,简称PROM),只读存储器(Read Only Memory,简称ROM),磁存储器,快闪存储器,磁盘或光盘。相应地,存储器210还可以包括存储器控制器,以提供处理器220对存储器210的访问。
发送器230,用于发送根据处理器220的指示,将读取请求告知分布式存储系统10,从而使分布式存储系统10根据读取请求读取数据,并将读取的数据发送给客户端20。可选地,发送器230还用于向分布式存储系统10发送写入请求。
接收器240,用于接收分布式存储系统10发送的根据读取请求读取的数据。可选地,接收器240还用于接收数据写入分布式存储系统10中至少W
个节点的内存后发送的响应消息。
分布式存储系统中的节点10可以采用图3所示的结构,如图3所示,分布式存储系统中的节点10包括存储器110、处理器120、发送器130和接收器140。图3中示出的分布式存储系统10中的节点结构并不构成对分布式存储系统中的节点的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置。
处理器120是分布式存储系统10中的节点的控制中心,利用各种接口和线路连接整个分布式存储系统10中的节点的各个部分,通过运行或执行存储在存储器110内的软件程序和/或模块,以及调用存储在存储器110内的数据,执行分布式存储系统10中的节点的各种功能和处理数据,从而对分布式存储系统10中的节点进行整体控制。可选的,处理器120可包括一个或多个处理核心。
存储器110可用于存储各种数据,例如各种配置参数,以及存储软件程序以及模块。处理器120通过运行存储在存储器110的软件程序以及模块,从而执行各种功能应用以及数据处理。存储器110可主要包括程序存储区和数据存储区,其中,程序存储区可存储操作系统111、读取请求接收模块112、数据发送模块113,还可存储写入请求接收模块114、数据写入模块115、写入响应发送模块116等;存储数据区可存储根据分布式存储系统10中的节点的使用所创建的数据,例如根据读取请求读取的数据等。此外,存储器110可以由任何类型的易失性或非易失性存储设备或者它们的组合实现,如静态随机存取存储器(Static Random Access Memory,简称SRAM),电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory,简称EEPROM),可擦除可编程只读存储器(Erasable Programmable Read Only Memory,简称EPROM),可编程只读存储器(Programmable Read-Only Memory,简称PROM),只读存储器(Read Only Memory,简称ROM),磁存储器,快闪存储器,磁盘或光盘。相应地,存储器110还可以包括存储器控制器,以提供处理器120对存储器110的访问。
接收器140,用于接收客户端20发送的读取请求。可选地,发送器130还用于接收客户端20发送的写入请求。
发送器130,用于根据处理器120的指示,向客户端20发送分布式存储
系统10中的R个节点的数据。可选地,发送器130还用于根据处理器120的指示,向客户端20发送数据写入分布式存储系统10中至少W个节点的内存后发送的响应消息。
图4示出了本发明实施例提供的一种处理数据的方法,该方法应用于图2所示的客户端和图3所示的分布式存储系统来举例说明,参见图4,该方法包括:
步骤301:第二客户端向分布式存储系统发送写入请求。
在本实施例中,分布式存储系统包括N个节点,N≥2且N为整数。一个节点可以为一个服务器,各个服务器之间相互独立。
写入请求包括待写入数据。可以理解地,写入请求还可以包括目的地址、源地址、以及待写入数据的标识。目的地址用于将写入请求准确发送到分布式存储系统;源地址用于分布式存储系统在接收到写入请求之后回应第二客户端,如发送响应消息;待写入数据的标识用于区分各个数据,分布式存储系统中的节点可以根据标识确定数据的存储位置。
具体地,该步骤301可以包括:
第二客户端向分布式存储系统中的第一节点发送写入请求。
在具体实现中,在配置客户端和分布式存储系统时,可以在客户端和分布式存储系统中存储配置文件,配置文件中包括分布式存储系统中的节点数量、判断数据是否成功写入分布式集群系统中的节点数量阈值、各个节点的地址等信息,第二客户端直接从配置文件中获取第一节点的地址,并按照该地址发送写入请求。
可选地,第一节点可以为分布式存储系统中的一个设定节点,也可以为分布式存储系统中的任意一个节点。若第一节点为分布式存储系统中的一个设定节点,则第一节点可以是人为设置的,也可以是分布式存储系统中的所有节点协商决定的(第一设定节点变动时会通知各个节点和客户端),本发明对此不作限制。
若第一节点为分布式存储系统中的任意一个节点,则可以有效避免由于设定节点故障而无法写入待写入数据的问题。而且分布式存储系统中的多个节点可以同时接收写入请求进行数据写入,与分布式存储系统中只有一个设定节点可以接收写入请求进行数据写入相比,数据写入效率提升至N倍,N
为分布式存储系统的节点数量。
步骤302:分布式存储系统根据写入请求将待写入数据写入分布式存储系统中的各个节点的内存中。
需要说明的是,由于客户端是从分布式存储系统中的至少一个节点的内存中读取数据,因此待写入数据需要写入分布式存储系统中的各个节点的内存中。
优选地,该步骤302可以包括:
第一节点根据写入请求将待写入数据写入磁盘中,并向分布式存储系统中的其它节点发送写入请求,其它节点为分布式存储系统中除第一节点之外的所有节点;
其它节点根据写入请求将待写入数据写入各自的磁盘中,并向第一节点发送接受消息;
第一节点接收并统计其它节点发送的接受消息;
当接收到的接受消息的个数大于或等于W时,第一节点将待写入数据写入内存中,并向其它节点发送承诺消息;
其它节点根据承诺消息将待写入数据写入各自的内存中。
在本实施例中,W≤N且W为正整数,N为分布式存储系统的节点数量,W为判断数据是否成功写入分布式存储系统的节点数量阈值。
需要说明的是,若节点故障,如掉电,则内存中的数据会丢失,此时内存会在故障恢复后按照磁盘中的数据进行恢复。利用该特点,先将数据写入磁盘中,从而确保无论节点是否故障,都能将数据写入内存中。
步骤303:当内存中写入待写入数据的节点数量大于或等于W时,分布式存储系统向第二客户端发送响应消息。
在本实施例中,响应消息用于表示待写入数据成功写入分布式存储系统。
可以理解地,当内存中写入待写入数据的节点数量小于W时,分布式存储系统不会向第二客户端发送响应消息,第二客户端若在设定时间没有收到响应消息,则判定待写入数据未成功写入分布式存储系统,重新开始执行步骤301,直到内存中写入待写入数据的节点数量大于或等于W时接收到分布式存储系统发送的响应消息,从而确保读取的数据成功写入分布式存储系统中。
具体地,该步骤303可以包括:
其它节点将待写入数据写入各自的内存后向第一节点发送确认消息;
第一节点接收并统计其它节点发送的确认消息;
当接收到的确认消息的个数大于或等于W时,第一节点向第二客户端发送响应消息。
需要说明的是,步骤301-步骤303实现了将数据写入分布式存储系统中。
步骤304:第一客户端向分布式存储系统发送读取请求。
在实际应用中,第一客户端和第二客户端可以为不同的客户端。
读取请求可以包括目的地址、源地址、以及待读取数据的标识。目的地址用于将读取请求准确发送到分布式存储系统;源地址用于分布式存储系统在接收到读取请求之后回应第一客户端,如待读取数据;待读取数据的标识用于区分各个数据,分布式存储系统中的节点可以根据标识确定数据的存储位置。
具体地,该步骤304可以包括:
第一客户端向分布式存储系统中的R个节点发送读取请求。
在本实施例中,N-W<R≤N且R为整数,N为分布式存储系统的节点数量,W为判断数据是否成功写入分布式存储系统的节点数量阈值。
需要说明的是,分布式存储系统中接收读取请求的R个节点是任意的,因此可以同时有多组R个节点分别接收不同的读取请求进行数据读取,与分布式存储系统中只有一个设定节点接收读取请求进行数据读取相比,每秒查询率(Query Per Second,简称QPS)提升到1+(N-1)/R倍,N为分布式存储系统的节点数量。
在具体应用中,在配置客户端和分布式存储系统时,可以在客户端和分布式存储系统中存储配置文件,配置文件中包括分布式存储系统中的节点数量、判断数据是否成功写入分布式集群系统中的节点数量阈值、各个节点的地址等信息,第一客户端直接从配置文件中获取R个节点的地址,并按照该地址发送写入请求。
步骤305:分布式存储系统根据读取请求向第一客户端发送分布式存储系统中的R个节点的数据。
在本实施例中,R个节点的数据均写入过N个节点中的至少W个节点的内存中,N-W<R≤N,N为分布式存储系统的节点数量,W为判断数据是否成功写入分布式存储系统的节点数量。
其中,R个节点的数据均写入过N个节点中的至少W个节点的内存中,包括R个节点中的至少一个数据写入N个节点中的至少W个节点的内存中、以及R个节点中的其它数据曾经写入过N个节点中的至少W个节点的内存中。比如,R个节点的数据包括数据A和数据B,数据B是数据A更新后的数据,在数据B写入分布式存储系统之前,数据A写入N个节点中的至少W个节点的内存中;在数据B写入分布式存储系统之后,数据B写入N个节点中的至少W个节点的内存中。
具体地,该步骤305可以包括:
分布式存储系统中的R个节点根据读取请求向第一客户端发送各自的数据。
步骤306:第一客户端选择R个节点的数据中版本号最新的数据。
在本实施例中,版本号是在数据写入过程中用于指示数据新旧的编号。例如,版本号为V2的数据是版本号为V1的数据更新后的数据。即若第一客户端读取的数据包括版本号为V2的数据和版本号为V1的数据,则第一客户端选择版本号为V2的数据。
需要说明的是,步骤304-步骤306实现了从分布式存储系统中读取数据。
上述步骤的执行可以通过客户端和分布式存储系统根据前述软件模块执行。例如,步骤301由客户端根据图2中的读取请求发送模块212和分布式存储系统根据图3中的读取请求接收模块112执行,步骤302由分布式存储系统根据图3中的数据发送模块113和客户端根据图2中的数据接收模块213执行,步骤303由客户端根据图2中的数据选择模块214执行,步骤304由客户端根据图2中的写入请求发送模块215和分布式存储系统根据图3中的写入请求接收模块114执行,步骤305由分布式存储系统根据图3中的数据写入模块115执行,步骤306由分布式存储系统根据图3中的写入响应发送模块116和客户端根据图2中的写入响应接收模块216执行。
参见图5,其示出了本发明实施例提供的一种处理数据的装置,该装置可以通过软件、硬件或者两者的结合实现成为客户端的全部或者一部分。该装置包括:读取请求发送模块401、数据接收模块402和数据选择模块403。
其中,读取请求发送模块401用于向分布式存储系统发送读取请求,分布式存储系统包括N个节点,N≥2且N为整数。数据接收模块402用于接
收N个节点中的R个节点根据读取请求发送的数据,R个节点发送的数据均写入过N个节点中的至少W个节点的内存中,W为判断数据是否成功写入分布式存储系统的节点数量阈值,N-W<R≤N且R、W均为整数。数据选择模块403用于选择R个节点发送的数据中版本号最新的数据。
可选地,该装置还可以包括:写入请求发送模块404和写入响应接收模块405。
其中,写入请求发送模块404用于向分布式存储系统发送写入请求,写入请求包括待写入数据。写入响应接收模块405用于接收响应消息,响应消息用于表示待写入数据成功写入分布式存储系统。
参见图6,其示出了本发明实施例提供的一种处理数据的装置,该装置可以通过软件、硬件或者两者的结合实现成为分布式存储系统的全部或者一部分。该装置包括:读取请求接收模块501和数据发送模块502。
其中,读取请求接收模块501用于接收第一客户端发送的读取请求。数据发送模块502用于根据读取请求向第一客户端发送分布式存储系统中的R个节点的数据,分布式存储系统包括N个节点,R个节点的数据均写入过N个节点中的至少W个节点的内存中,W为判断数据是否成功写入分布式存储系统的节点数量阈值,N-W<R≤N、W≤N、N≥2且N、R、W均为整数。
可选地,该装置还可以包括:写入请求接收模块503、数据写入模块504和写入响应发送模块505。
其中,写入请求接收模块503用于接收第二客户端发送的写入请求,写入请求包括待写入数据。数据写入模块504用于根据写入请求将待写入数据写入分布式存储系统中的各个节点的内存中。写入响应发送模块505用于当内存中写入待写入数据的节点数量大于或等于W时,向第二客户端发送响应消息,响应消息用于表示待写入数据成功写入分布式存储系统。
优选地,写入请求接收模块503可以用于通过分布式存储系统中的第一节点接收写入请求,第一节点为分布式存储系统中的一个设定节点或者分布式存储系统中的任意一个节点。
更优选地,数据写入模块504可以包括:磁盘写入单元504a、接受消息统计单元504b和内存写入单元504c。
其中,磁盘写入单元504a用于根据写入请求将待写入数据写入第一节
点的磁盘中,并向分布式存储系统中的其它节点发送写入请求,其它节点为分布式存储系统中除第一节点之外的所有节点,写入请求用于指示其它节点将待写入数据写入各自的磁盘中。接受消息统计单元504b用于接收并统计其它节点发送的接受消息,接受消息是其它节点将待写入数据写入各自的磁盘后发送的。内存写入单元504c用于当接收到的接受消息的个数大于或等于W时,将待写入数据写入第一节点的内存中,并向其它节点发送承诺消息,承诺消息用于指示其它节点将待写入数据写入各自的内存中。
进一步地,写入响应发送模块505可以包括:确认消息统计单元505a和写入响应发送单元505b。
其中,确认消息统计单元505a用于接收并统计其它节点发送的确认消息,确认消息是其它节点将待写入数据写入各自的内存后向第一节点发送的。写入响应发送单元505b用于当接收到的确认消息的个数大于或等于W时,向第二客户端发送响应消息。
参见图7,其示出了本发明实施例提供的一种处理数据的系统,该系统可以包括第一客户端601和分布式存储系统602.。
其中,第一客户端601用于向分布式存储系统602发送读取请求,分布式存储系统602包括N个节点,N≥2且N为整数。分布式存储系统602用于根据读取请求向第一客户端601发送分布式存储系统中的R个节点的数据,R个节点的数据均写入过N个节点中的至少W个节点的内存中,W为判断数据是否成功写入分布式存储系统的节点数量阈值,N-W<R≤N、W≤N且R、W均为整数。第一客户端603还用于选择R个节点发送的数据中版本号最新的数据。
可选地,该系统还可以包括第二客户端603。
其中,第二客户端603用于向分布式存储系统602发送写入请求,写入请求包括待写入数据。相应地,分布式存储系统602用于根据写入请求将待写入数据写入分布式存储系统中的各个节点的内存中;当内存中写入待写入数据的节点数量大于或等于W时,向第二客户端发送响应消息,响应消息用于表示待写入数据成功写入分布式存储系统。
需要说明的是:上述实施例提供的处理数据的装置在处理数据时,仅以
上述各功能模块的划分进行举例说明,实际应用中,可以根据需要而将上述功能分配由不同的功能模块完成,即将装置的内部结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。另外,上述实施例提供的处理数据的装置与处理数据的方法实施例属于同一构思,其具体实现过程详见方法实施例,这里不再赘述。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成,也可以通过程序来指令相关的硬件完成,所述的程序可以存储于一种计算机可读存储介质中,上述提到的存储介质可以是只读存储器,磁盘或光盘等。
以上所述仅为本发明的较佳实施例,并不用以限制本发明,凡在本发明的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本发明的保护范围之内。
Claims (22)
- 一种处理数据的方法,其特征在于,所述方法包括:向分布式存储系统发送读取请求,所述分布式存储系统包括N个节点,N≥2且N为整数;接收所述N个节点中的R个节点根据所述读取请求发送的数据,所述R个节点发送的数据均写入过所述N个节点中的至少W个节点的内存中,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N且R、W均为整数;选择所述R个节点发送的数据中版本号最新的数据。
- 一种处理数据的方法,其特征在于,所述方法包括:接收第一客户端发送的读取请求;根据所述读取请求向所述第一客户端发送分布式存储系统中的R个节点的数据,所述分布式存储系统包括N个节点,所述R个节点的数据均写入过所述N个节点中的至少W个节点的内存中,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N、W≤N、N≥2且N、R、W均为整数。
- 根据权利要求2所述的方法,其特征在于,所述方法还包括:接收第二客户端发送的写入请求,所述写入请求包括待写入数据;根据所述写入请求将所述待写入数据写入所述分布式存储系统中的各个节点的内存中;当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,所述响应消息用于表示所述待写入数据成功写入所述分布式存储系统。
- 根据权利要求3所述的方法,其特征在于,所述接收第二客户端发送的写入请求,包括:所述分布式存储系统中的第一节点接收所述写入请求,所述第一节点为所述分布式存储系统中的一个设定节点或者所述分布式存储系统中的任意一个节点。
- 根据权利要求4所述的方法,其特征在于,所述根据所述写入请求将所述待写入数据写入所述分布式存储系统中的各个节点的内存中,包括:所述第一节点根据所述写入请求将所述待写入数据写入磁盘中,并向所述分布式存储系统中的其它节点发送所述写入请求,所述其它节点为所述分布式存储系统中除所述第一节点之外的所有节点,所述写入请求用于指示所述其它节点将所述待写入数据写入各自的磁盘中;所述第一节点接收并统计所述其它节点发送的接受消息,所述接受消息是所述其它节点将所述待写入数据写入各自的磁盘后发送的;当接收到的所述接受消息的个数大于或等于W时,所述第一节点将所述待写入数据写入内存中,并向所述其它节点发送承诺消息,所述承诺消息用于指示所述其它节点将所述待写入数据写入各自的内存中。
- 根据权利要求5所述的方法,其特征在于,所述当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,包括:所述第一节点接收并统计所述其它节点发送的所述确认消息,所述确认消息是所述其它节点将所述待写入数据写入各自的内存后向所述第一节点发送的;当接收到的所述确认消息的个数大于或等于W时,所述第一节点向所述第二客户端发送所述响应消息。
- 一种处理数据的方法,其特征在于,所述方法包括:接收第二客户端发送的写入请求,所述写入请求包括待写入数据;根据所述写入请求将所述待写入数据写入分布式存储系统中的各个节点的内存中,所述分布式存储系统包括N个节点,N≥2且N为整数;当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,W≤N且W为正整数,所述响应消息用于表示所述待写入数据成功写入所述分布式存储系统。
- 根据权利要求7所述的方法,其特征在于,所述接收第二客户端发送的写入请求,包括:所述分布式存储系统中的第一节点接收所述写入请求,所述第一节点为所述分布式存储系统中的一个设定节点或者所述分布式存储系统中的任意一个节点。
- 根据权利要求8所述的方法,其特征在于,所述根据所述写入请求将所述待写入数据写入分布式存储系统中的各个节点的内存中,包括:所述第一节点根据所述写入请求将所述待写入数据写入磁盘中,并向所述分布式存储系统中的其它节点发送所述写入请求,所述其它节点为所述分布式存储系统中除所述第一节点之外的所有节点,所述写入请求用于指示所述其它节点将所述待写入数据写入各自的磁盘中;所述第一节点接收并统计所述其它节点发送的接受消息,所述接受消息是所述其它节点将所述待写入数据写入各自的磁盘后发送的;当接收到的所述接受消息的个数大于或等于W时,所述第一节点将所述待写入数据写入内存中,并向所述其它节点发送承诺消息,所述承诺消息用于指示所述其它节点将所述待写入数据写入各自的内存中。
- 根据权利要求9所述的方法,其特征在于,所述当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,包括:所述第一节点接收并统计所述其它节点发送的所述确认消息,所述确认消息是所述其它节点将所述待写入数据写入各自的内存后向所述第一节点发送的;当接收到的所述确认消息的个数大于或等于W时,所述第一节点向所述第二客户端发送所述响应消息。
- 一种处理数据的装置,其特征在于,所述装置包括:读取请求发送模块,用于向分布式存储系统发送读取请求,所述分布式存储系统包括N个节点,N≥2且N为整数;数据接收模块,用于接收所述N个节点中的R个节点根据所述读取请求发送的数据,所述R个节点发送的数据均写入过所述N个节点中的至少W个节点的内存中,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N且R、W均为整数;数据选择模块,用于选择所述R个节点发送的数据中版本号最新的数据。
- 一种处理数据的装置,其特征在于,所述装置包括:读取请求接收模块,用于接收第一客户端发送的读取请求;数据发送模块,用于根据所述读取请求向所述第一客户端发送分布式存储系统中的R个节点的数据,所述分布式存储系统包括N个节点,所述R个节点的数据均写入过所述N个节点中的至少W个节点的内存中,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N、W≤N、N≥2且N、R、W均为整数。
- 根据权利要求12所述的装置,其特征在于,所述装置还包括:写入请求接收模块,用于接收第二客户端发送的写入请求,所述写入请求包括待写入数据;数据写入模块,用于根据所述写入请求将所述待写入数据写入所述分布式存储系统中的各个节点的内存中;写入响应发送模块,用于当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,所述响应消息用于表示所述待写入数据成功写入所述分布式存储系统。
- 根据权利要求13所述的装置,其特征在于,所述写入请求接收模块用于,通过所述分布式存储系统中的第一节点接收所述写入请求,所述第一节点为所述分布式存储系统中的一个设定节点或者所述分布式存储系统中的任意一个节点。
- 根据权利要求14所述的装置,其特征在于,所述数据写入模块包括:磁盘写入单元,用于根据所述写入请求将所述待写入数据写入所述第一节点的磁盘中,并向所述分布式存储系统中的其它节点发送所述写入请求,所述其它节点为所述分布式存储系统中除所述第一节点之外的所有节点,所述写入请求用于指示所述其它节点将所述待写入数据写入各自的磁盘中;接受消息统计单元,用于接收并统计所述其它节点发送的接受消息,所述接受消息是所述其它节点将所述待写入数据写入各自的磁盘后发送的;内存写入单元,用于当接收到的所述接受消息的个数大于或等于W时,将所述待写入数据写入所述第一节点的内存中,并向所述其它节点发送承诺消息,所述承诺消息用于指示所述其它节点将所述待写入数据写入各自的内存中。
- 根据权利要求15所述的装置,其特征在于,所述写入响应发送模 块包括:确认消息统计单元,用于接收并统计所述其它节点发送的所述确认消息,所述确认消息是所述其它节点将所述待写入数据写入各自的内存后向所述第一节点发送的;写入响应发送单元,用于当接收到的所述确认消息的个数大于或等于W时,向所述第二客户端发送所述响应消息。
- 一种处理数据的装置,其特征在于,所述装置包括:写入请求接收模块,用于接收第二客户端发送的写入请求,所述写入请求包括待写入数据;数据写入模块,用于根据所述写入请求将所述待写入数据写入分布式存储系统中的各个节点的内存中,所述分布式存储系统包括N个节点,N≥2且N为整数;写入响应发送模块,用于当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,W≤N且W为正整数,所述响应消息用于表示所述待写入数据成功写入所述分布式存储系统。
- 根据权利要求17所述的装置,其特征在于,所述写入请求接收模块用于,通过所述分布式存储系统中的第一节点接收所述写入请求,所述第一节点为所述分布式存储系统中的一个设定节点或者所述分布式存储系统中的任意一个节点。
- 根据权利要求18所述的装置,其特征在于,所述数据写入模块包括:磁盘写入单元,用于根据所述写入请求将所述待写入数据写入所述第一节点的磁盘中,并向所述分布式存储系统中的其它节点发送所述写入请求,所述其它节点为所述分布式存储系统中除所述第一节点之外的所有节点,所述写入请求用于指示所述其它节点将所述待写入数据写入各自的磁盘中;接受消息统计单元,用于接收并统计所述其它节点发送的接受消息,所述接受消息是所述其它节点将所述待写入数据写入各自的磁盘后发送的;内存写入单元,用于当接收到的所述接受消息的个数大于或等于W时, 将所述待写入数据写入内存中,并向所述其它节点发送承诺消息,所述承诺消息用于指示所述其它节点将所述待写入数据写入各自的内存中。
- 根据权利要求19所述的装置,其特征在于,所述写入响应发送模块包括:确认消息统计单元,用于接收并统计所述其它节点发送的所述确认消息,所述确认消息是所述其它节点将所述待写入数据写入各自的内存后向所述第一节点发送的;写入响应发送单元,用于当接收到的所述确认消息的个数大于或等于W时,向所述第二客户端发送所述响应消息。
- 一种处理数据的系统,其特征在于,所述系统包括:第一客户端,用于向分布式存储系统发送读取请求,所述分布式存储系统包括N个节点,N≥2且N为整数;所述分布式存储系统,用于根据所述读取请求向所述第一客户端发送分布式存储系统中的R个节点的数据,所述R个节点的数据均写入过所述N个节点中的至少W个节点的内存中,W为判断数据是否成功写入所述分布式存储系统的节点数量阈值,N-W<R≤N、W≤N且R、W均为整数;所述第一客户端还用于,选择所述R个节点发送的数据中版本号最新的数据。
- 一种处理数据的系统,其特征在于,所述系统包括:第二客户端,用于向分布式存储系统发送写入请求,所述分布式存储系统包括N个节点,N≥2且N为整数,所述写入请求包括待写入数据;所述分布式存储系统,用于根据所述写入请求将所述待写入数据写入所述分布式存储系统中的各个节点的内存中;当内存中写入所述待写入数据的节点数量大于或等于W时,向所述第二客户端发送响应消息,W≤N且W为整数,所述响应消息用于表示所述待写入数据成功写入所述分布式存储系统。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510818833.3 | 2015-11-23 | ||
CN201510818833.3A CN105468302B (zh) | 2015-11-23 | 2015-11-23 | 一种处理数据的方法、装置及系统 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017088572A1 true WO2017088572A1 (zh) | 2017-06-01 |
Family
ID=55606055
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2016/099590 WO2017088572A1 (zh) | 2015-11-23 | 2016-09-21 | 一种处理数据的方法、装置及系统 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN105468302B (zh) |
WO (1) | WO2017088572A1 (zh) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110083306A (zh) * | 2019-03-14 | 2019-08-02 | 南京时沃信息科技有限公司 | 一种分布式对象存储系统及存储方法 |
CN111936960A (zh) * | 2018-12-25 | 2020-11-13 | 华为技术有限公司 | 分布式存储系统中数据存储方法、装置及计算机程序产品 |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105468302B (zh) * | 2015-11-23 | 2018-10-30 | 华为技术有限公司 | 一种处理数据的方法、装置及系统 |
CN106888245B (zh) * | 2016-06-07 | 2020-04-24 | 阿里巴巴集团控股有限公司 | 一种数据处理方法、装置及系统 |
CN108572793B (zh) * | 2017-10-18 | 2021-09-10 | 北京金山云网络技术有限公司 | 数据写入和数据恢复方法、装置、电子设备及存储介质 |
CN108959549A (zh) * | 2018-06-29 | 2018-12-07 | 北京奇虎科技有限公司 | 数据写入方法、装置、计算设备及计算机存储介质 |
CN110825309B (zh) * | 2018-08-08 | 2021-06-29 | 华为技术有限公司 | 数据读取方法、装置及系统、分布式系统 |
CN109691065B (zh) * | 2018-08-23 | 2021-11-09 | 袁振南 | 分布式存储系统及其数据读写方法、存储终端及存储介质 |
CN109407981A (zh) * | 2018-09-28 | 2019-03-01 | 深圳市茁壮网络股份有限公司 | 一种数据处理方法及装置 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102014153A (zh) * | 2010-10-19 | 2011-04-13 | 浪潮(北京)电子信息产业有限公司 | 一种数据存储系统及其数据存储方法 |
CN103064635A (zh) * | 2012-12-19 | 2013-04-24 | 华为技术有限公司 | 分布式存储方法和分布式存储装置 |
CN105468302A (zh) * | 2015-11-23 | 2016-04-06 | 华为技术有限公司 | 一种处理数据的方法、装置及系统 |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9602472B2 (en) * | 2012-06-15 | 2017-03-21 | Alcatel Lucent | Methods and systems for privacy protection of network end users including profile slicing |
-
2015
- 2015-11-23 CN CN201510818833.3A patent/CN105468302B/zh active Active
-
2016
- 2016-09-21 WO PCT/CN2016/099590 patent/WO2017088572A1/zh active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102014153A (zh) * | 2010-10-19 | 2011-04-13 | 浪潮(北京)电子信息产业有限公司 | 一种数据存储系统及其数据存储方法 |
CN103064635A (zh) * | 2012-12-19 | 2013-04-24 | 华为技术有限公司 | 分布式存储方法和分布式存储装置 |
CN105468302A (zh) * | 2015-11-23 | 2016-04-06 | 华为技术有限公司 | 一种处理数据的方法、装置及系统 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111936960A (zh) * | 2018-12-25 | 2020-11-13 | 华为技术有限公司 | 分布式存储系统中数据存储方法、装置及计算机程序产品 |
EP3889752A4 (en) * | 2018-12-25 | 2021-12-01 | Huawei Technologies Co., Ltd. | DATA STORAGE METHOD AND DEVICE IN A DISTRIBUTED STORAGE SYSTEM AND COMPUTER PROGRAM PRODUCT |
CN111936960B (zh) * | 2018-12-25 | 2022-08-19 | 华为云计算技术有限公司 | 分布式存储系统中数据存储方法、装置及计算机程序产品 |
US11775194B2 (en) | 2018-12-25 | 2023-10-03 | Huawei Cloud Computing Technolgoies Co., Ltd. | Data storage method and apparatus in distributed storage system, and computer program product |
CN110083306A (zh) * | 2019-03-14 | 2019-08-02 | 南京时沃信息科技有限公司 | 一种分布式对象存储系统及存储方法 |
Also Published As
Publication number | Publication date |
---|---|
CN105468302B (zh) | 2018-10-30 |
CN105468302A (zh) | 2016-04-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2017088572A1 (zh) | 一种处理数据的方法、装置及系统 | |
KR101862718B1 (ko) | 블록-기반 저장을 위한 데이터 볼륨 지속성 상태 감소 | |
US10592464B2 (en) | Methods for enabling direct memory access (DMA) capable devices for remote DMA (RDMA) usage and devices thereof | |
US10064025B2 (en) | Offline peer-assisted notification delivery | |
US7814479B2 (en) | Simultaneous download to multiple targets | |
JP4696089B2 (ja) | 分散ストレージシステム | |
US8135987B2 (en) | Collection ordering for replicated state machines | |
US9940042B2 (en) | Distributed storage system, and data-access method therefor | |
US9614646B2 (en) | Method and system for robust message retransmission | |
US9307011B2 (en) | Synchronous mirroring of NVLog to multiple destinations (architecture level) | |
CN106572153A (zh) | 集群的数据存储方法及装置 | |
WO2018121456A1 (zh) | 一种数据存储方法、服务器以及存储系统 | |
CN109714409B (zh) | 一种消息的管理方法和系统 | |
CN111031135B (zh) | 消息传送方法、装置及电子设备 | |
US20230098190A1 (en) | Data processing method, apparatus, device and medium based on distributed storage | |
WO2016107443A1 (zh) | 一种快照处理方法及相关设备 | |
US20160092324A1 (en) | Fast single-master failover | |
US20190227888A1 (en) | Handling node failure in multi-node data storage systems | |
US10162720B2 (en) | Copy-on-read process in disaster recovery | |
US11030220B2 (en) | Global table management operations for multi-region replicated tables | |
CN107453845B (zh) | 应答确认方法及设备 | |
CN110442601B (zh) | 一种Openstack镜像数据并行加速的方法和装置 | |
CN112148206A (zh) | 一种数据读写方法、装置、电子设备及介质 | |
US20210160320A1 (en) | Data transmission and network interface controller | |
JP2012089049A (ja) | 計算機システム及びサーバ |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16867799 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 16867799 Country of ref document: EP Kind code of ref document: A1 |