CN111198662B - Data storage method, device and computer readable storage medium - Google Patents

Data storage method, device and computer readable storage medium Download PDF

Info

Publication number
CN111198662B
CN111198662B CN202010004352.XA CN202010004352A CN111198662B CN 111198662 B CN111198662 B CN 111198662B CN 202010004352 A CN202010004352 A CN 202010004352A CN 111198662 B CN111198662 B CN 111198662B
Authority
CN
China
Prior art keywords
data
node
nodes
slave
master
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010004352.XA
Other languages
Chinese (zh)
Other versions
CN111198662A (en
Inventor
曹智颖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Cloud Computing Changsha Co Ltd
Original Assignee
Tencent Cloud Computing Changsha Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Cloud Computing Changsha Co Ltd filed Critical Tencent Cloud Computing Changsha Co Ltd
Priority to CN202010004352.XA priority Critical patent/CN111198662B/en
Publication of CN111198662A publication Critical patent/CN111198662A/en
Application granted granted Critical
Publication of CN111198662B publication Critical patent/CN111198662B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The embodiment of the invention discloses a data storage method, a data storage device and a computer readable storage medium; after receiving a data writing request sent by a terminal by a slave node, the data writing request carries writing data, the slave node acquires a data number of the writing data from a master node according to the data writing request, the slave node sends the writing data and the data number as synchronous data to other nodes of a distributed data system, receives feedback information sent by the other nodes, determines the first node number of the other nodes successfully receiving the synchronous data according to the feedback information, and stores the writing data by the slave node based on the first node number; the scheme can greatly improve the efficiency of data storage.

Description

Data storage method, device and computer readable storage medium
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data storage method, apparatus, and computer readable storage medium.
Background
In recent years, with the rapid development of big data technology, a single database or service node cannot support the growth of business and data, so that distributed data systems begin to be widely applied, and most of existing distributed data systems adopt a single-main-class data processing mode, wherein the single-main-class data processing mode is that a main node exists in the distributed data system, and the main node is responsible for writing data and synchronizing the data to other nodes.
In the research and practice process of the prior art, the inventor finds that only the master node is allowed to process data writing in a single master type data processing mode, and when more data needs to be processed, the master node can be caused to generate data backlog and the like, so that the efficiency of data storage can be greatly reduced.
Disclosure of Invention
The embodiment of the invention provides a data storage method, a data storage device and a computer readable storage medium. The efficiency of data storage can be improved.
A data storage method, comprising:
the slave node receives a data writing request sent by a terminal, wherein the data writing request carries writing data and is used for writing the writing data into nodes of the distributed data system;
the slave node acquires the data number of the written data from the master node according to the data writing request, wherein the data number is the sequential identification of the written data in the distributed data system;
the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system and receives feedback information sent by the other nodes;
The slave node determines the first node quantity of other nodes which successfully receive the synchronous data according to the feedback information;
the slave node stores the write data based on the first node number.
Accordingly, an embodiment of the present invention provides a data storage device, including:
the receiving unit is used for receiving a data writing request sent by the terminal from the node, wherein the data writing request carries writing data, and the data writing request is used for writing the writing data into the nodes of the distributed data system;
the obtaining unit is used for obtaining the data number of the writing data from the master node according to the data writing request, wherein the data number is the sequence identifier of the writing data in the distributed data system;
the sending unit is used for sending the writing data and the data number as synchronous data to other nodes of the distributed data system by the slave node and receiving feedback information sent by the other nodes;
the determining unit is used for determining the first node number of other nodes which successfully receive the synchronous data according to the feedback information by the slave node;
And the storage unit is used for storing the write data by the slave node based on the first node number.
Optionally, in some embodiments, the storage unit may be specifically configured to, when the number of the first nodes exceeds half of the number of all nodes, write the synchronization data to a local database of the slave node, and send a data saving request to the other node to save the synchronization data to the local database of the other node; when the number of the first nodes does not exceed half of the number of all nodes, the slave node discards the write data and sends a data discarding request to the other nodes to discard the synchronous data received by the other nodes.
Optionally, in some embodiments, the acquiring unit may be specifically configured to generate, according to the data writing request, a data synchronization request by the slave node; the slave node sends the data synchronization request to the master node; when the master node returns the data number, the slave node receives the data number; and when the master node does not return the data number, the slave node returns prompt information of data writing failure to the terminal.
Optionally, in some embodiments, the obtaining unit may be specifically configured to send a test request to the master node by using the slave node, where the test request is used to test a state of the master node; the slave node receives feedback information of the test request and determines whether the master node is in a normal state according to the feedback information; when the feedback information is that the master node is in an abnormal state, the slave node determines that the master node is in an abnormal state, and the slave node triggers a new master node determination flow; the slave node sending the data synchronization request to the master node, comprising: and when the master node is in a normal state, the slave node sends the synchronous data request to the master node.
Optionally, in some embodiments, the obtaining unit may be specifically configured to send, to the other slave nodes, a voting request for selecting a new master node; the slave node receives the voting results of the other slave nodes aiming at the voting request reply, wherein the voting results comprise that the slave node which agrees to issue the voting request is a new master node and the slave node which does not agree to issue the voting request is a new master node; when the voting result is that the number of slave nodes which agree to issue a voting request is half of the total number of the slave nodes, the slave nodes are selected as the new master nodes.
Optionally, in some embodiments, the obtaining unit may be specifically configured to receive, by the other slave node, the voting request sent by the slave node; the other slave nodes compare the first data number with the second data number with the largest current value of the other slave nodes; when the second data number does not exceed the first data number, the other slave nodes send voting results which agree to send voting requests to the slave nodes and are taken as new master nodes by the slave nodes; and when the second data number exceeds the first data number, the other slave nodes send voting results of the slave nodes which do not agree to send voting requests to the slave nodes as new master nodes.
Optionally, in some embodiments, the obtaining unit may be specifically configured to, when the second data number does not exceed the first data number and the other slave node receives at least two voting requests within a preset time interval, send, to the slave node that sends the voting request for the first time, a voting result that the slave node that sends the voting request is a new master node.
Optionally, in some embodiments, the data storage device may further include a first failure recovery unit, where the first failure recovery unit is applied to a slave node, and may specifically be used for the slave node to send a data detection request to the master node, where the data detection request is used to obtain a largest target data number among data numbers of the master node; the slave node receives the target data number sent by the master node; when the target data number exceeds the largest current data number in the data numbers of the slave nodes, the slave nodes determine that own data are missing, and acquire the data missing from the master nodes.
Optionally, in some embodiments, the first failure unit may be specifically configured to determine that there is a loss of self data by the slave node; the slave node sends a first data acquisition request to the master node, wherein the first data acquisition request carries the current data number of the slave node; and the slave node receives the data missing from the slave node, which is sent by the master node according to the current data number.
Optionally, in some embodiments, the data storage device may further include a second fault recovery unit, applied to a master node, and specifically may be used for the master node to receive the data acquisition request; the main node compares the current data number in the data acquisition request with the target number; the master node determines the data missing from the slave node according to the comparison result; the master node sends the data missing from the slave node to the slave node.
Optionally, in some embodiments, the second fault recovery unit may be specifically configured to, when the master node receives a data write request, send the synchronization data to the other node by the master node; when other master nodes in other nodes receive the synchronous data, the master node determines the identity of the other master nodes and acquires the identity of the other master nodes.
Optionally, in some embodiments, the second failure recovery unit may be specifically configured to, when the master node receives the synchronization data sent by the other master node, determine identities of the other master nodes, and obtain identities of the other master nodes.
Optionally, in some embodiments, the second failure recovery unit may be specifically configured to send, by the master node, the data comparison request to the other master nodes according to the identity of the other master nodes, where the data comparison request is used to obtain a first target data number with a maximum data number of the other master nodes; the master node receives a first target data number sent by other master nodes; when the first target data number exceeds the second target data number which is the largest in the data numbers of the master nodes, the master node becomes a slave node, and the data missing from the master node is acquired from the other master nodes.
Optionally, in some embodiments, the second fault recovery unit may be specifically configured to determine that the master node has a data miss; the master node sends a second data acquisition request to the other master nodes, wherein the second data acquisition request carries a second target data number of the master node; and the master node receives the data missing from the master node, which is sent by the other master nodes according to the second target data number.
Optionally, in some embodiments, the data storage device may further include a writing unit, where the writing unit is specifically applied to data writing of the master node, and the writing unit may be specifically used for the master node to receive a data writing request sent by the terminal; the master node generates a data number of the written data according to the data writing request; the master node sends the written data and the data number as synchronous data to slave nodes of the distributed data system, and receives feedback information sent by the slave nodes; the master node determines the number of second nodes of slave nodes which successfully receive the synchronous data according to the feedback information; when the number of the second nodes exceeds half of the total number of nodes of the distributed data system, the master node stores the writing data to a local database of the master node and sends the data saving request to the slave node so as to save the synchronous data to the local database of the slave node; when the number of the second nodes does not exceed half of the total number of nodes of the distributed data system, the master node discards the write data and sends the data discarding request to the slave node so that the slave node discards the synchronous data.
In addition, the embodiment of the invention also provides electronic equipment, which comprises a processor and a memory, wherein the memory stores application programs, and the processor is used for running the application programs in the memory to realize the data storage method provided by the embodiment of the invention.
In addition, the embodiment of the invention also provides a computer readable storage medium, which stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor to execute the steps in any of the data storage methods provided by the embodiment of the invention.
After a slave node receives a data writing request sent by a terminal, the data writing request carries writing data, the data writing request is used for writing the writing data into nodes of a distributed data system, a slave node obtains a data number of the writing data from a master node according to the data writing request, the data number is a sequential identification of the writing data in the distributed data system, the slave node sends the writing data and the data number as synchronous data to other nodes of the distributed data system, receives feedback information sent by the other nodes, determines the first node number of the other nodes which successfully receive the synchronous data according to the feedback information, and stores the writing data according to the first node number by the slave node; according to the scheme, the slave nodes can be used for receiving the written data and numbering the written data, the master nodes are used for synchronizing the data numbers and the written data as synchronous data to other nodes of the distributed data system, a plurality of nodes are used for receiving the written data and operating the master nodes for numbering, so that a plurality of nodes can be used for receiving more written data, and orderly synchronization of the written data can be ensured, and therefore, the efficiency of data storage can be greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic view of a scenario of a data storage method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a data storage method according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of acquiring a data number from a node according to an embodiment of the present invention;
FIG. 4 is a schematic flow chart of testing the status of a master node according to an embodiment of the present invention;
FIG. 5 is a schematic flow chart of determining a new master node according to an embodiment of the present invention;
FIG. 6 is a flow chart of write data synchronization provided by an embodiment of the present invention;
FIG. 7 is a schematic flow chart of single node fault recovery provided by an embodiment of the present invention;
FIG. 8 is a schematic diagram of an original master node in a first sub-node cluster in a split brain fault according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of an original master node in a second sub-node cluster in a split brain fault according to an embodiment of the present invention;
FIG. 10 is a schematic diagram of another flow of data storage provided by an embodiment of the present invention;
FIG. 11 is a schematic diagram of a data storage device according to an embodiment of the present invention;
FIG. 12 is a schematic diagram of an acquisition unit of a data storage device according to an embodiment of the present invention;
FIG. 13 is a schematic diagram of another structure of a data storage device according to an embodiment of the present invention;
fig. 14 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention.
The embodiment of the invention provides a data processing method, a data processing device and a computer readable storage medium. The data processing device can be integrated in an electronic device, and the electronic device can be a server or a terminal, wherein the data processing device is mainly used for data writing operation of a distributed data system, can be a data writing operation device of a master node and a slave node in the distributed data system, can be a node cluster, and can comprise the master node and the slave node in the node cluster.
For example, referring to fig. 1, taking a data processing apparatus integrated in an electronic device and used for data writing of a slave node in a distributed data system as an example, the slave node receives a data writing request sent by a terminal, the data writing request carries writing data, the slave node obtains a data number of the writing data from a master node according to the data writing request, the slave node sends the writing data and the data number as synchronous data to other nodes of the distributed data system and receives feedback information sent by the other nodes, and the slave node determines the number of nodes of the other nodes successfully receiving the synchronous data according to the feedback information, and stores the writing data based on the number of nodes.
Optionally, when the write data is written from a master node of the distributed data system, the master node receives a data write request sent by the terminal, the master node generates a data number of the write data according to the data write request, the master node sends the write data and the data number as synchronous data to a slave node of the distributed data system, receives feedback information sent by the slave node, determines the number of the slave nodes successfully receiving the synchronous data according to the feedback information, and stores the write data based on the number of the nodes.
The following will describe in detail. The following description of the embodiments is not intended to limit the preferred embodiments.
The present embodiment will be described in terms of a data storage device, which may be specifically integrated in an electronic apparatus, where the electronic apparatus may be a server or may be a device such as a terminal; the terminal may include a tablet computer, a notebook computer, a personal computer (PC, personal Computer), and the like.
A data storage method, applied to a slave node of a distributed data system, comprising:
the slave node receives a data writing request sent by a terminal, the data writing request carries writing data, the slave node obtains a data number of the writing data from the master node according to the data writing request, the slave node sends the writing data and the data number as synchronous data to other nodes of the distributed data system and receives feedback information sent by the other nodes, the slave node determines the number of nodes of the other nodes successfully receiving the synchronous data according to the feedback information, and the slave node stores the writing data based on the number of nodes.
As shown in fig. 2, the specific flow of the data storage method is as follows:
101. And receiving a data writing request sent by the terminal from the node.
The data writing request carries writing data, for example, can carry data such as video, audio, image and text, and is mainly used for writing the writing data into nodes of the distributed data system.
For example, there may be various ways of receiving the data writing request sent by the terminal from the node, for example, the terminal may directly send the data writing request to the slave node of the distributed data system by using the address of the node, and the slave node receives the data writing request. The terminal can also store the writing data carried in the writing data request to the third party database, and send the storage address of the writing data stored in the third party database to the slave node in the writing data request, after receiving the data writing request from the slave node, the terminal obtains the storage address of the writing data stored in the third party database in the data writing request, and according to the storage address, obtains the writing data in the third party database, and can also send prompt information to the terminal after obtaining the writing data, so as to prompt the terminal to successfully obtain the writing data from the slave node.
102. And the slave node acquires the data number of the written data from the master node according to the data writing request.
The data number (Transaction Identity document, txID) is a sequential identifier of the write data in the distributed data system, for example, when a node of the distributed data system receives the first write data, the master node may number the data with TxID1, when the node of the distributed data system receives the second write data, the master node may number the data with TxID2, when the node of the distributed data system receives the third write data, the master node may number the data with TxID3, and so on, where the data number is mainly the sequence of identifying the received write data in the distributed data system.
For example, the process of the slave node acquiring the data number of the write data according to the data write request may be as shown in fig. 3, and specifically is as follows:
s1, generating a data synchronization request from a node according to a data writing request.
The data synchronization request is used for acquiring a data number corresponding to the write-in data from the master node.
For example, the slave node generates a data synchronization request according to the data writing request, for example, after receiving the data writing request from the slave node, generates the data synchronization request according to the received data writing request, and adds information related to the writing data, such as the name, the receiving time, the type and/or the data size of the writing data, to the data synchronization request.
S2, the slave node sends a data synchronization request to the master node.
For example, the slave node obtains an address of a master node in the distributed data system, and according to the address, the slave node transmits a data synchronization request to the master node.
And S3, when the master node returns the data number, the slave node receives the data number.
For example, after the master node successfully receives the data synchronization request, the master node numbers the written data according to the related information of the written data in the data synchronization request, generates a data number of the written data, sends the data number of the written data to the slave node, and the slave node receives the data number.
And S4, when the master node does not return the data number, the slave node returns prompt information of data writing failure to the terminal.
For example, when the master node does not successfully receive the data synchronization request or does not return the data number of the write data after receiving the data synchronization request, the slave node does not receive the number of the write data, and at this time, the slave node returns a prompt message of data write failure to the terminal and discards the write data.
Optionally, before the slave node sends the data synchronization request to the master node, the state of the master node may be tested, and when the master node is in a normal state, the data synchronization request is sent to the master node, and when the master node is in an abnormal state, a new master node needs to be reselected and selected in the distributed data system. Thus, before the step "slave node sends a data synchronization request to master node", the data storage method may further include:
The slave node sends a test request to the master node, the test request is used for testing the state of the master node, the slave node receives feedback information of the test request and determines whether the master node is in a normal state according to the feedback information, and when the master node is in the normal state, the slave node sends a synchronous data request to the master node.
For example, as shown in fig. 4, when the master node receives the test request, it delays for a period of time (Td) to send the feedback information, so that the state of the master node can be determined according to the time when the feedback information sent by the master node is received, when the feedback information sent by the master node is not received in the Td, the distributed data system can generate the feedback information that the master node is in an abnormal state, when the feedback information sent by the master node is received in the Td, it can indicate that the master node is in a normal state at the moment, and when the feedback information is that the master node is in a normal state, it indicates that the master node is still in a normal state at the moment, and the master-slave relationship between the master node and the slave node is still in a normal state. When the master node is in a normal state, the master-slave relationship in the distributed data system is still present, and the slave node can send a data synchronization request to the master node.
The slave node tests whether the master node is normal, that is, whether the master-slave relationship between the slave node and the master node is normal, and the testing process may be that testing is performed before synchronous data is sent to the master node after each data writing request is received, or a time interval may be set, and the slave node sends a test request at every other time interval to test the state of the master node at this time, i.e. no data writing request is received at this time, for example, it may be set that the slave node sends a test request to the master node at every 1 minute.
Optionally, when the feedback information received by the slave node is that the master node is in an abnormal state, the slave node determines that the master node is in an abnormal state. When the master node is in an abnormal state, the slave node triggers a new master node determining process, for example, when the master node is in an abnormal state, the slave node triggers a process of determining a new master node in the distributed data system, the process is used for selecting the new master node from the distributed data system, and a candidate object of the new master node is the slave node triggering the process. The main node is in an abnormal state and needs to confirm for many times, for example, when the main node is sent with a test request, the main node does not receive feedback information within a preset time interval or the feedback information is not received by the main node, the auxiliary node needs to send the test request again outside the preset time interval, if the feedback information of the test request is still abnormal for the main node or the main node does not receive the test request, the main node can be truly confirmed to have the abnormality, if the feedback information of the second test request is normal for the main node, the main node is still in the normal state, and when the main node is in the normal state, the main node is continuously sent with the data synchronization request.
Optionally, when the slave node determines that the master node is in an abnormal state, a new master node needs to be determined in the distributed data system, and at this time, the distributed data system enters a new master node determining process, and the working process of the slave node is shown in fig. 5, which specifically includes the following steps:
the slave node sends a voting request for selecting a new master node to other slave nodes in the distributed data system, wherein the voting request carries the largest first data number in the data numbers of the slave nodes, the slave node receives the voting results replied by the other slave nodes aiming at the voting request, and the voting results can be in two cases, one is that the slave node which agrees to send the voting request is the new master node, and the other is that the slave node which does not agree to send the voting request is the new master node. When the number of voting results of the slave node agreeing to issue a voting request for the new master node is more than half of the total number of slave nodes of the distributed data system, the slave node automatically selects as the master node. After the slave node becomes the master node after the master node is selected, the slave node itself can generate the data number of the write data and perform the next step of storing the write data. When the slave nodes receive the voting results of the slave nodes which agree to send out the voting request as the new master nodes and the number of the voting results of the slave nodes does not exceed half of the total number of the slave nodes of the distributed data system, the process of selecting the new master nodes initiated by the slave nodes fails in the process of selecting the master nodes, and the slave nodes return prompt information of data writing failure to a terminal which sends the data writing request.
Optionally, in the process of the main flow, after receiving the voting request, the other slave nodes compare the first data number of the slave node carried in the voting request with the largest second data number in the self data numbers. When the second data number does not exceed the first data number, the other slave nodes send the voting result of the slave node which agrees to send the voting request to the slave node as the new master node, for example, when the first data number is TxID5 and the second data number is TxID4, the other slave nodes find that the second data number of the slave node does not exceed the first data number of the slave node, and at this time, the other slave nodes send the voting result of the slave node which agrees to send the voting request to the slave node as the new master node. When the largest second data number in the self data numbers of other slave nodes exceeds the first data number, the other slave nodes send voting results of the slave nodes which do not agree to send voting requests to the slave nodes as new master nodes, for example, the second data number is TxID5, and when the first data number is TxID4, the other slave nodes send voting results of the slave nodes which do not agree to send voting requests to the slave nodes as new master nodes.
Optionally, when the second data number does not exceed the first data number and the other slave nodes receive the voting requests sent by two different slave nodes within the preset time interval, in this case, the voting results need to be sent to the slave nodes corresponding to the different voting requests according to the number of times of receiving the voting requests, and in general, the other slave nodes send the voting results of the slave nodes which agree to send the voting request to the first slave node sending the voting request as the new master node. For example, when the first data number of two different slave nodes is TxID5 and the second data number is TxID4, in a time interval, other slave nodes send voting results of the slave nodes which agree to send the voting request to the slave node which sends the voting request to itself for the first time, for example, in five minutes, other slave nodes receive two voting requests, the two voting requests respectively correspond to the first slave node and the second slave node, but the other slave nodes first receive the voting request sent by the first slave node, at this time, the first data number of the first slave node exceeds the second data number of the slave node, and the other slave nodes send voting results which agree to the first slave node as a new master node to the first slave node, although the first data number of the second slave node is also greater than the second data number of the slave node, the other slave nodes still need to send voting results which do not agree to the second slave node as a new master node to the second slave node. Assuming that the first slave node transmits a voting request, the other slave nodes transmit voting results agreeing to the first slave node as a new master node. After the preset time interval is exceeded, for example, the preset time interval is ten minutes, but at the eleventh minute, the other slave nodes receive the voting request sent by the second slave node again, and when the first data number of the second slave node exceeds the second data number of the other slave nodes, the other slave nodes can still send the voting result of agreeing to the second slave node as the new master node to the voting request of the second slave node.
103. The slave node transmits the written data and the data number as synchronous data to other nodes of the distributed data system and receives feedback information transmitted by the other nodes.
For example, the slave node transmits the write data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information transmitted by the other nodes. For example, in the case of a distributed data system, there are a master node and two slave nodes, namely, a slave node a and a slave node B, respectively, the slave node a transmits the write data and the data number of the write data, such as TxID2, as the synchronization data to the master node and the slave node B, when the master node receives the synchronization data, feedback information of successfully receiving the synchronization data is transmitted to the slave node a, and when the master node does not receive the synchronization data, the master node cannot transmit the feedback information to the slave node a, and similarly, the slave node B transmits the feedback information to the slave node a only when the slave node receives the synchronization data successfully. When the master node or the slave node B transmits feedback information to the slave node a, the slave node receives the feedback information.
104. And the slave node determines the first node number of other nodes which successfully receive the synchronous data according to the feedback information.
For example, the slave node determines, according to the feedback information, the number of nodes of other nodes that successfully receive the synchronization data, for example, whether a distributed data system has one master node and two slave nodes, namely, a slave node a and a slave node B, and when the master node successfully receives the synchronization data, the master node sends feedback information to the slave node a, so that when the slave node a receives the feedback information of the master node, the master node can be determined to successfully receive the synchronization data, and similarly, the slave node B can be determined, according to the feedback information, the number of nodes that successfully receives the synchronization data, for example, the number of first nodes that successfully receives the synchronization data can be determined to be two, and for example, the number of first nodes that successfully receives the synchronization data can be determined to be one.
105. The slave node stores the write data based on the first node number.
For example, the slave node compares the first node number of the other nodes that successfully received the synchronization data with half of the total number of nodes of the distributed data system, when the first node number exceeds half of the total number of nodes of the distributed data system, for example, when the distributed data system has one master node and three slave nodes, the slave node sends the synchronization data when the number of the other nodes that successfully received the synchronization data exceeds two, for example, when the other nodes that successfully received the synchronization data are the master node, the slave node B and the slave node C, at this time, the slave node writes the synchronization data into the local database of the slave node a while sending a data save request to the master node, the slave node B and the slave node C, and when the master node, the slave node B and the slave node C receive the data save request, the master node, the slave node B and the slave node C store the synchronization data to their own local databases, respectively, to complete synchronization of the written data in the distributed data system. When the number of the first nodes does not exceed half of the total number of nodes of the distributed data system, for example, when one master node and three slave nodes exist in the distributed data system, when the number of nodes of other nodes which successfully receive the synchronous data does not exceed two, for example, when the slave node A sends the synchronous data, the other nodes which successfully receive the synchronous data are the master node and the slave node B, at this time, the slave nodes discard the written data and send data discarding requests to the other nodes, and when the other nodes receive the data discarding requests, the nodes which receive the synchronous data in the other nodes discard the synchronous data.
Optionally, when the master node of the distributed data system receives a data writing request, the synchronization process for the writing data is as shown in fig. 6, and specifically includes the following steps:
the method comprises the steps that a master node receives a data writing request sent by a terminal, the master node generates a data number of the writing data according to the data writing request, the master node sends the writing data and the data number to slave nodes of a distributed data system as synchronous data, receives feedback information sent by the slave nodes, determines the number of second nodes of the slave nodes successfully receiving the synchronous data according to the received feedback information, stores the writing data to a local database of the master node when the number of the second nodes exceeds half of the total number of nodes of the distributed data system, and sends a data saving request to the slave nodes so as to save the synchronous data to the local database of the slave nodes.
For example, the master node receives a data writing request sent by the terminal, generates a data number of writing data in the writing data request according to the data writing request, sends the writing data and the data number as synchronous data to the slave nodes of the distributed data system, when the slave nodes receive the synchronous data, the slave nodes send feedback information of successfully receiving the synchronous data to the master nodes, the master nodes determine the second node number of the slave nodes successfully receiving the synchronous data according to the feedback information, when the second node number exceeds half of the total number of nodes of the distributed data system, for example, the total number of nodes of the distributed data system is 6, when the second node number of the slave nodes receiving the synchronous data exceeds 3, for example, the second node number is 4, the master nodes store (write) the writing data to the local database of the master nodes, send a data saving request to the slave nodes, and the slave nodes receiving the synchronous data save the synchronous data to the local database of the slave nodes after receiving the data saving request.
Optionally, in order to solve the problem of node failure recovery in the distributed data system, the slave node and the master node may be set to meet the requirement of the use of the distributed data system, and specifically, the slave node and the master node may be set in failure recovery of a single node and in brain fracture (network isolation) failure recovery.
(1) Handling single node failure recovery
For example, as shown in fig. 7, the slave node may send a data detection request to the master node at a preset time interval, where the data detection request is used to obtain the largest target data number among the data numbers of the master node. When the slave node fails, a data detection request cannot be sent, but once the slave node fails and is recovered, the data detection request is sent to the master node immediately after the failure recovery, when the master node is in a normal state, the slave node can receive the target data number sent by the master node, and the slave node compares the target data number with the largest current data number in the self data numbers. When the target data number exceeds the current data number, for example, the target data number is TxID5 and the current data number is TxID4, the slave node determines that there is a missing of its own data and acquires the missing data from the master node. For example, when the slave node confirms that the own data is missing, the slave node sends a data acquisition request to the master node, the data acquisition request carries the current data number of the slave node, and the slave node receives the missing data sent by the master node according to the current data number and stores the missing data in a local database. When the target data number does not exceed the current data number, the slave node determines that no missing data exists, and at the moment, the slave node is in a normal state, can receive synchronous data and can also receive a data writing request.
Optionally, in this case, the master node receives the data acquisition request, the master node compares the current data number in the data acquisition request with the own target data number, and the master node determines, according to the comparison result, the data missing from the slave node, for example, when the target data number is TxID5 and the current data number is TxID4, the master node determines that the data missing from the slave node is write data corresponding to TxID5, and the master node sends the write data corresponding to TxID5 to the slave node as the data missing from the slave node.
Optionally, when the original master node in the distributed data system fails, the slave node may select a new master node in the distributed data system again, and when the data number of the original master node is smaller than the maximum data number of the new master node, the original master node is used as a slave node to send a data acquisition request to the new master node at this time, and the process of recovering from the failure of the slave node is repeated.
(2) Coping with split brain (network isolated) fault recovery
The brain fracture fault is that a node in a node cluster of the distributed data system is isolated by a network, for example, one node cluster is split into two sub-node clusters with different numbers, and the nodes of the two sub-node clusters cannot communicate with each other. At this time, there are two cases, the first case is a first sub-node cluster in which the number of the original master nodes in the distributed data system exceeds half of the total number of the slave nodes, and the second case is a second sub-node cluster in which the number of the original master nodes in the distributed data system does not exceed half of the total number of the slave nodes.
And C1, when the original main node is in the first sub-node cluster, performing fault recovery process of the distributed data system.
For example, when the original master node is in the first sub-node cluster, as shown in fig. 8, since the number of slave nodes in the first sub-node cluster exceeds half of all slave nodes, when the slave node or the master node receives a data writing request in the first sub-node cluster, since the master node and the slave node are in a normal state in the first sub-node cluster, writing data in the data writing request can be synchronized, and in this case, the entire distributed data system can be operated normally. However, when the slave node 4 or the slave node 5 in the second sub-node cluster receives the data writing request, the slave node 4 or the slave node 5 sends a data synchronization request to the master node, but due to the occurrence of a brain fracture fault, the slave node 4 or the slave node 5 cannot acquire the data number of the writing data from the master node, and the slave node 4 or the slave node 5 determines that the master node is in an abnormal state, so that the slave node 4 or the slave node 5 triggers a new master node determining process, but after the slave node 4 or the slave node 5 triggers the new master node determining process, only one slave node receives a voting request and returns a voting result, and the slave node which agrees to send the voting request can determine the new master node because the number of the agreed voting results does not exceed half of the total number of the slave nodes, thereby causing the data writing failure. When the network isolation fault in this case is recovered, when the slave node 4 or the slave node 5 of the second sub-node cluster can receive the synchronous data sent by the slave node except the other party, the slave node 4 or the slave node 5 can know that the master node is in a normal state, in this case, the slave node 4 or the slave node 5 will send a data detection request to the master node to determine whether the data is missing or not, after receiving the target data number sent by the master node, the target data number is compared with the current data number with the maximum value, when the target data number exceeds the current data number, the slave node determines that the data is missing and sends the current data number to the master node, and after receiving the current data number sent by the slave node 4 or the slave node 5, the master node determines the data missing from the slave node 4 or the slave node 5 and sends the missing data to the slave node 4 or the slave node 5.
Wherein in this case a slave node in the second cluster of child nodes is in need of failure recovery.
And C2, when the original main node is in the second sub-node cluster, performing fault recovery process of the distributed data system.
For example, when the original master node is in the second sub-node cluster, as shown in fig. 9, when a network isolation fault occurs and a slave node in the first sub-node cluster receives a data writing request, such as a slave node 1 receives a data writing request sent by a terminal, at this time, the slave node sends a data synchronization request to the master node, and as a result, the slave node 1 determines that the master node is in an abnormal state due to the network isolation fault, so that the slave node 1 triggers a flow of determining a new master node in the first sub-node cluster, the slave node 1 sends a voting request to the slave nodes 2-4, the slave node 1 receives voting results sent by the slave nodes 2-4, and when three slave nodes 1 which agree to send the voting request exist in the voting results as new master nodes, the slave node 1 selects as the new master node in the first sub-node cluster. In the second sub-node cluster, when the master node or the slave node 5 receives the data writing request, the number of other nodes which successfully receive the synchronous data is not more than half of the total number of nodes when the synchronous data is sent to the other nodes, so that the data writing is failed, and therefore the second sub-node cluster cannot work normally.
When the network fails back, there will be two master nodes in the distributed data system, one being the original master node and the other being the slave node becoming the new master node. In a distributed data system, only when the original master node and the new master node have the process of sending and receiving synchronous data, the original master node and the new master node can sense the existence of the other party.
The original master node and the new master node perceive the existence of the other party mainly in two cases:
the first method is that when any one of the original master node or the new master node receives a data writing request, for example, when the original master node receives the data writing request, the original master node generates a data number of writing data, the writing data and the data number of the writing data are used as synchronous data to be sent to other nodes of the distributed data system, when the synchronous data are sent to the new master node, and the new master node successfully receives the synchronous data, the original master node can determine the identity of the new master node, and when the original master node determines the identity of the new master node, the identity of the new master node is obtained.
The second is that any node in the original master node or the new master node receives the synchronous data sent by the other party, for example, taking the example that the original master node receives the synchronous data sent by the new master node, the original master node can determine the identity of the new master node sending the synchronous data and acquire the identity of the new master node.
When the master node determines that other master nodes exist in the distributed data system, the master node sends a data comparison request to the other master nodes according to the acquired identity of the other master nodes, wherein the data comparison request is used for acquiring the first largest target data number in the data numbers of the other master nodes. The master node receives a first target data number sent by other master nodes, compares the first target data number with a second maximum target data number in own data numbers, and when the first target data number exceeds the second target data number of the master node, the master node becomes a slave node and acquires data missing from the master node from the other master nodes. For example, when the first target data is numbered TxID5 and the second target data is numbered TxID4, the original master node becomes a slave node and acquires the missing data of the original master node from the new master node.
Alternatively, the process of acquiring missing data by the master node is similar to that of acquiring missing data by the slave node, when the master node determines that there is a data missing by itself, the master node becomes the slave node at this time, and thus, the slave node acquires missing data from another master node.
As can be seen from the foregoing, in this embodiment, after a slave node receives a data write request sent by a terminal, the data write request carries write data, where the data write request is used to write the write data into a node of a distributed data system, the slave node obtains, according to the data write request, a data number of the write data from a master node, where the data number is a sequential identifier of the write data in the distributed data system, the slave node sends the write data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information sent by the other nodes, and the slave node determines, according to the feedback information, a first node number of the other nodes that successfully receives the synchronous data, where the slave node stores the write data based on the first node number; according to the scheme, the slave nodes can be used for receiving the written data and numbering the written data, the master nodes are used for synchronizing the data numbers and the written data as synchronous data to other nodes of the distributed data system, a plurality of nodes are used for receiving the written data and numbering the written data, and the master nodes are used for numbering, so that the plurality of nodes can be used for receiving more written data, and the orderly synchronization of the written data can be ensured, and therefore, the efficiency of data storage can be greatly improved.
According to the method described in the above embodiments, examples are described in further detail below.
In this embodiment, the data storage device is specifically integrated in an electronic device, and the electronic device may include a master node server and a slave node server, for example, will be described.
As shown in fig. 10, a data storage method specifically includes the following steps:
201. and receiving a data writing request sent by the terminal from the node server.
For example, the terminal may send a data write request directly to the slave node server of the distributed data system at the address of the node, and the slave node server receives the data write request. The terminal can also store the writing data carried in the writing data request to a third party database, and send the storage address of the writing data stored in the third party database to the slave node server in the writing data request, after receiving the data writing request from the node server, the terminal obtains the storage address of the writing data stored in the third party database in the data writing request, and according to the storage address, obtains the writing data in the third party database, and can also send prompt information to the terminal after obtaining the writing data, so as to prompt the terminal to successfully obtain the writing data from the node server.
202. A data synchronization request is generated from the node server in accordance with the data write request.
For example, after receiving a data writing request from a node server, a data synchronization request is generated according to the received data writing request, and relevant information of the writing data, such as the name, the receiving time, the type and/or the data size of the writing data, is added in the data synchronization request.
203. The slave node server transmits a data synchronization request to the master node server.
For example, the slave node server obtains an address of a master node server in the distributed data system, and according to the address, the slave node server transmits a data synchronization request to the master node server.
Optionally, before the slave node server sends the data synchronization request to the server of the master node, the state of the master node server may be tested, and when the master node server is in a normal state, the data synchronization request is sent to the master node, and when the master node server is in an abnormal state, a new master node server needs to be reselected and selected in the distributed data system. Thus, before the step of "the slave node server transmits the data synchronization request to the master node server", the data storage method may further include:
The slave node server sends a test request to the master node server, the test request is used for testing the state of the master node server, the slave node server receives feedback information of the test request and determines whether the master node server is in a normal state according to the feedback information, and when the master node server is in the normal state, the slave node server sends a synchronous data request to the master node server.
For example, whether the master node server is in a normal state is tested, the slave node server sends a test request to the master node server, when the master node server receives the test request, a period of time (Td) is delayed to send feedback information, so that the state of the master node server can be judged according to the time when the feedback information sent by the master node server is not received in the Td, the distributed data system can generate feedback information that the master node server is in an abnormal state, when the feedback information sent by the master node server is received in the Td, the state of the master node server can be indicated to be in a normal state at the moment, the slave node server receives the feedback information of the test request, and when the feedback information is that the master node server is in a normal state, the master node server is still in a normal state at the moment, and the master-slave relationship between the master node server and the slave node server is still in a normal state. When the master node server is in a normal state, which indicates that the master-slave relationship still exists in the distributed data system, the slave node server can send a data synchronization request to the master node server.
Optionally, when the feedback information received from the node server is that the master node server is in an abnormal state, the slave node server determines that the master node server is in an abnormal state. When the master node server is in an abnormal state, the slave node server triggers a new master node server determining process, for example, when the master node server is in an abnormal state, the slave node server triggers a process of determining a new master node server in the distributed data system, the process is used for selecting the new master node server from the distributed data system, and a candidate object of the new master node server is the slave node server triggering the process. The main node server needs to confirm for many times when in an abnormal state, for example, when the main node server sends a test request, the main node server does not receive feedback information within a preset time interval or the feedback information is not received by the main node server, the auxiliary node server needs to send a test request again outside the preset time interval, if the feedback information of the test request is still abnormal or the main node server does not receive the test request, the main node server can be truly confirmed to have the abnormality, if the feedback information of the second test request is normal, the main node server is still in the normal state, and when the main node server is in the normal state, the main node server is continuously sent to the main node server.
Optionally, when the slave node server determines that the master node server is in an abnormal state, a new master node server needs to be determined in the distributed data system, and at this time, the distributed data system enters a new master node server determining process, and the working process of the slave node server is specifically as follows:
the slave node server sends a voting request for selecting a new master node server to other slave node servers in the distributed data system, wherein the voting request carries the largest first data number in the data numbers of the slave node servers, the slave node server receives voting results replied by the other slave node servers for the voting request, and the voting results can have two cases, namely, the slave node server which agrees to send the voting request is the new master node server, and the slave node server which does not agree to send the voting request is the new master node server. When the slave node server receives the voting result of the slave node server agreeing to issue the voting request for the new master node server, the slave node server automatically selects as the master node server when the number of voting results exceeds half of the total number of the slave node servers of the distributed data system. When the slave node server becomes the master node server after the master node server is selected, the slave node server itself can generate the data number of the write data and perform the next step of storing the write data. When the slave node servers receive the voting results of the slave node servers which agree to send out the voting request for the new master node server, and the number of the voting results of the slave node servers for the new master node server does not exceed half of the total number of the slave node servers of the distributed data system, the process initiated by the slave node servers for selecting the new master node server fails in selecting the master, and the slave node servers return prompt information of data writing failure to a terminal which sends the data writing request.
Optionally, in the process of the main flow selection, after receiving the voting request, the other slave node servers compare the first data number of the slave node server carried in the voting request with the largest second data number in the self data numbers. When the second data number does not exceed the first data number, the other slave node servers send the voting result of the slave node server agreeing to send the voting request to the slave node server as the new master node server, for example, when the first data number is TxID5 and the second data number is TxID4, the other slave node servers find that the second data number of the slave node server does not exceed the first data number of the slave node server, and at this time, the other slave node servers send the voting result of the slave node server agreeing to send the voting request to the slave node server as the new master node server. When the largest second data number in the self data numbers of the other slave node servers exceeds the first data number, the other slave node servers send voting results of the slave node servers which do not agree to send the voting request to the slave node servers for the new master node server, for example, when the second data number is TxID5 and the first data number is TxID4, the other slave node servers send voting results of the slave node servers which do not agree to send the voting request to the slave node servers for the new master node server.
Optionally, when the second data number does not exceed the first data number and the other slave node servers receive the voting requests sent by the two different slave node servers within the preset time interval, in this case, the voting results need to be sent to the slave node servers corresponding to the different voting requests according to the times of receiving the voting requests, and generally, the other slave node servers will send the voting results of the slave node servers agreeing to send the voting requests to the first slave node server sending the voting requests as the new master node server. For example, when the first data number of two different slave node servers is TxID5 and the second data number is TxID4, the other slave node servers send voting results of the slave node servers agreeing to issue voting requests to the slave node servers itself for the first time in a time interval, for example, in five minutes, the other slave node servers receive two voting requests, the two voting requests respectively corresponding to the first slave node server and the second slave node server, however, when the first data number of the first slave node server exceeds the second data number thereof, the other slave node servers send the voting result which agrees that the first slave node server is the new master node server to the first slave node server, and the first data number of the second slave node server is also larger than the second data number thereof, but the other slave node servers still need to send the voting result which does not agree that the second slave node server is the new master node server to the second slave node server. Assuming that the first slave node server transmits a voting request, the other slave node servers transmit voting results agreeing to the first slave node server as a new master node server. After the preset time interval is exceeded, for example, the preset time interval is ten minutes, but at the eleventh minute, the other slave node servers receive the voting request sent by the second slave node server again, and when the first data number of the second slave node server exceeds the second data number of the other slave node servers, the other slave node servers can still send the voting result of agreeing to the second slave node server as the new master node server to the voting request of the second slave node server.
204. When the master node server returns the data number, the slave node server receives the data number.
For example, after the master node server successfully receives the data synchronization request, the master node server numbers the written data according to the related information of the written data in the data synchronization request, generates a data number of the written data, sends the data number of the written data to the slave node server, and receives the data number from the slave node server.
205. And when the master node server does not return the data number, the slave node server returns prompt information of data writing failure to the terminal.
For example, when the master node server does not successfully receive the data synchronization request or does not return the data number of the write data after receiving the data synchronization request, the slave node server does not receive the number of the write data, and at this time, the slave node server returns a prompt message of data write failure to the terminal and discards the write data.
206. When the slave node server receives the data number, the slave node server transmits the written data and the data number as synchronous data to other node servers of the distributed data system, and receives feedback information transmitted by the other node servers.
For example, in the case of a distributed data system in which there are a master node server and two slave node servers, namely, a master node server, a slave node server a and a slave node server B, the slave node server a transmits the write data and the data number of the write data, such as TxID2, as synchronization data to the master node server and the slave node server B, when the master node server receives the synchronization data, feedback information that the synchronization data is successfully received is transmitted to the slave node server a, and when the master node server does not receive the synchronization data, the master node server cannot transmit the feedback information to the slave node server a, and similarly, the slave node server B transmits the feedback information to the slave node server a only when the synchronization data is successfully received. When the master node server or the slave node server B transmits feedback information to the slave node server a, the slave node server receives the feedback information.
207. And determining the first node quantity of other node servers which successfully receive the synchronous data according to the feedback information by the slave node servers.
For example, there is a master node server and two slave node servers in the distributed data system, namely, a master node server, a slave node server a and a slave node server B, and when the master node server successfully receives the synchronization data, the master node server sends feedback information to the slave node server a, so that when the slave node server a receives the feedback information of the master node server, the master node server can be determined to successfully receive the synchronization data, and similarly, for the slave node server B, the number of nodes successfully receiving the synchronization data, for example, the number of first nodes successfully receiving the synchronization data can be determined to be two according to the feedback information of the master node and the slave node server B, and for example, the number of first nodes successfully receiving the synchronization data can be determined to be one according to the feedback information of the master node and the slave node server B.
208. The slave node server stores the write data based on the first node number.
For example, the slave node server compares the first node number of the other node servers that successfully received the synchronization data with half of the total node servers of the distributed data system, when the first node number exceeds half of the total node servers of the distributed data system, for example, when the distributed data system has one master node server and three slave node servers, the master node server, the slave node server B and the slave node server C store the synchronization data to their own local databases respectively when the node numbers of the other node servers that successfully received the synchronization data exceed two, for example, the slave node server a sends the synchronization data, and the other node servers that successfully received the synchronization data are the master node server, the slave node server B and the slave node server C, at this time, the slave node server a writes the synchronization data to the local databases of the slave node server a while sending data save requests to the master node server, the slave node server B and the slave node server C, and when the master node server, the slave node server B and the slave node server C receive the data save requests, respectively, to complete the synchronization data writing of the local databases to the distributed data system. When the number of the first node does not exceed half of the total number of node servers of the distributed data system, for example, when one master node server and three slave node servers exist in the distributed data system, and when the number of nodes of other node servers which successfully receive the synchronous data does not exceed two, for example, when the slave node server A sends the synchronous data, the other node servers which successfully receive the synchronous data are the master node server and the slave node server B, at this time, the slave node servers discard the written data and send data discarding requests to the other node servers, and when the other node servers receive the data discarding requests, the node servers which receive the synchronous data, for example, the master node server and the slave node server B discard the synchronous data.
Alternatively, when a master node server in the distributed data system receives a data write request, the synchronization process for the write data may be as follows:
the master node server receives a data writing request sent by the terminal, generates a data number of the writing data according to the data writing request, sends the writing data and the data number to a slave node server of the distributed data system as synchronous data, receives feedback information sent by the slave node server, determines the number of second nodes of the slave node server successfully receiving the synchronous data according to the received feedback information, and when the number of the second nodes exceeds half of the total number of the node servers of the distributed data system, the master node server stores the writing data to a local database of the master node server and sends a data saving request to the slave node server so as to save the synchronous data to the local database of the slave node server.
For example, the master node server receives a data writing request sent by the terminal, generates a data number of writing data in the writing data request according to the data writing request, sends the writing data and the data number as synchronous data to the slave node servers of the distributed data system, when the slave node server receives the synchronous data, the slave node server sends feedback information of successfully receiving the synchronous data to the master node server, the master node server determines the second node number of the slave node servers successfully receiving the synchronous data according to the feedback information, when the second node number exceeds half of the total number of the node servers of the distributed data system, for example, the total number of nodes of the distributed data system is 6, when the second node number of the slave node servers receiving the synchronous data exceeds 3, for example, the second node number is 4, the master node server stores (writes) the writing data to the local database of the master node server, sends a data saving request to the slave node server, and after the slave node server receives the data saving request, the slave node server saves the synchronous data to the local database of the slave node server.
Optionally, in order to solve the problem that the node server in the distributed data system fails and recovers, the master node server and the slave node server may be set to meet the requirement of the use of the distributed data system. The method can be particularly used for setting a slave node server and a master node server in the fault recovery of a single node server and the recovery and the fault recovery of brain fracture (network isolation) in a distributed data system.
(1) Processing failure recovery for a single node server
For example, when the slave node server fails and recovers, the slave node server may transmit a data detection request to the master node server at a preset time interval, the data detection request being used to acquire a maximum target data number among the data numbers of the master node server. When the slave node server fails, a data detection request cannot be sent, but once the slave node server fails to recover, the data detection request is sent to the master node server immediately after the failure recovery, when the master node server is in a normal state, the slave node server can receive the target data number sent by the master node server, and the slave node server compares the target data number with the largest current data number in the self data numbers. When the target data number exceeds the current data number, for example, the target data number is TxID5 and the current data number is TxID4, the slave node server determines that the own data is missing, and acquires the missing data from the master node server. For example, when the slave node server confirms that the own data is missing, the slave node server sends a data acquisition request to the master node server, the data acquisition request carries the current data number of the slave node server, the slave node server receives the missing data sent by the master node server according to the current data number, and the missing data is stored in the local database.
Optionally, in this case, the master node server receives the data acquisition request, the master node server compares the current data number in the data acquisition request with the own target data number, and determines, according to the comparison result, the data missing from the slave node server, for example, when the target data number is TxID5 and the current data number is TxID4, the master node server determines that the data missing from the slave node server is write data corresponding to TxID5, and the master node server sends the write data corresponding to TxID5 to the slave node server as the data missing from the slave node server.
Optionally, when the original master node server in the distributed data system fails, the slave node server may select a new master node server in the distributed data system again, and the original master node server is used as the slave node server to send a data acquisition request to the new master node server at this time, and the process of recovering from the failure of the slave node server is repeated.
(2) Fault recovery for handling brain cracks (network isolation)
For example, in handling a fault recovery of a brain split (network isolation), since a node server cluster may split into a first sub-node service cluster and a second sub-node server cluster with unequal numbers of nodes. The first case is considered to be a first sub-node server cluster in which the number of slave nodes of the original master node server in the distributed data system exceeds half of the total number of slave node servers, and the second case is considered to be a second sub-node server cluster in which the number of slave node servers of the original master node server in the distributed data system does not exceed half of the total number of slave node servers.
A1, when the original main node server is in the first sub node server cluster, a fault recovery process of the distributed data system is carried out.
For example, when the original master node server is in the first sub node server cluster, since the number of slave node servers in the first sub node server cluster exceeds half of all slave node servers, when the slave node server or the master node server receives the data writing request in the first sub node server cluster, since the master node server and the slave node server are both in a normal state, the writing data in the data writing request can be synchronized, and in this case, the entire distributed data system can be operated normally. However, when the slave node server 4 or the slave node server 5 in the second sub node server cluster receives the data writing request, the slave node server 4 or the slave node server 5 sends the data synchronization request to the master node server, but due to the occurrence of the brain-splitting fault, the slave node server 4 or the slave node server 5 cannot acquire the data number of the writing data from the master node server, and the slave node server 4 or the slave node server 5 determines that the master node server is in an abnormal state, so the slave node server 4 or the slave node server 5 triggers a new master node server determining process, but after the slave node server 4 or the slave node server 5 triggers the new master node server determining process, only one slave node server receives the voting request and returns the voting result, even if the slave node server which sends the voting request is agreed to send, the new master node server is determined to fail because the number of the agreed voting results does not exceed half of the total number of the slave node servers, and further causes the data writing failure. When the network isolation fault in this case is recovered, when the slave node server 4 or the slave node server 5 of the second sub node server cluster can receive the synchronization data sent from the slave node servers except the other party, the slave node server 4 or the slave node server 5 can know that the master node server is in a normal state, in this case, the slave node server 4 or the slave node server 5 will send a data detection request to the master node server to determine whether there is a data missing or not, after receiving the target data number sent by the master node server, the target data number is compared with the current data number with the largest value of the target data number, when the target data number exceeds the current data number, the slave node server determines that there is a data missing and sends the current data number of the slave node server to the master node server, and after receiving the current data number sent by the slave node server 4 or the slave node server 5, the master node server determines the missing data of the slave node server 4 or the slave node server 5 and sends the missing data to the slave node server 4 or the slave node server 5.
Wherein in this case what is in need of failure recovery is a slave node server in the second cluster of child node servers.
A2, when the original main node server is in the second sub node server cluster, the fault recovery process of the distributed data system is carried out.
For example, when the original master node server is in the second sub node cluster, when a network isolation fault occurs and the slave node server in the first sub node server cluster receives a data writing request, for example, the slave node server 1 receives a data writing request sent by a terminal, at this time, the slave node server sends a data synchronization request to the master node server, and as a result, the slave node server 1 determines that the master node server is in an abnormal state due to the network isolation fault, therefore, the slave node server 1 triggers a process of determining a new master node server in the first sub node server cluster, the slave node server 1 sends a voting request to the slave node servers 2-4, the slave node server 1 receives a voting result sent by the slave node servers 2-4, and when three slave node servers 1 which agree to send the voting request exist in the voting result as new master node servers, the slave node server 1 is selected as the new master node server in the first sub node server cluster. In the second sub-node cluster, when the master node server or the slave node server 5 receives the data writing request, the number of other nodes which successfully receive the synchronous data is not more than half of the total number of nodes when the synchronous data is sent to the other nodes, so that the data writing fails, and therefore the second sub-node cluster cannot work normally.
When the network fails back, there will be two master node servers in the distributed data system, one being the original master node server and the other being the slave node server becoming the new master node server. In the distributed data system, only when the original master node server and the new master node server have the process of sending and receiving synchronous data, the original master node server and the new master node server can sense the existence of the other side.
The original master node server and the new master node server perceive the existence of the other party mainly in two cases:
the first method is that when any one of the original master node server or the new master node server receives a data writing request, for example, when the original master node server receives the data writing request, the original master node server generates a data number of writing data, the writing data and the data number of the writing data are used as synchronous data to be sent to other nodes of the distributed data system, when the synchronous data are sent to the new master node server, and the new master node server successfully receives the synchronous data, the original master node server can determine the identity of the new master node server, and when the original master node server determines the identity of the new master node server, the identity of the new master node server is obtained.
The second is that any node in the original master node server or the new master node server receives the synchronous data sent by the other party, for example, taking the example that the original master node server receives the synchronous data sent by the new master node server, the original master node server can determine the identity of the new master node server sending the synchronous data, and acquire the identity of the new master node server.
When the master node server determines that other master node servers exist in the distributed data system, the master node server sends a data comparison request to the other master node servers according to the acquired identity of the other master node servers, wherein the data comparison request is used for acquiring the first largest target data number in the data numbers of the other master node servers. The master node server receives first target data numbers sent by other master node servers, compares the first target data numbers with the largest second target data numbers in the own data numbers, and when the first target data numbers exceed the own second target data numbers, the master node server becomes a slave node server and acquires data missing from the master node server from the other master node servers. For example, when the first target data is numbered TxID5 and the second target data is numbered TxID4, the original master node server becomes a slave node server and acquires the missing data of the original master node server from the new master node server.
Alternatively, the process of acquiring missing data by the master node server is similar to that of acquiring missing data by the slave node server, when the master node server determines that there is a data missing by itself, the master node server becomes the slave node server at this time, and thus, the slave node server acquires missing data from another master node server.
As can be seen from the foregoing, in this embodiment, after a slave node server receives a data write request sent by a terminal, the data write request carries write data, where the data write request is used to write the write data into a node server of a distributed data system, the slave node server obtains, according to the data write request, a data number of the write data from a master node server, where the data number is a sequential identifier of the write data in the distributed data system, the slave node server sends the write data and the data number as synchronous data to other node servers of the distributed data system, and receives feedback information sent by the other node servers, and determines, according to the feedback information, a first node number of the other node servers that successfully receives the synchronous data, where the slave node server stores the write data based on the first node number; the scheme can adopt the slave node servers to receive the written data, adopts the master node server to number the written data, and uses the slave node servers to synchronize the data numbers and the written data as synchronous data to other node servers of the distributed data system, adopts a plurality of node servers to receive the written data, and uses the master node server to number, so that the plurality of node servers can receive more written data, and can ensure the orderly synchronization of the written data, thereby greatly improving the efficiency of data storage.
In order to better implement the method, the embodiment of the invention also provides a data storage device, which comprises a master node and a slave node, and can be integrated in electronic equipment, such as a server and the like, and can be a single server or a server cluster formed by a plurality of servers.
For example, as shown in fig. 11, the data storage device may include a receiving unit 301, an acquiring unit 302, a transmitting unit 303, a determining unit 304, and a storing unit 305, as follows:
(1) A receiving unit 301;
the receiving unit 301 is configured to receive, from a node, a data write request sent by a terminal, where the data write request carries write data, and the data write request is used to write the write data to the node of the distributed data system.
For example, the receiving unit 301 may be specifically configured to send, by the terminal, a data write request to a slave node of the distributed data system directly with an address of the node, where the data write request is received by the slave node. The terminal can also store the writing data carried in the writing data request to the third party database, and send the storage address of the writing data stored in the third party database to the slave node in the writing data request, after receiving the data writing request from the slave node, the terminal obtains the storage address of the writing data stored in the third party database in the data writing request, and according to the storage address, the terminal obtains the writing data in the third party database.
(2) An acquisition unit 302;
an obtaining unit 302, configured to obtain, from the master node, a data number of the write data according to the data write request by the slave node.
The acquiring unit 302 may include a generating subunit 3021, a transmitting subunit 3022, a receiving subunit 3023, and a returning subunit 3024, as shown in fig. 12, specifically as follows:
a generating subunit 3021, configured to generate a data synchronization request from a node according to the data write request;
a transmitting subunit 3022, configured to transmit a data synchronization request to the master node from the node;
a receiving subunit 3023, configured to receive the data number from the slave node when the master node returns the data number;
and the return subunit 3024 is configured to return, when the master node does not return the data number, a prompt message indicating that the data writing has failed to the terminal from the slave node.
For example, according to the data write request, the slave node of the generating subunit 3021 generates a data synchronization request, the slave node of the transmitting subunit 3022 transmits the data synchronization request to the master node, and when the master node returns the data number, the slave node of the receiving subunit 3023 receives the data number, and when the master node does not return the data number, the slave node of the returning subunit 3024 returns the notification information of the data write failure to the terminal.
(3) A transmission unit 303;
and a transmitting unit 303, configured to transmit the write data and the data number as synchronous data from the node to other nodes of the distributed data system, and receive feedback information transmitted by the other nodes.
For example, the sending unit 303 may be specifically configured to send the write data and the data number to other nodes of the distributed data system as the synchronization data, where when the other nodes successfully receive the synchronization data, the other nodes send feedback information that the other nodes successfully receive the synchronization data to the slave nodes, and when the other nodes do not successfully receive the synchronization data, the other nodes do not return the feedback information, and the slave nodes receive the feedback information sent by the other nodes that successfully receive the synchronization data.
(4) A determination unit 304;
the determining unit 304 is configured to determine, according to the feedback information, the first node number of other nodes that successfully receive the synchronization data.
For example, the determining unit 304 may be specifically configured to determine that the other node successfully receives the synchronization data when receiving feedback information sent by the other node, so that only the amount of received feedback information needs to be analyzed to determine the first node amount of the other node successfully receiving the synchronization data.
(5) A storage unit 305;
the storage unit 305 is configured to store the write data by the slave node based on the first node number.
For example, the storage unit 305 may specifically be configured to compare, by the slave node, a first node number of other nodes that successfully receive the synchronization data with half of the total number of nodes of the distributed data system, and when the first node number exceeds half of the total number of nodes of the distributed data system, write the synchronization data to a local database of the slave node, and send a data save request to the other nodes to save the synchronization data to the local database of the other nodes; when the number of the first nodes does not exceed half of the total number of the nodes, the slave nodes discard the written data and send data discarding requests to other nodes so as to discard the synchronous data received by the other nodes.
Optionally, the data storage device may further include a writing unit 306, as shown in fig. 13, specifically as follows:
and a writing unit 306, configured to receive, by the master node, a data writing request sent by the terminal, generate, by the master node, a data number of the writing data according to the data writing request, send, by the master node, the writing data and the data number as synchronous data to the slave nodes of the distributed data system, receive feedback information sent by the slave nodes, determine, by the master node, a second number of slave nodes that successfully receive the synchronous data according to the feedback information, store, by the master node, the writing data to a local database of the master node when the second number of slave nodes exceeds half of the total number of nodes of the distributed data system, and send, by the master node, a data saving request to the slave nodes to save the synchronous data to the local database of the slave nodes, and discard, by the master node, the writing data when the second number of slave nodes does not exceed half of the total number of nodes of the distributed data system, and send, by the slave nodes a data discarding request to discard the synchronous data.
In the implementation, each unit may be implemented as an independent entity, or may be implemented as the same entity or several entities in any combination, and the implementation of each unit may be referred to the foregoing method embodiment, which is not described herein again.
As can be seen from the foregoing, in this embodiment, after the slave node of the receiving unit 301 receives the data writing request sent by the terminal, the data writing request carries writing data, the data writing request is used for writing the writing data into the nodes of the distributed data system, the slave node of the obtaining unit 302 obtains, according to the data writing request, the data number of the writing data from the master node, where the data number is the sequential identifier of the writing data in the distributed data system, the slave node of the sending unit 303 sends the writing data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information sent by the other nodes, the slave node of the determining unit 304 determines, according to the feedback information, the first node number of the other nodes that successfully receives the synchronous data, and the slave node of the storing unit 305 stores the writing data based on the first node number; according to the scheme, the slave nodes can be used for receiving the written data and numbering the written data, the master nodes are used for synchronizing the data numbers and the written data as synchronous data to other nodes of the distributed data system, a plurality of nodes are used for receiving the written data and operating the master nodes for numbering, so that a plurality of nodes can be used for receiving more written data, and orderly synchronization of the written data can be ensured, and therefore, the efficiency of data storage can be greatly improved.
The embodiment of the invention also provides an electronic device, as shown in fig. 14, which shows a schematic structural diagram of the electronic device according to the embodiment of the invention, specifically:
the electronic device may include one or more processing cores 'processors 401, one or more computer-readable storage media's memory 402, power supply 403, and input unit 404, among other components. It will be appreciated by those skilled in the art that the electronic device structure shown in fig. 14 is not limiting of the electronic device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components. Wherein:
the processor 401 is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the memory 402, and calling data stored in the memory 402, thereby performing overall detection of the electronic device. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, an application program, etc., and the modem processor mainly processes wireless communication. It will be appreciated that the modem processor described above may not be integrated into the processor 401.
The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by executing the software programs and modules stored in the memory 402. The memory 402 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data created according to the use of the electronic device, etc. In addition, memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.
The electronic device further comprises a power supply 403 for supplying power to the various components, preferably the power supply 403 may be logically connected to the processor 401 by a power management system, so that functions of managing charging, discharging, and power consumption are performed by the power management system. The power supply 403 may also include one or more of any of a direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.
The electronic device may further comprise an input unit 404, which input unit 404 may be used for receiving input digital or character information and generating keyboard, mouse, joystick, optical or trackball signal inputs in connection with user settings and function control.
Although not shown, the electronic device may further include a display unit or the like, which is not described herein. In particular, in this embodiment, the processor 401 in the electronic device loads executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 executes the application programs stored in the memory 402, so as to implement various functions as follows:
after receiving a data writing request sent by a terminal by a slave node, the data writing request carries writing data, the data writing request is used for writing the writing data into nodes of a distributed data system, the slave node obtains a data number of the writing data from a master node according to the data writing request, the data number is a sequential identification of the writing data in the distributed data system, the slave node sends the writing data and the data number as synchronous data to other nodes of the distributed data system, receives feedback information sent by the other nodes, determines the first node number of the other nodes successfully receiving the synchronous data by the slave node according to the feedback information, and stores the writing data by the slave node based on the first node number
For example, the slave node receives a data writing request sent by the terminal, generates a data synchronization request according to the data writing request, sends the data synchronization request to the master node, receives the data number when the master node returns the data number, and returns prompt information of data writing failure to the terminal when the master node does not return the data number. When the slave node receives the data number, the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system, receives feedback information sent by the other nodes, determines the first node number of the other nodes successfully receiving the synchronous data according to the feedback information, writes the synchronous data into a local database of the slave node when the first node number exceeds half of the total number of the nodes, and sends a data saving request to the other nodes so as to save the synchronous data to the local database of the other nodes, and when the first node number does not exceed half of the total number of the nodes, the slave node discards the written data and sends a data discarding request to the other nodes so as to discard the synchronous data received by the other nodes.
The specific implementation of each operation may be referred to the previous embodiments, and will not be described herein.
As can be seen from the foregoing, in the embodiment of the present invention, after a slave node receives a data write request sent by a terminal, the data write request carries write data, where the data write request is used to write the write data into a node of a distributed data system, a slave node obtains, according to the data write request, a data number of the write data from a master node, where the data number is a sequential identifier of the write data in the distributed data system, the slave node sends the write data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information sent by the other nodes, and the slave node determines, according to the feedback information, a first node number of the other nodes that successfully receives the synchronous data, and stores the write data based on the first node number; according to the scheme, the slave nodes can be used for receiving the written data and numbering the written data, the master nodes are used for synchronizing the data numbers and the written data as synchronous data to other nodes of the distributed data system, a plurality of nodes are used for receiving the written data and operating the master nodes for numbering, so that a plurality of nodes can be used for receiving more written data, and orderly synchronization of the written data can be ensured, and therefore, the efficiency of data storage can be greatly improved.
Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.
To this end, embodiments of the present invention provide a computer readable storage medium having stored therein a plurality of instructions capable of being loaded by a processor to perform the steps of any of the data storage methods provided by the embodiments of the present invention. For example, the instructions may perform the steps of:
after receiving a data writing request sent by a terminal by a slave node, the data writing request carries writing data, the data writing request is used for writing the writing data into nodes of a distributed data system, the slave node obtains a data number of the writing data from a master node according to the data writing request, the data number is a sequential identification of the writing data in the distributed data system, the slave node sends the writing data and the data number as synchronous data to other nodes of the distributed data system, receives feedback information sent by the other nodes, determines the first node number of the other nodes successfully receiving the synchronous data by the slave node according to the feedback information, and stores the writing data by the slave node based on the first node number
For example, the slave node receives a data writing request sent by the terminal, generates a data synchronization request according to the data writing request, sends the data synchronization request to the master node, receives the data number when the master node returns the data number, and returns prompt information of data writing failure to the terminal when the master node does not return the data number. When the slave node receives the data number, the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system, receives feedback information sent by the other nodes, determines the first node number of the other nodes successfully receiving the synchronous data according to the feedback information, writes the synchronous data into a local database of the slave node when the first node number exceeds half of the total number of the nodes, and sends a data saving request to the other nodes so as to save the synchronous data to the local database of the other nodes, and when the first node number does not exceed half of the total number of the nodes, the slave node discards the written data and sends a data discarding request to the other nodes so as to discard the synchronous data received by the other nodes.
The specific implementation of each operation above may be referred to the previous embodiments, and will not be described herein.
Wherein the computer-readable storage medium may comprise: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.
Because the instructions stored in the computer readable storage medium may execute the steps in any data storage method provided by the embodiments of the present invention, the beneficial effects that any data storage method provided by the embodiments of the present invention can be achieved, which are detailed in the previous embodiments and are not described herein.
The foregoing has outlined some of the more detailed description of the method, apparatus and computer readable storage medium of the present invention for the purposes of providing a detailed description of the principles and embodiments of the present invention, wherein the detailed description is provided for the purpose of providing an improved understanding of the method and core concepts of the present invention; meanwhile, as those skilled in the art will vary in the specific embodiments and application scope according to the ideas of the present invention, the present description should not be construed as limiting the present invention in summary.

Claims (17)

1. A data storage method for data write operations of a distributed data system, wherein the distributed data system comprises a master node and at least two slave nodes, comprising:
the slave node receives a data writing request sent by a terminal, wherein the data writing request carries writing data and is used for writing the writing data into nodes of the distributed data system;
the slave node acquires the data number of the written data from the master node according to the data writing request, wherein the data number is the sequential identification of the written data in the distributed data system;
the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system and receives feedback information sent by the other nodes;
the slave node determines the first node quantity of other nodes which successfully receive the synchronous data according to the feedback information;
the slave node stores the write data based on the first node number.
2. The data storage method according to claim 1, wherein the slave node stores the write data based on the first node number, comprising:
When the number of the first nodes exceeds half of the total number of the nodes, the slave node writes the synchronous data into a local database of the slave node and sends a data saving request to the other nodes so as to save the synchronous data to the local databases of the other nodes;
when the number of the first nodes does not exceed half of the total number of nodes, the slave node discards the writing data and sends a data discarding request to the other nodes so as to discard the synchronous data received by the other nodes.
3. The data storage method according to claim 1, wherein the slave node acquires the data number of the write data from the master node according to the data write request, comprising:
generating a data synchronization request by the slave node according to the data writing request;
the slave node sends the data synchronization request to the master node;
when the master node returns the data number, the slave node receives the data number;
and when the master node does not return the data number, the slave node returns prompt information of data writing failure to the terminal.
4. A data storage method according to claim 3, wherein the slave node before sending the data synchronization request to the master node, further comprises:
the slave node sends a test request to the master node, wherein the test request is used for testing the state of the master node;
the slave node receives feedback information of the test request and determines whether the master node is in a normal state according to the feedback information;
when the feedback information is that the master node is in an abnormal state, the slave node determines that the master node is in an abnormal state, and the slave node triggers a new master node determination flow;
the slave node sending the data synchronization request to the master node, comprising: and when the master node is in a normal state, the slave node sends the synchronous data request to the master node.
5. The data storage method of claim 4, wherein the new master node determination procedure comprises:
the slave node sends voting requests for selecting a new master node to other slave nodes;
the slave node receives the voting results of the other slave nodes aiming at the voting request reply, wherein the voting results comprise that the slave node which agrees to issue the voting request is a new master node and the slave node which does not agree to issue the voting request is a new master node;
When the voting result is that the number of slave nodes which agree to issue a voting request is half of the total number of the slave nodes, the slave nodes are selected as the new master nodes.
6. The data storage method according to claim 5, wherein the voting request carries a first data number having a largest data number of the slave node, and the slave node further includes, after transmitting the voting request for selecting a new master node to the other slave nodes:
the other slave nodes receive the voting requests sent by the slave nodes;
the other slave nodes compare the first data number with the second data number with the largest current value of the other slave nodes;
when the second data number does not exceed the first data number, the other slave nodes send voting results which agree to send voting requests to the slave nodes and are taken as new master nodes by the slave nodes;
and when the second data number exceeds the first data number, the other slave nodes send voting results of the slave nodes which do not agree to send voting requests to the slave nodes as new master nodes.
7. The data storage method of claim 6, further comprising:
When the second data number does not exceed the first data number and the other slave nodes receive at least two voting requests within a preset time interval, the other slave nodes send voting results of the slave nodes which agree to send the voting requests to the slave nodes which send the voting requests for the first time as new master nodes.
8. The data storage method of claim 1, further comprising:
the slave node sends a data detection request to the master node, wherein the data detection request is used for acquiring the largest target data number in the data numbers of the master node;
the slave node receives the target data number sent by the master node;
when the target data number exceeds the largest current data number in the data numbers of the slave nodes, the slave nodes determine that the own data is missing, and acquire the missing data from the master nodes.
9. The data storage method according to claim 8, wherein the slave node determining that there is a missing of its own data and acquiring the missing data from the master node includes:
the slave node determines that own data is missing;
the slave node sends a first data acquisition request to the master node, wherein the first data acquisition request carries the current data number of the slave node;
And the slave node receives the data missing from the slave node, which is sent by the master node according to the current data number.
10. The data storage method of claim 1, wherein when the distributed data system includes at least two master nodes, further comprising:
when the master node receives a data writing request, the master node sends the synchronous data to the other nodes;
when other master nodes in other nodes receive the synchronous data, the master node determines the identity of the other master nodes and acquires the identity of the other master nodes.
11. The data storage method of claim 10, further comprising:
when the master node receives the synchronous data sent by the other master nodes, the master node determines the identity of the other master nodes and acquires the identity of the other master nodes.
12. The data storage method according to claim 10 or 11, wherein after the master node determines identities of other master nodes and obtains identities of the other master nodes, the method further comprises:
the master node sends a data comparison request to the other master nodes according to the identity of the other master nodes, wherein the data comparison request is used for acquiring a first target data number with the largest data number of the other master nodes;
The master node receives a first target data number sent by other master nodes;
when the first target data number exceeds the second target data number which is the largest in the data numbers of the master nodes, the master node becomes a slave node, and the data missing from the master node is acquired from the other master nodes.
13. The data storage method according to claim 12, wherein the master node obtaining data missing from the other master nodes includes:
the master node determines that data loss exists in the master node;
the master node sends a second data acquisition request to the other master nodes, wherein the second data acquisition request carries a second target data number of the master node;
and the master node receives the data missing from the master node, which is sent by the other master nodes according to the second target data number.
14. The data storage method of claim 1, further comprising:
the master node receives a data writing request sent by the terminal;
the master node generates a data number of the written data according to the data writing request;
the master node sends the written data and the data number as synchronous data to slave nodes of the distributed data system, and receives feedback information sent by the slave nodes;
The master node determines the number of second nodes of slave nodes which successfully receive the synchronous data according to the feedback information;
when the number of the second nodes exceeds half of the total number of nodes of the distributed data system, the master node stores the synchronous data into a local database of the master node and sends a data storage request to the slave node so as to store the synchronous data into the local database of the slave node;
when the number of the second nodes does not exceed half of the total number of nodes of the distributed data system, the master node discards the write data and sends the data discarding request to the slave node so that the slave node discards the synchronous data.
15. A data storage device, comprising:
the receiving unit is used for receiving a data writing request sent by the terminal from the node, wherein the data writing request carries writing data, and the data writing request is used for writing the writing data into the nodes of the distributed data system;
the obtaining unit is used for obtaining the data number of the writing data from the master node according to the data writing request by the slave node, wherein the data number is the sequence identifier of the writing data in the distributed data system;
The sending unit is used for sending the writing data and the data number as synchronous data to other nodes of the distributed data system by the slave node and receiving feedback information sent by the other nodes;
the determining unit is used for determining the first node number of other nodes which successfully receive the synchronous data according to the feedback information by the slave node;
and the storage unit is used for storing the write data by the slave node based on the first node number.
16. An electronic device comprising a processor and a memory, the memory storing an application, the processor being configured to run the application in the memory to perform the steps in the data storage method of any one of claims 1 to 14.
17. A computer readable storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the steps in the data storage method of any one of claims 1 to 14.
CN202010004352.XA 2020-01-03 2020-01-03 Data storage method, device and computer readable storage medium Active CN111198662B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010004352.XA CN111198662B (en) 2020-01-03 2020-01-03 Data storage method, device and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010004352.XA CN111198662B (en) 2020-01-03 2020-01-03 Data storage method, device and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN111198662A CN111198662A (en) 2020-05-26
CN111198662B true CN111198662B (en) 2023-07-14

Family

ID=70746679

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010004352.XA Active CN111198662B (en) 2020-01-03 2020-01-03 Data storage method, device and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN111198662B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111625601B (en) * 2020-05-27 2023-03-14 北京火山引擎科技有限公司 Data writing method and device, computer equipment and storage medium
CN112068997B (en) * 2020-09-09 2023-12-19 恒生电子股份有限公司 Data backup method, device, equipment and storage medium
CN112527901A (en) * 2020-12-10 2021-03-19 杭州比智科技有限公司 Data storage system, method, computing device and computer storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104933132A (en) * 2015-06-12 2015-09-23 广州巨杉软件开发有限公司 Distributed database weighted voting method based on operating sequence number
CN107295080A (en) * 2017-06-19 2017-10-24 北京百度网讯科技有限公司 Date storage method and server applied to distributed server cluster
JP2017224253A (en) * 2016-06-17 2017-12-21 富士通株式会社 Parallel processor and memory cache control method

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016018262A1 (en) * 2014-07-29 2016-02-04 Hewlett-Packard Development Company, L.P. Storage transactions
CN107092437B (en) * 2016-02-17 2019-11-22 杭州海康威视数字技术股份有限公司 Data write-in, read method and device, cloud storage system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104933132A (en) * 2015-06-12 2015-09-23 广州巨杉软件开发有限公司 Distributed database weighted voting method based on operating sequence number
JP2017224253A (en) * 2016-06-17 2017-12-21 富士通株式会社 Parallel processor and memory cache control method
CN107295080A (en) * 2017-06-19 2017-10-24 北京百度网讯科技有限公司 Date storage method and server applied to distributed server cluster

Also Published As

Publication number Publication date
CN111198662A (en) 2020-05-26

Similar Documents

Publication Publication Date Title
US11360854B2 (en) Storage cluster configuration change method, storage cluster, and computer system
CN111198662B (en) Data storage method, device and computer readable storage medium
CN113014634B (en) Cluster election processing method, device, equipment and storage medium
US10331625B2 (en) Managing sequential data store
WO2018107772A1 (en) Method, device and apparatus for processing write request
US20080288812A1 (en) Cluster system and an error recovery method thereof
CN106960060B (en) Database cluster management method and device
CN111541762B (en) Data processing method, management server, device and storage medium
CN107231435B (en) Data synchronization monitoring method and system
CN110716793A (en) Execution method, device, equipment and storage medium of distributed transaction
US20200403850A1 (en) Facilitating resilient and fault tolerant asynchronous messaging
CN114265753A (en) Management method and management system of message queue and electronic equipment
CN105323271B (en) Cloud computing system and processing method and device thereof
CN113190620B (en) Method, device, equipment and storage medium for synchronizing data between Redis clusters
CN112000444B (en) Database transaction processing method and device, storage medium and electronic equipment
CN114625566A (en) Data disaster tolerance method and device, electronic equipment and storage medium
US10169440B2 (en) Synchronous data replication in a content management system
CN111092956A (en) Resource synchronization method, device, storage medium and equipment
CN116540938A (en) Data reading method, device, distributed storage system, equipment and storage medium
CN114564458B (en) Method, device, equipment and storage medium for synchronizing data among clusters
CN113342893B (en) Node synchronization method and device based on block chain, storage medium and server
US20090106781A1 (en) Remote call handling methods and systems
CN112671636B (en) Group message pushing method and device, computer equipment and storage medium
US11381642B2 (en) Distributed storage system suitable for sensor data
CN115145782A (en) Server switching method, mooseFS system and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20201010

Address after: Room 2012, building 12, 13, 15, fangmaoyuan (phase II), 1177 Huanhu Road, Tianding street, Yuelu District, Changsha City, Hunan Province

Applicant after: Tencent cloud computing (Changsha) Co.,Ltd.

Address before: 518057 Nanshan District science and technology zone, Guangdong, Zhejiang Province, science and technology in the Tencent Building on the 1st floor of the 35 layer

Applicant before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd.

GR01 Patent grant
GR01 Patent grant