CN111198662A - Data storage method and device and computer readable storage medium - Google Patents

Data storage method and device and computer readable storage medium Download PDF

Info

Publication number
CN111198662A
CN111198662A CN202010004352.XA CN202010004352A CN111198662A CN 111198662 A CN111198662 A CN 111198662A CN 202010004352 A CN202010004352 A CN 202010004352A CN 111198662 A CN111198662 A CN 111198662A
Authority
CN
China
Prior art keywords
data
node
nodes
slave
request
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010004352.XA
Other languages
Chinese (zh)
Other versions
CN111198662B (en
Inventor
曹智颖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Cloud Computing Changsha Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202010004352.XA priority Critical patent/CN111198662B/en
Publication of CN111198662A publication Critical patent/CN111198662A/en
Application granted granted Critical
Publication of CN111198662B publication Critical patent/CN111198662B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention discloses a data storage method, a data storage device and a computer readable storage medium; according to the embodiment of the invention, after a slave node receives a data write-in request sent by a terminal, the data write-in request carries write-in data, the slave node obtains a data number of the write-in data from a master node according to the data write-in request, the slave node sends the write-in data and the data number to other nodes of a distributed data system as synchronous data and receives feedback information sent by the other nodes, the slave node determines the number of first nodes of the other nodes which successfully receive the synchronous data according to the feedback information, and the slave node stores the write-in data based on the number of the first nodes; the scheme can greatly improve the efficiency of data storage.

Description

Data storage method and device and computer readable storage medium
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data storage method, an apparatus, and a computer-readable storage medium.
Background
In recent years, with the rapid development of big data technology, a single database or service node cannot support the growth of services and data, so that distributed data systems are widely used, and most of existing distributed data systems adopt a single-master-class data processing mode, wherein a master node exists in the distributed data system and is responsible for writing data and synchronizing the data to other nodes.
In the research and practice process of the prior art, the inventor of the present invention finds that only the main node is allowed to process data writing for a single main type data processing mode, and when more data needs to be processed, the main node may be overstocked with data, and thus, the efficiency of data storage may be greatly reduced.
Disclosure of Invention
The embodiment of the invention provides a data storage method, a data storage device and a computer readable storage medium. The efficiency of data storage can be improved.
A method of data storage, comprising:
the slave node receives a data writing request sent by a terminal, wherein the data writing request carries writing data, and the data writing request is used for writing the writing data into the nodes of the distributed data system;
the slave node acquires a data number of the written data from the master node according to the data writing request, wherein the data number is a sequential identifier of the written data in the distributed data system;
the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information sent by the other nodes;
the slave node determines the number of first nodes of other nodes which successfully receive the synchronous data according to the feedback information;
the slave node stores the write data based on the first number of nodes.
Accordingly, an embodiment of the present invention provides a data storage device, including:
a receiving unit, configured to receive, by the slave node, a data write request sent by a terminal, where the data write request carries write data, and the data write request is used to write the write data into a node of the distributed data system;
an obtaining unit, configured to obtain, by the slave node, a data number of the write data from the master node according to the data write request, where the data number is a sequential identifier of the write data in the distributed data system;
the sending unit is used for sending the written data and the data number as synchronous data to other nodes of the distributed data system by the slave node and receiving feedback information sent by the other nodes;
a determining unit, configured to determine, by the slave node, a first node number of other nodes that successfully receive the synchronization data according to the feedback information;
and the storage unit is used for storing the write data by the slave node based on the first node number.
Optionally, in some embodiments, the storage unit may be specifically configured to, when the number of the first nodes exceeds half of the number of all nodes, the slave node writes the synchronization data into a local database of the slave node, and sends a data saving request to the other node, so as to save the synchronization data to the local database of the other node; when the number of the first nodes does not exceed half of the number of all the nodes, the slave nodes discard the write data and send data discarding requests to the other nodes to discard the synchronous data received by the other nodes.
Optionally, in some embodiments, the obtaining unit may be specifically configured to generate, by the slave node, a data synchronization request according to the data write request; the slave node sends the data synchronization request to the master node; when the master node returns the data number, the slave node receives the data number; and when the master node does not return the data number, the slave node returns prompt information of data writing failure to the terminal.
Optionally, in some embodiments, the obtaining unit may be specifically configured to send a test request to the master node by the slave node, where the test request is used to test a state of the master node; the slave node receives feedback information of the test request and determines whether the master node is in a normal state according to the feedback information; when the feedback information indicates that the master node is in an abnormal state, the slave node determines that the master node is in the abnormal state, and the slave node triggers a new master node determination process; the slave node sends the data synchronization request to the master node, and the data synchronization request comprises: and when the main node is in a normal state, the slave node sends the synchronous data request to the main node.
Optionally, in some embodiments, the obtaining unit may be specifically configured to send, by the slave node, a voting request for selecting a new master node to another slave node; the slave node receives the voting results of the responses of the other slave nodes to the voting requests, wherein the voting results comprise that the slave node which sends out the voting requests is a new master node and the slave node which does not send out the voting requests is a new master node; and when the voting result is that the number of the slave nodes which agree to send out the voting request to be the new master node exceeds half of the total number of the slave nodes, selecting the slave nodes as the new master nodes.
Optionally, in some embodiments, the obtaining unit may be specifically configured to receive, by the other slave node, the voting request sent by the slave node; the other slave nodes compare the first data number with the current maximum second data number of the other slave nodes; when the second data number does not exceed the first data number, the other slave nodes send voting results which enable the slave node sending the voting request to be a new master node to the slave nodes; and when the second data number exceeds the first data number, the other slave nodes send voting results which do not agree that the slave node sending the voting request is a new master node to the slave nodes.
Optionally, in some embodiments, the obtaining unit may be specifically configured to, when the second data number does not exceed the first data number, and the other slave node receives at least two voting requests within a preset time interval, send, to the slave node that first sends the voting request, a voting result that the slave node that issued the voting request agrees to be a new master node.
Optionally, in some embodiments, the data storage apparatus may further include a first failure recovery unit, which is applied to a slave node, and specifically, may be used to send a data detection request to the master node by the slave node, where the data detection request is used to obtain a maximum target data number in data numbers of the master node; the slave node receives the target data number sent by the master node; and when the target data number exceeds the maximum current data number in the data numbers of the slave nodes, the slave nodes determine that own data is missing, and acquire the missing data of the slave nodes from the master node.
Optionally, in some embodiments, the first failure unit may be specifically configured to determine that data of the slave node is missing; the slave node sends a first data acquisition request to the master node, wherein the first data acquisition request carries the current data number of the slave node; and the slave node receives the data which is sent by the master node according to the current data number and is missing from the slave node.
Optionally, in some embodiments, the data storage device may further include a second failure recovery unit, which is applied to a master node, and specifically may be used for the master node to receive the data acquisition request; the main node compares the current data number in the data acquisition request with the target number; the master node determines the data missing from the slave node according to the comparison result; and the master node transmits the data missing from the slave node to the slave node.
Optionally, in some embodiments, the second failure recovery unit may be specifically configured to, when the master node receives a data write request, send the synchronization data to the other node by the master node; when other main nodes in other nodes receive the synchronous data, the main node determines the identities of the other main nodes and acquires the identity identifications of the other main nodes.
Optionally, in some embodiments, the second failure recovery unit may be specifically configured to, when the master node receives the synchronization data sent by the other master node, determine the identity of the other master node, and acquire the identity of the other master node.
Optionally, in some embodiments, the second failure recovery unit may be specifically configured to send, by the master node, the data comparison request to the other master nodes according to the identity identifiers of the other master nodes, where the data comparison request is used to obtain a first target data number of the other master nodes, where the data number is the largest; the main node receives first target data numbers sent by other main nodes; and when the first target data number exceeds a second target data number which is the largest in the data numbers of the main nodes, the main nodes become slave nodes, and the data which are missing from the main nodes are obtained from other main nodes.
Optionally, in some embodiments, the second failure recovery unit may be specifically configured to determine that there is data missing in the master node itself; the main node sends a second data acquisition request to the other main nodes, wherein the second data acquisition request carries a second target data number of the main node; and the master node receives the data which are lost by the master node and are sent by the other master nodes according to the second target data number.
Optionally, in some embodiments, the data storage device may further include a writing unit, where the writing unit is specifically applied to data writing of a master node, and the writing unit may be specifically used for the master node to receive a data writing request sent by the terminal; the main node generates a data number of the written data according to the data writing request; the master node sends the written data and the data number to slave nodes of the distributed data system as synchronous data and receives feedback information sent by the slave nodes; the master node determines the number of second nodes of the slave nodes which successfully receive the synchronous data according to the feedback information; when the number of the second nodes exceeds half of the total number of the nodes of the distributed data system, the master node stores the write-in data to a local database of the master node and sends the data storage request to the slave node so as to store the synchronous data to the local database of the slave node; when the second number of nodes does not exceed half of the total number of nodes of the distributed data system, the master node discards the write data and sends the data discard request to the slave node, so that the slave node discards the synchronization data.
In addition, an electronic device is further provided in an embodiment of the present invention, and includes a processor and a memory, where the memory stores an application program, and the processor is configured to run the application program in the memory to implement the data storage method provided in the embodiment of the present invention.
In addition, the embodiment of the present invention further provides a computer-readable storage medium, where a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to perform the steps in any one of the data storage methods provided by the embodiment of the present invention.
According to the embodiment of the invention, after a slave node receives a data write-in request sent by a terminal, the data write-in request carries write-in data, the data write-in request is used for writing the write-in data into nodes of a distributed data system, the slave node obtains a data number of the write-in data from a master node according to the data write-in request, the data number is a sequential identifier of the write-in data in the distributed data system, the slave node sends the write-in data and the data number as synchronous data to other nodes of the distributed data system and receives feedback information sent by the other nodes, the slave node determines the number of first nodes of the other nodes which successfully receive the synchronous data according to the feedback information, and the slave node stores the write-in data based on the number of the first nodes; according to the scheme, the slave nodes can receive the written data, the master nodes are used for numbering the written data, the slave nodes synchronize the data numbers and the written data to other nodes of the distributed data system as synchronous data, the plurality of nodes are used for receiving the written data, the master nodes are operated for numbering, a plurality of nodes can receive more written data, ordered synchronization of the written data can be guaranteed, and therefore the data storage efficiency can be greatly improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram of a data storage method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a data storage method according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of acquiring a data number from a node according to an embodiment of the present invention;
fig. 4 is a schematic flowchart of testing the state of a master node according to an embodiment of the present invention;
fig. 5 is a schematic flowchart of determining a new master node according to an embodiment of the present invention;
FIG. 6 is a flow chart illustrating write data synchronization according to an embodiment of the present invention;
FIG. 7 is a flowchart illustrating a single-node failure recovery according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a cluster of an original master node in a first child node in a split brain fault according to an embodiment of the present invention;
fig. 9 is a schematic diagram of a cluster of an original master node in a second child node in a split brain fault according to an embodiment of the present invention;
FIG. 10 is a schematic flow chart of data storage provided by the embodiment of the invention;
FIG. 11 is a schematic structural diagram of a data storage device according to an embodiment of the present invention;
FIG. 12 is a schematic structural diagram of an obtaining unit of a data storage device according to an embodiment of the present invention;
FIG. 13 is a schematic diagram of another structure of a data storage device according to an embodiment of the present invention;
fig. 14 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides a data processing method, a data processing device and a computer readable storage medium. The data processing apparatus may be integrated in an electronic device, where the electronic device may be a server or a terminal, and the data processing apparatus is mainly used for data writing operation of a distributed data system, may be a data writing operation apparatus of a master node and a slave node in the distributed data system, may be a node cluster, and may include the master node and the slave node in the node cluster.
For example, referring to fig. 1, taking data processing apparatus integrated in an electronic device and being used for data writing of a slave node in a distributed data system as an example, the slave node receives a data writing request sent by a terminal, where the data writing request carries written data, the slave node obtains a data number of the written data from a master node according to the data writing request, the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system and receives feedback information sent by the other nodes, the slave node determines, according to the feedback information, the number of nodes of the other nodes that successfully receive the synchronous data, and the slave node stores the written data based on the number of nodes.
Optionally, when the write-in data is written from the master node of the distributed data system, the master node receives a data write-in request sent by the terminal, the master node generates a data number of the write-in data according to the data write-in request, the master node sends the write-in data and the data number as synchronous data to slave nodes of the distributed data system and receives feedback information sent by the slave nodes, the master node determines the number of the slave nodes that successfully receive the synchronous data according to the feedback information, and the master node stores the write-in data based on the number of the slave nodes.
The following are detailed below. It should be noted that the following description of the embodiments is not intended to limit the preferred order of the embodiments.
This embodiment will be described from the perspective of a data storage device, where the data storage device may be specifically integrated in an electronic device, and the electronic device may be a server or a terminal; the terminal may include a tablet Computer, a notebook Computer, a Personal Computer (PC), and other devices.
A data storage method is applied to a slave node of a distributed data system and comprises the following steps:
the method comprises the steps that a slave node receives a data writing request sent by a terminal, the data writing request carries writing data, the slave node obtains a data number of the writing data from a master node according to the data writing request, the slave node sends the writing data and the data number to other nodes of a distributed data system as synchronous data and receives feedback information sent by the other nodes, the slave node determines the number of the other nodes which successfully receive the synchronous data according to the feedback information, and the slave node stores the writing data based on the number of the nodes.
As shown in fig. 2, the specific flow of the data storage method is as follows:
101. and receiving a data write request sent by the terminal from the node.
The data write request carries write data, for example, data such as video, audio, image, and text may be carried, and the data write request is mainly used to write the write data into a node of the distributed data system.
For example, there are various ways for receiving a data write request sent by a terminal from a node, for example, the terminal may directly send the data write request to a slave node of the distributed data system at the address of the node, and the slave node receives the data write request. The terminal can also store the written data carried in the written data request to a third-party database, and send the storage address of the written data stored in the third-party database to the slave node in the written data request, after receiving the data writing request, the slave node obtains the storage address of the written data stored in the third-party database in the data writing request, obtains the written data in the third-party database according to the storage address, and can also send prompt information to the terminal after obtaining the written data, so as to prompt the terminal to successfully obtain the written data from the node.
102. And the slave node acquires the data number of the written data from the master node according to the data writing request.
For example, when a node of the distributed data system receives first write data, the master node may number the data as TxID1, when a node of the distributed data system receives second write data, the master node may number the data as TxID2, when a node of the distributed data system receives third write data, the master node may number the data as TxID3, and so on, and the data number mainly identifies the order of receiving the write data in the distributed data system.
For example, the process of the slave node obtaining the data number of the write data from the node according to the data write request may be as shown in fig. 3, specifically as follows:
and S1, generating a data synchronization request from the node according to the data writing request.
The data synchronization request is used for acquiring a data number corresponding to the written data from the main node.
For example, the slave node generates a data synchronization request according to the data write request, for example, after receiving the data write request from the slave node, the slave node generates the data synchronization request according to the received data write request, and adds information related to the write data, such as the name, the receiving time, the type, and/or the data size of the write data, to the data synchronization request.
And S2, the slave node sends the data synchronization request to the master node.
For example, the slave node acquires an address of a master node in the distributed data system, and according to the address, the slave node transmits a data synchronization request to the master node.
S3, when the master node returns the data number, the slave node receives the data number.
For example, after the master node successfully receives the data synchronization request, the master node numbers the written data according to the related information of the written data in the data synchronization request, generates a data number of the written data, sends the data number of the written data to the slave node, and the slave node receives the data number.
And S4, when the main node does not return the data number, the slave node returns the prompt information of data writing failure to the terminal.
For example, when the master node does not successfully receive the data synchronization request, or does not return the data number of the write data after receiving the data synchronization request, the slave node does not receive the number of the write data, and at this time, the slave node returns a notification message of data write failure to the terminal, and discards the write data.
Optionally, before the slave node sends the data synchronization request to the master node, the state of the master node may also be tested, when the master node is in a normal state, the data synchronization request is sent to the master node, and when the master node is in an abnormal state, a new master node needs to be reselected and selected in the distributed data system. Therefore, before the step "the slave node sends the data synchronization request to the master node", the data storage method may further include:
the method comprises the steps that a slave node sends a test request to a master node, the test request is used for testing the state of the master node, the slave node receives feedback information of the test request and determines whether the master node is in a normal state or not according to the feedback information, and when the master node is in the normal state, the slave node sends a synchronous data request to the master node.
For example, as shown in fig. 4, the slave node sends a test request to the master node, and after receiving the test request, the master node delays to send feedback information for a period of time (Td), so that the state of the master node can be determined according to the time when the feedback information is received, and when the feedback information sent by the master node is not received within the Td, the distributed data system can generate feedback information that the master node is in an abnormal state. When the master node is in a normal state, the master-slave relationship still exists in the distributed data system, and the slave node can send the data synchronization request to the master node.
The slave node tests whether the master node is normal, that is, whether the master-slave relationship between the slave node and the master node is normal, the test process may be to test before sending synchronous data to the master node after each data write-in request is received, a time interval may also be set, the slave node sends a test request every other time interval to test the state of the master node at this time, and the slave node sends a test request to the master node even if the data write-in request is not received at this time, for example, every 1 minute may be set.
Optionally, when the feedback information received by the slave node indicates that the master node is in the abnormal state, the slave node determines that the master node is in the abnormal state. When the master node is in an abnormal state, the slave node triggers a new master node determination process, for example, when the master node is in an abnormal state, the slave node triggers a process for determining a new master node in the distributed data system, the process is used for selecting the new master node in the distributed data system, and the candidate object of the new master node is the slave node triggering the process. For example, when a test request is sent to the master node, the master node does not receive feedback information within a preset time interval or the feedback information is not received by the master node, the slave node needs to send a test request again outside the preset time interval, if the feedback information of the test request is still abnormal or the master node does not receive the test request, it can be really determined that the master node is abnormal, if the feedback information of the second test request is normal, it is determined that the master node is still in a normal state, and when it is determined that the master node is in a normal state, a data synchronization request is continuously sent to the master node.
Optionally, when the slave node determines that the master node is in the abnormal state, a new master node needs to be determined in the distributed data system, at this time, the distributed data system enters a new master node determination process, and a working process of the slave node is as shown in fig. 5, which specifically includes the following steps:
the method comprises the steps that a slave node sends a voting request for selecting a new master node to other slave nodes in a distributed data system, wherein the voting request carries the maximum first data number in the data numbers of the slave nodes, the slave nodes receive voting results of other slave nodes responding to the voting request, and the voting results can have two conditions, namely that the slave node sending the voting request is approved to be the new master node, and the slave node not agreeing to send the voting request is not the new master node. When the slave nodes receive the voting results of the slave nodes which agree to send out the voting requests as the new master node, the slave nodes automatically select as the master node. After the master node is selected, the slave node becomes the master node, and the slave node can generate the data number of the write data and perform the next step of storing the write data. When the slave nodes receive the voting results of the slave nodes which agree to send out the voting requests as the new master node, the number of the voting results does not exceed half of the total number of the slave nodes of the distributed data system, at this time, the flow initiated by the slave nodes for selecting the new master node fails to select the master, and the slave nodes return prompt information of data writing failure to the terminal sending the data writing request.
Optionally, in the process of the main election process, after receiving the voting request, the other slave nodes compare the first data number of the slave node carried in the voting request with the largest second data number in their own data numbers. When the second data number does not exceed the first data number, the other slave nodes send the voting result of the slave node which agrees to send out the voting request as the new master node to the slave node, for example, when the first data number is TxID5, the second data number is TxID4, and the other slave nodes find that the second data number of the other slave nodes does not exceed the first data number of the slave node, then the other slave nodes send the voting result of the slave node which agrees to send out the voting request as the new master node to the slave node. When the largest second data number in the data numbers of the other slave nodes exceeds the first data number, the other slave nodes send the voting results that the slave node which does not agree to send the voting request is the new master node to the slave nodes, for example, when the second data number is TxID5 and the first data number is TxID4, the other slave nodes send the voting results that the slave node which does not agree to send the voting request is the new master node to the slave nodes.
Optionally, when the second data number does not exceed the first data number, and other slave nodes receive the voting requests sent by two different slave nodes within a preset time interval, in this case, the voting results need to be sent to the slave nodes corresponding to different voting requests according to the number of times of receiving the voting requests, and generally, the other slave nodes send, to the slave node sending the voting request, the voting result that the slave node sending the voting request agrees to be the new master node. For example, when the first data number of two different slave nodes is TxID5 and the second data number is TxID4, in a time interval, the other slave nodes send the voting result of the new master node agreeing to the slave node sending the voting request to itself for the first time, for example, within five minutes, the other slave nodes receive two voting requests, the two voting requests respectively correspond to the first slave node and the second slave node, but the voting request sent by the first slave node and received by the first slave node of other slave nodes, at the moment, the first data number of the first slave node exceeds the second data number of the first slave node, the other slave nodes send the first slave node the voting results agreeing that the first slave node is the new master node, although the first data number of the second slave node is also larger than the second data number of the second slave node, other slave nodes still send the voting result which does not agree that the second slave node is the new master node to the second slave node. It is assumed that the first slave node sends a voting request and the other slave nodes send voting results agreeing that the first slave node is a new master node. After the preset time interval is exceeded, for example, the preset time interval is ten minutes, but in the eleventh minute, the other slave nodes receive the voting request sent by the second slave node, and when the first data number of the second slave node exceeds the second data number of the other slave nodes themselves, the other slave nodes can still send the voting result agreeing that the second slave node is the new master node to the voting request of the second slave node.
103. And the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information sent by other nodes.
For example, the slave node transmits the write data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information transmitted by the other nodes. For example, in the case of a distributed data system having a master node and two slave nodes, i.e., a slave node a and a slave node B, respectively, the slave node a transmits a data number of write data and a data number of write data, such as TxID2, as synchronization data to the master node and the slave node B, transmits feedback information of successfully receiving the synchronization data to the slave node a when the master node receives the synchronization data, and fails to transmit feedback information to the slave node a when the master node does not receive the synchronization data. When the master node or the slave node B transmits feedback information to the slave node a, the slave node receives the feedback information.
104. And the slave node determines the first node number of other nodes which successfully receive the synchronous data according to the feedback information.
For example, the slave node determines the number of nodes of other nodes that successfully receive the synchronization data according to the feedback information, for example, whether the distributed data system has one master node and two slave nodes, i.e., the slave node a and the slave node B, by way of example, the slave node a sends the synchronization data, when the master node successfully receives the synchronization data, the master node sends the feedback information to the slave node a, so that when the slave node a receives the feedback information of the master node, it can be determined that the master node successfully receives the synchronization data, and similarly, the slave node B also can determine the number of nodes that successfully receive the synchronization data according to how many feedback information are received, for example, the number of first nodes that successfully receive the synchronization data is two, and for example, the number of the first nodes which successfully receive the synchronous data can be determined to be one by successfully receiving the feedback information sent by either the master node or the slave node B.
105. The slave node stores the write data based on the first number of nodes.
For example, the slave node compares the first number of other nodes that successfully receive the synchronization data with half of the total number of nodes of the distributed data system, when the first number of nodes exceeds half of the total number of nodes of the distributed data system, for example, when the distributed data system has one master node and three slave nodes, when the number of other nodes that successfully receive the synchronization data exceeds two, for example, the slave node a sends the synchronization data, and the other nodes that successfully receive the synchronization data are the master node, the slave node B, and the slave node C, when the slave node writes the synchronization data into the local database of the slave node a and sends data storage requests to the master node, the slave node B, and the slave node C, when the master node, the slave node B, and the slave node C receive the data storage requests, the master node, the slave node B, and the slave node C respectively store the synchronization data into their local databases, to complete the synchronization of the write data in the distributed data system. When the number of the first nodes does not exceed half of the total number of the nodes of the distributed data system, for example, when the distributed data system has one master node and three slave nodes, and the number of other nodes that successfully receive the synchronization data does not exceed two, for example, when the slave node a sends the synchronization data, the other nodes that successfully receive the synchronization data are the master node and the slave node B, at this time, the slave nodes discard the write data and send data discard requests to the other nodes, and when the other nodes receive the data discard requests, the nodes that have received the synchronization data in the other nodes discard the synchronization data.
Optionally, when the master node of the distributed data system receives a data write request, as shown in fig. 6, the process of synchronizing the write data specifically includes the following steps:
the method comprises the steps that a main node receives a data writing request sent by a terminal, the main node generates a data number of the written data according to the data writing request, the main node sends the written data and the data number to slave nodes of a distributed data system as synchronous data and receives feedback information sent by the slave nodes, the main node determines the number of second nodes of the slave nodes which successfully receive the synchronous data according to the received feedback information, and when the number of the second nodes exceeds half of the total number of the nodes of the distributed data system, the main node stores the written data to a local database of the main node and sends a data storage request to the slave nodes so as to store the synchronous data to the local database of the slave nodes.
For example, the master node receives a data write request sent by the terminal, the master node generates a data number of the write data in the write data request according to the data write request, sends the write data and the data number to slave nodes of the distributed data system as synchronous data, when the slave nodes receive the synchronous data, the slave nodes send feedback information of successfully receiving the synchronous data to the master node, the master node determines the number of second nodes of the slave nodes successfully receiving the synchronous data according to the feedback information, when the number of the second nodes exceeds half of the total number of the nodes of the distributed data system, for example, the total number of the nodes of the distributed data system is 6, and when the number of the second nodes of the slave nodes receiving the synchronous data exceeds 3, for example, the number of the second nodes is 4, the master node stores (writes) the write data into a local database of the master node, and sending a data storage request to the slave node, and after receiving the data storage request from the slave node, storing the synchronous data to a local database of the slave node by the slave node receiving the synchronous data.
Optionally, in order to solve the problem of node failure recovery in the distributed data system, the slave node and the master node may be set to meet the use requirement of the distributed data system, and specifically, the slave node and the master node may be set in failure recovery of a single node and in brain split (network isolation) failure recovery.
(1) Handling single node failure recovery
For example, as shown in fig. 7, the process of dealing with single node failure recovery may be that the slave node sends a data detection request to the master node at preset time intervals, where the data detection request is used to obtain the largest target data number in the data numbers of the master node. When the slave node fails, the data detection request cannot be sent, but once the slave node fails and is recovered, the data detection request is sent to the master node immediately after the failure is recovered, when the master node is in a normal state, the slave node can receive a target data number sent by the master node, and the slave node compares the target data number with the maximum current data number in the own data numbers. When the target data number exceeds the current data number, for example, the target data number is TxID5, and the current data number is TxID4, the slave node determines that the own data is missing, and acquires the missing data from the master node. For example, when the slave node determines that the data of the slave node is missing, the slave node sends a data acquisition request to the master node, the data acquisition request carries the current data number of the slave node, the slave node receives the missing data sent by the master node according to the current data number, and stores the missing data in the local database. When the target data number does not exceed the current data number, the slave node determines that missing data does not exist in the slave node, and at the moment, the slave node is in a normal state and can receive synchronous data or a write data request.
Optionally, in this case, the master node receives the data obtaining request, the master node compares the current data number in the data obtaining request with the target data number of the master node, and the master node determines the data missing from the slave node according to the comparison result, for example, when the target data number is TxID5 and the current data number is TxID4, the master node determines that the data missing from the slave node is the write-in data corresponding to TxID5, and the master node sends the write-in data corresponding to TxID5 to the slave node as the data missing from the slave node.
Optionally, when the original master node in the distributed data system fails, the slave node may select a new master node in the distributed data system again, and when the data number of the original master node is smaller than the maximum data number of the new master node, the original master node serves as the slave node to send a data acquisition request to the new master node at this time, and the process of recovering from the node failure is repeated.
(2) Coping with split brain (network isolation) fault recovery
The split brain fault is network isolation of nodes in a node cluster of the distributed data system, for example, a node cluster is split into two sub-node clusters with different numbers, and nodes of the two sub-node clusters cannot communicate with each other. In this case, there are two cases, the first case is that the original master node is in a first child node cluster in which the number of slave nodes exceeds half of the total number of slave nodes in the distributed data system, and the second case is that the original master node is in a second child node cluster in which the number of slave nodes does not exceed half of the total number of slave nodes in the distributed data system.
C1, when the original main node is in the first sub-node cluster, the failure recovery process of the distributed data system.
For example, when the original master node is in the first child node cluster, as shown in fig. 8, since the number of slave nodes in the first child node cluster exceeds half of all the slave nodes, when a slave node or a master node in the first child node cluster receives a data write request, in the first child node cluster, since the master node and the slave node are in a normal state, the write data in the data write request may be synchronized, and in this case, the entire distributed data system may operate normally. However, when the slave node 4 or the slave node 5 in the second child node cluster receives a data write request, the slave node 4 or the slave node 5 sends a data synchronization request to the master node, but due to a split-brain fault, the slave node 4 or the slave node 5 cannot acquire a data number of the write data from the master node, and the slave node 4 or the slave node 5 determines that the master node is in an abnormal state, so the slave node 4 or the slave node 5 triggers a new master node determination process, but after the slave node 4 or the slave node 5 triggers the new master node determination process, only one slave node receives a voting request and returns a voting result, and even if the slave node that sends the voting request is granted, the new master node is determined to fail because the number of granted voting results does not exceed half of the total number of the slave nodes, thereby causing a data write failure. When the network isolation fault is recovered, the slave node 4 or the slave node 5 of the second child node cluster can know that the master node is in a normal state when the slave node 4 or the slave node 5 of the second child node cluster can receive synchronous data transmitted by slave nodes except the other side, in this case, the slave node 4 or the slave node 5 transmits a data detection request to the master node to judge whether the slave node has data missing or not, compares the target data number with the current data number which is the maximum of the slave node after receiving the target data number transmitted by the master node, determines that the slave node 4 or the slave node 5 has data missing when the target data number exceeds the current data number, and transmits the current data number of the master node to the master node, determines the data missing of the slave node 4 or the slave node 5 after receiving the current data number transmitted by the slave node 4 or the slave node 5, and sends the missing data to either slave node 4 or slave node 5.
Wherein in this case it is the slave node in the second cluster of child nodes that is requiring failover.
C2, when the original main node is in the second sub node cluster, the failure recovery process of the distributed data system.
For example, when the original master node is in the second child node cluster, as shown in fig. 9, when a network isolation fault occurs and a slave node in the first child node cluster receives a data write request, for example, the slave node 1 receives a data write request sent by a terminal, the slave node sends a data synchronization request to the master node, and as a result, the slave node 1 determines that the master node is in an abnormal state due to the fault of network isolation, and therefore, the slave node 1 triggers a process of determining a new master node in the first child node cluster, the slave node 1 sends a voting request to the slave nodes 2 to 4, the slave node 1 receives voting results sent by the slave nodes 2 to 4, and when there are three slave nodes 1 that have agreed to send out voting requests in the voting results, the slave node 1 selects as a new master node in the first child node cluster. In the second sub-node cluster, when the master node or the slave node 5 receives a data write request and sends synchronous data to other nodes, the number of other nodes which successfully receive the synchronous data does not exceed half of the total number of the nodes, so that the data write fails, and therefore the second sub-node cluster cannot work normally.
When a network failure is recovered, there are two master nodes in the distributed data system, one being the original master node and the other being the slave node becoming the new master node. In the distributed data system, only in the process of sending and receiving synchronous data between the original main node and the new main node, the original main node and the new main node can sense the existence of each other.
The original main node and the new main node sense the existence of each other mainly in two situations:
the first method is that any one of the original master node or the new master node receives a data write-in request, for example, when the original master node receives the data write-in request, the original master node generates a data number of the write-in data, the data number of the write-in data and the data number of the write-in data are sent to other nodes of the distributed data system as synchronous data, when the synchronous data are sent to the new master node and the new master node successfully receives the synchronous data, the original master node can determine the identity of the new master node, and when the original master node determines the identity of the new master node, the identity of the new master node is obtained.
The second is that any one of the original master node or the new master node receives the synchronization data sent by the other party, for example, taking the example that the original master node receives the synchronization data sent by the new master node, the original master node may determine the identity of the new master node sending the synchronization data, and obtain the identity of the new master node.
When the main node determines that other main nodes exist in the distributed data system, the main node sends a data comparison request to the other main nodes according to the obtained identity marks of the other main nodes, wherein the data comparison request is used for obtaining the maximum first target data number in the data numbers of the other main nodes. The master node receives the first target data numbers sent by other master nodes, compares the first target data numbers with the largest second target data number in the self data numbers, and when the first target data numbers exceed the self second target data numbers, the master node becomes a slave node and acquires data missing from the other master nodes. For example, when the first target data number is TxID5 and the second target data number is TxID4, the original master node becomes a slave node and acquires the data missing from the original master node from the new master node.
Optionally, the process of acquiring the missing data by the master node is similar to that of acquiring the missing data by the slave node, and when the master node determines that there is data missing, the master node becomes the slave node, and thus the slave node acquires the missing data from another master node.
As can be seen from the above, in this embodiment, after a slave node receives a data write request sent by a terminal, the data write request carries write data, the data write request is used to write the write data into nodes of a distributed data system, the slave node obtains a data number of the write data from a master node according to the data write request, the data number is a sequential identifier of the write data in the distributed data system, the slave node sends the write data and the data number to other nodes of the distributed data system as synchronous data, and receives feedback information sent by the other nodes, the slave node determines, according to the feedback information, a first node number of the other nodes that successfully receive the synchronous data, and the slave node stores the write data based on the first node number; according to the scheme, the slave nodes can receive the written data, the master nodes are used for numbering the written data, the slave nodes synchronize the data numbers and the written data to other nodes of the distributed data system as synchronous data, the plurality of nodes are used for receiving the written data, the master nodes are used for numbering, a plurality of nodes can receive more written data, ordered synchronization of the written data can be guaranteed, and therefore the data storage efficiency can be greatly improved.
The method described in the above examples is further illustrated in detail below by way of example.
In this embodiment, the data storage device is specifically integrated in an electronic device, and the electronic device may include a master node server and a slave node server.
As shown in fig. 10, a data storage method specifically includes the following processes:
201. and receiving a data writing request sent by the terminal from the node server.
For example, the terminal may directly send a data write request to a slave node server of the distributed data system at the address of the node, and the slave node server receives the data write request. The terminal can also store the written data carried in the written data request to a third-party database, and send the storage address of the written data stored in the third-party database to the slave node server in the written data request, after receiving the data writing request from the slave node server, the slave node server obtains the storage address of the written data stored in the third-party database in the data writing request, obtains the written data in the third-party database according to the storage address, and can also send prompt information to the terminal after obtaining the written data, so as to prompt the terminal to successfully obtain the written data from the node server.
202. And generating a data synchronization request from the node server according to the data writing request.
For example, after receiving a data write request from the node server, a data synchronization request is generated according to the received data write request, and information related to the write data, such as the name, the receiving time, the type, and/or the data size of the write data, is added to the data synchronization request.
203. The slave node server sends a data synchronization request to the master node server.
For example, the slave node server obtains an address of a master node server in the distributed data system, and sends a data synchronization request to the master node server according to the address.
Optionally, before the slave node server sends the data synchronization request to the server before the master node, the state of the master node server may also be tested, when the master node server is in a normal state, the data synchronization request is sent to the master node, and when the master node server is in an abnormal state, a new master node server needs to be reselected and selected in the distributed data system. Therefore, before the step "the slave node server sends the data synchronization request to the master node server", the data storage method may further include:
the slave node server sends a test request to the master node server, the test request is used for testing the state of the master node server, the slave node server receives feedback information of the test request and determines whether the master node server is in a normal state or not according to the feedback information, and when the master node server is in the normal state, the slave node server sends a synchronous data request to the master node server.
For example, whether the master node server is in a normal state is tested, the slave node server sends a test request to the master node server, and the master node server sends feedback information after delaying for a period of time (Td) after receiving the test request, so that the state of the master node server can be judged according to the time when the feedback information is received, the distributed data system can generate the feedback information that the master node server is in an abnormal state when the feedback information sent by the master node server is not received within the Td, the master node server can be indicated to be in a normal state when the feedback information sent by the master node server is received within the Td, the slave node server receives the feedback information of the test request, and the master node server is indicated to be still in a normal state when the feedback information is that the master node server is in a normal state, the master-slave relationship between the master node server and the slave node server is still in a normal state. When the master node server is in a normal state, the master-slave relationship still exists in the distributed data system, and the slave node server can send the data synchronization request to the master node server.
Optionally, when the feedback information received from the slave node server indicates that the master node server is in the abnormal state, the slave node server determines that the master node server is in the abnormal state. When the master node server is in an abnormal state, the slave node server triggers a new master node server to determine a process, for example, when the master node server is in an abnormal state, the slave node server triggers a process of determining the new master node server in the distributed data system, the process is used for selecting the new master node server in the distributed data system, and the candidate object of the new master node server is the slave node server triggering the process. For example, when a test request is sent to the master node server, the master node server does not receive feedback information within a preset time interval or the feedback information is not received by the master node server, the slave node server needs to send a test request again outside the preset time interval, if the feedback information of the test request is still abnormal or the master node server does not receive the test request, it can be really determined that the master node server is abnormal, if the feedback information of the second test request is normal, it is determined that the master node server is still in a normal state, and when it is determined that the master node server is in a normal state, a data synchronization request is continuously sent to the master node server.
Optionally, when the slave node server determines that the master node server is in an abnormal state, a new master node server needs to be determined in the distributed data system, at this time, the distributed data system enters a new master node server determination process, and the working process of the slave node server is specifically as follows:
the method comprises the steps that a slave node server sends a voting request for selecting a new master node server to other slave node servers in a distributed data system, wherein the voting request carries the largest first data number in the data numbers of the slave node servers, the slave node servers receive voting results of other slave node servers responding to the voting request, and the voting results can be in two situations, namely that the slave node server which agrees to send out the voting request is the new master node server, and the slave node server which does not agree to send out the voting request is the new master node server. When the slave node servers receive the voting results of the slave node servers which agree to send out the voting requests as the new master node server, the slave node servers automatically select the master node server. After the master node server is selected, the slave node server becomes the master node server, and the slave node server can generate the data number of the written data and perform the next step of storing the written data. When the number of the voting results of the slave node servers which agree to send out the voting requests as the new master node server is not more than half of the total number of the slave node servers of the distributed data system, at the moment, the flow initiated by the slave node server for selecting the new master node server fails to master, and the slave node server returns prompt information of data writing failure to the terminal sending the data writing request.
Optionally, in the main process of voting, after receiving the voting request, the other slave node servers compare the first data number of the slave node server carried in the voting request with the largest second data number in their own data numbers. When the second data number does not exceed the first data number, the other slave node servers send the voting result of the slave node server which agrees to send the voting request as the new master node server to the slave node server, for example, when the first data number is TxID5, the second data number is TxID4, and the other slave node servers find that the second data number of the other slave node servers does not exceed the first data number of the slave node server, the other slave node servers send the voting result of the slave node server which agrees to send the voting request as the new master node server to the slave node server. When the largest second data number in the data numbers of the other slave node servers exceeds the first data number, the other slave node servers send the voting result that the slave node server not agreeing to send the voting request is the new master node server, for example, when the second data number is TxID5 and the first data number is TxID4, the other slave node servers send the voting result that the slave node server not agreeing to send the voting request is the new master node server.
Optionally, when the second data number does not exceed the first data number, and other slave node servers receive the voting requests sent by two different slave node servers within a preset time interval, in this case, the voting results need to be sent to the slave node servers corresponding to different voting requests according to the number of times of receiving the voting requests. For example, when the first data number of two different slave node servers is TxID5 and the second data number is TxID4, and within a time interval, the other slave node servers send the voting request to the slave node server which sends the voting request for the first time as the voting result of the new slave node server agreeing to send the voting request, for example, within five minutes, the other slave node servers receive two voting requests which respectively correspond to the first slave node server and the second slave node server, but the other slave node server first receives the voting request sent by the first slave node server, when the first data number of the first slave node server exceeds the second data number of the other slave node server, the other slave node servers send the voting result to the first slave node server agreeing to send the first slave node server as the new master node, although the first data number of the second slave node server is also larger than the second data number of the second slave node server, the other slave node servers still send the voting result which does not agree that the second slave node server is the new master node server to the second slave node server. It is assumed that the first slave node server transmits a voting request and the other slave node servers transmit voting results agreeing that the first slave node server is the new master node server. After the preset time interval is exceeded, for example, the preset time interval is ten minutes, but in the eleventh minute, the other slave node servers receive the voting request sent by the second slave node server, and when the first data number of the second slave node server exceeds the second data number of the other slave node server itself, the other slave node servers can still send the voting result agreeing that the second slave node server is the new master node server to the voting request of the second slave node server.
204. When the master node server returns the data number, the slave node server receives the data number.
For example, when the master node server successfully receives the data synchronization request, the master node server numbers the written data according to the information related to the written data in the data synchronization request, generates a data number of the written data, sends the data number of the written data to the slave node server, and the slave node server receives the data number.
205. And when the main node server does not return the data number, the slave node server returns prompt information of data writing failure to the terminal.
For example, when the master node server does not successfully receive the data synchronization request, or does not return the data number of the write data after receiving the data synchronization request, the slave node server does not receive the number of the write data, and at this time, the slave node server returns a notification message of data write failure to the terminal, and discards the write data.
206. And when the slave node server receives the data number, the slave node server sends the written data and the data number as synchronous data to other node servers of the distributed data system, and receives feedback information sent by the other node servers.
For example, in the case where the distributed data system includes a master node server and two slave node servers, i.e., a master node server, a slave node server a and a slave node server B, the slave node server a transmits a data number of write data and write data, such as TxID2, as synchronization data to the master node server and the slave node server B, and when the master node server receives the synchronization data, the master node server transmits feedback information indicating that the synchronization data was successfully received to the slave node server a, and when the master node server does not receive the synchronization data, the master node server cannot transmit feedback information to the slave node server a. When the master node server or the slave node server B transmits feedback information to the slave node server a, the slave node server receives the feedback information.
207. And the slave node server determines the first node number of other node servers which successfully receive the synchronous data according to the feedback information.
For example, in the distributed data system, there are a master node server and two slave node servers, which are respectively a master node server, a slave node server a and a slave node server B, and the slave node server a sends synchronization data, when the master node server successfully receives the synchronization data, the master node server sends feedback information to the slave node server a, so that when the slave node server a receives the feedback information of the master node server, it can be determined that the master node server successfully receives the synchronization data, and similarly, the slave node server B also can determine the number of nodes that successfully receive the synchronization data according to how much feedback information is received, for example, the number of first nodes that successfully receive the synchronization data is two by successfully receiving the feedback information of the master node and the slave node B, for another example, if the feedback information sent by either the master node server or the slave node server B is successfully received, the number of the first nodes that successfully receive the synchronization data can be determined to be one.
208. The slave node server stores the write data based on the first number of nodes.
For example, the slave node server compares the first number of nodes of the other node servers that successfully received the synchronization data with half of the total number of node servers of the distributed data system, and when the first number of nodes exceeds half of the total number of node servers of the distributed data system, for example, when the distributed data system has one master node server and three slave node servers, that is when the number of nodes of the other node servers that successfully received the synchronization data exceeds two, for example, when the synchronization data is transmitted from the node server a, and the other node servers that successfully received the synchronization data are the master node server, the slave node server B, and the slave node server C, at this time, the slave node server a writes the synchronization data into the local database of the slave node server a, and simultaneously transmits a data storage request to the master node server, the slave node server B, and the slave node server C, when the master node server, the slave node server B and the slave node server C receive data storage requests, the master node server, the slave node server B and the slave node server C respectively store the synchronous data into local databases of the master node server, the slave node server B and the slave node server C so as to complete the synchronization of the written data in the distributed data system. When the number of the first node is not more than half of the total number of the node servers of the distributed data system, for example, when the distributed data system has one master node server and three slave node servers, and the number of the other node servers which successfully receive the synchronization data is not more than two, for example, when the slave node server a transmits the synchronization data, the other node servers which successfully receive the synchronization data are the master node server and the slave node server B, at this time, the slave node servers discard the write data and transmit a data discard request to the other node servers, and when the other node servers receive the data discard request, the node servers which have received the synchronization data, such as the master node server and the slave node server B, discard the synchronization data.
Optionally, when the master node server in the distributed data system receives a data write request, the synchronization process for the write data may be as follows:
the method comprises the steps that a main node server receives a data writing request sent by a terminal, the main node server generates a data number of the written data according to the data writing request, the main node server sends the written data and the data number as synchronous data to slave node servers of a distributed data system and receives feedback information sent by the slave node servers, the main node server determines the number of second nodes of the slave node servers which successfully receive the synchronous data according to the received feedback information, and when the number of the second nodes exceeds half of the total number of the node servers of the distributed data system, the main node server stores the written data to a local database of the main node server and sends a data storage request to the slave node servers so as to store the synchronous data to the local database of the slave node servers.
For example, the master node server receives a data write request sent by the terminal, the master node server generates a data number of the write data in the write data request according to the data write request, sends the write data and the data number as synchronous data to the slave node servers of the distributed data system, when the slave node servers receive the synchronous data, the slave node servers send feedback information of successfully receiving the synchronous data to the master node server, the master node server determines a second node number of the slave node servers successfully receiving the synchronous data according to the feedback information, when the second node number exceeds half of the total number of the node servers of the distributed data system, for example, the total number of the nodes of the distributed data system is 6, when the second node number of the slave node servers receiving the synchronous data exceeds 3, for example, the second node number is 4, the master node server stores (writes) the write data into a local database of the master node server, sends a data storage request to the slave node server, and after receiving the data storage request, the slave node server receiving the synchronous data stores the synchronous data into the local database of the slave node server.
Optionally, in order to solve the problem that the node server in the distributed data system fails and recovers, the master node server and the slave node server may be set to meet the use requirement of the distributed data system. The setting can be specifically performed for the slave node server and the master node server in the fault recovery and split brain (network isolation) fault recovery of a single node server in the distributed data system.
(1) Handling failover of a single node server
For example, when the slave node server fails and recovers, the slave node server may send a data detection request to the master node server at a preset time interval, where the data detection request is used to obtain a maximum target data number from the data numbers of the master node server. When the slave node server fails, the data detection request cannot be sent, but once the slave node server fails and is recovered, the data detection request is sent to the master node server immediately after the failure is recovered, when the master node server is in a normal state, the slave node server can receive a target data number sent by the master node server, and the slave node server compares the target data number with the maximum current data number in the self data numbers. When the target data number exceeds the current data number, for example, the target data number is TxID5, and the current data number is TxID4, the slave node server determines that the own data is missing, and acquires the missing data from the master node server. For example, when the slave node server determines that the data of the slave node server is missing, the slave node server sends a data acquisition request to the master node server, the data acquisition request carries the current data number of the slave node server, the slave node server receives the missing data sent by the master node server according to the current data number, and stores the missing data in the local database.
Optionally, in this case, the master node server receives the data obtaining request, compares the current data number in the data obtaining request with the target data number of the master node server, and determines the data missing from the slave node server according to the comparison result, for example, when the target data number is TxID5 and the current data number is TxID4, the master node server determines that the data missing from the slave node server is the write-in data corresponding to TxID5, and the master node server sends the write-in data corresponding to TxID5 to the slave node server as the data missing from the slave node server.
Optionally, when the original master node server in the distributed data system fails, the slave node server may select a new master node server in the distributed data system again, where the original master node server is used as the slave node server to send a data acquisition request to the new master node server, and the process of recovering from the failure of the slave node server is repeated.
(2) Fault recovery to handle split brain (network isolation)
For example, when dealing with a split brain (network isolation) fault recovery, the node server cluster may be split into a first child node server cluster and a second child node server cluster with two nodes of different numbers. The method can be considered in two cases, the first case is that the original main node server in the distributed data system is in a first sub-node server cluster with the number of slave nodes exceeding half of the total number of the slave node servers, and the second case is that the original main node server in the distributed data system is in a second sub-node server cluster with the number of slave node servers not exceeding half of the total number of the slave node servers.
A1, when the original main node server is in the first sub node server cluster, the failure recovery process of the distributed data system.
For example, when the original master node server is in the first child node server cluster, since the number of slave node servers in the first child node server cluster exceeds half of all the slave node servers, when a slave node server or a master node server in the first child node server cluster receives a data write request, in the first child node server cluster, since the master node server and the slave node server are both in a normal state, the write data in the data write request can be synchronized, and in this case, the entire distributed data system can be operated normally. However, when the slave node server 4 or the slave node server 5 in the second child node server cluster receives a data write request, the slave node server 4 or the slave node server 5 sends a data synchronization request to the master node server, but because a split-brain fault occurs, the slave node server 4 or the slave node server 5 cannot acquire the data number of the write data from the master node server, and the slave node server 4 or the slave node server 5 determines that the master node server is in an abnormal state, therefore, the slave node server 4 or the slave node server 5 triggers a new master node server determination flow, but after the slave node server 4 or the slave node server 5 triggers the new master node server determination flow, only one slave node server receives a voting request and returns a voting result, and then the slave node server that agrees to send the voting request is counted, it is also determined that the new master node server fails because the number of agreed voting results does not exceed half of the total number of slave node servers, thereby causing a data write failure. When the network isolation fault is recovered, the slave node server 4 or the slave node server 5 of the second child node server cluster can know that the master node server is in a normal state when the slave node server 4 or the slave node server 5 can receive the synchronous data transmitted by the slave node servers except the other party, in this case, the slave node server 4 or the slave node server 5 transmits a data detection request to the master node server to judge whether the slave node server has data missing or not, compares the target data number with the current data number which is the maximum of the slave node server after receiving the target data number transmitted by the master node server, determines that the slave node server has data missing when the target data number exceeds the current data number, transmits the current data number of the slave node server to the master node server, and after receiving the current data number transmitted by the slave node server 4 or the slave node server 5, determines missing data from the node server 4 or from the node server 5 and sends the missing data to the slave node server 4 or the slave node server 5.
Wherein, in this case, it is the slave node server in the second cluster of child node servers that is requiring failover.
And A2, when the original main node server is in the second sub node server cluster, the failure recovery process of the distributed data system is carried out.
For example, when the original master node server is in the second child node cluster, and when a network isolation fault occurs, and a slave node server in the first child node server cluster receives a data write request, such as a data write request sent by a terminal, from the node server 1, the slave node server sends a data synchronization request to the master node server, and as a result, the slave node server 1 determines that the master node server is in an abnormal state due to the fault of network isolation, therefore, the slave node server 1 triggers a process of determining a new master node server in the first child node server cluster, the slave node server 1 sends a voting request to the slave node servers 2 to 4, the slave node server 1 receives voting results sent by the slave node servers 2 to 4, and when there are three slave node servers 1 in the voting results that agree to send out the voting request as new master node servers, the slave node server 1 is selected as the new master node server in the first cluster of child node servers. In the second child node cluster, when the master node server or the slave node server 5 receives a data write request and sends synchronous data to other nodes, the number of other nodes which successfully receive the synchronous data does not exceed half of the total number of the nodes, so that the data write fails.
When a network failure is recovered, there are two master node servers in the distributed data system, one is the original master node server and the other is the slave node server becoming the new master node server. In the distributed data system, only in the process of sending and receiving synchronous data between the original main node server and the new main node server, the original main node server and the new main node server can sense the existence of the other side.
The original main node server and the new main node server sense the existence of the other party, and there are two main situations:
the first method is that any one of the original master node server and the new master node server receives a data write-in request, for example, when the original master node server receives the data write-in request, the original master node server generates a data number of the write-in data, the data number of the write-in data and the data number of the write-in data are sent to other nodes of the distributed data system as synchronous data, when the synchronous data are sent to the new master node server and the new master node server successfully receives the synchronous data, the original master node server can determine the identity of the new master node server, and when the original master node server determines the identity of the new master node server, the identity of the new master node server is obtained.
The second is that any node in the original master node server or the new master node server receives the synchronization data sent by the other party, for example, taking the example that the original master node server receives the synchronization data sent by the new master node server, the original master node server may determine the identity of the new master node server sending the synchronization data, and obtain the identity of the new master node server.
When the main node server determines that other main node servers exist in the distributed data system, the main node server sends a data comparison request to the other main node servers according to the obtained identity marks of the other main node servers, wherein the data comparison request is used for obtaining the maximum first target data number in the data numbers of the other main node servers. The master node server receives the first target data numbers sent by other master node servers, compares the first target data numbers with the largest second target data number in the self data numbers, and when the first target data numbers exceed the self second target data numbers, the master node server becomes a slave node server and acquires data missing from the master node servers. For example, when the first target data number is TxID5 and the second target data number is TxID4, the original master node server becomes a slave node server and acquires the missing data of the original master node server from the new master node server.
Optionally, the process of acquiring the missing data by the master node server is similar to that of acquiring the missing data by the slave node server, and when the master node server determines that the data is missing, the master node server becomes the slave node server, and thus the slave node server acquires the missing data from another master node server.
As can be seen from the above, in this embodiment, after receiving a data write request sent by a terminal from a node server, the data write request carries write data, the data write request is used to write the write data into the node servers of a distributed data system, the slave node server obtains a data number of the write data from a master node server according to the data write request, where the data number is a sequence identifier of the write data in the distributed data system, the slave node server sends the write data and the data number as synchronous data to other node servers of the distributed data system and receives feedback information sent by the other node servers, the slave node server determines, according to the feedback information, a first number of nodes of the other node servers that successfully receive the synchronous data, and the slave node server determines, based on the first number of nodes, storing the written data; according to the scheme, the slave node servers can receive the written data, the master node servers are used for numbering the written data, the slave node servers synchronize the data numbers and the written data to other node servers of the distributed data system as synchronous data, the plurality of node servers are used for receiving the written data, the master node servers are used for numbering, a plurality of node servers can receive more written data, ordered synchronization of the written data can be guaranteed, and therefore the data storage efficiency can be greatly improved.
In order to better implement the above method, embodiments of the present invention further provide a data storage apparatus, where the data storage apparatus includes a master node and a slave node, and the data storage apparatus may be integrated in an electronic device, such as a server, and may be a single server or a server cluster formed by multiple servers.
For example, as shown in fig. 11, the data storage device may include a receiving unit 301, an obtaining unit 302, a transmitting unit 303, a determining unit 304, and a storing unit 305, as follows:
(1) a receiving unit 301;
a receiving unit 301, configured to receive, from a node, a data write request sent by a terminal, where the data write request carries write data, and the data write request is used to write the write data into a node of a distributed data system.
For example, the receiving unit 301 may be specifically configured to send a data write request to a slave node of the distributed data system directly by the terminal at the address of the node, and the slave node receives the data write request. The terminal can also store the written data carried in the written data request to a third-party database, send the storage address of the written data stored in the third-party database to the slave node in the written data request, obtain the storage address of the written data stored in the third-party database in the data writing request after the slave node receives the data writing request, and obtain the written data in the third-party database according to the storage address.
(2) An acquisition unit 302;
an obtaining unit 302, configured to obtain, by the slave node, a data number of the write data from the master node according to the data write request.
The obtaining unit 302 may include a generating subunit 3021, a sending subunit 3022, a receiving subunit 3023, and a returning subunit 3024, as shown in fig. 12, specifically as follows:
a generation subunit 3021 configured to generate a data synchronization request from a node according to the data write request;
a transmitting subunit 3022, configured to transmit a data synchronization request from the slave node to the master node;
a receiving subunit 3023, configured to receive the data number from the slave node when the master node returns the data number;
a return subunit 3024, configured to, when the master node does not return the data number, return the notification information of the data write failure to the terminal from the slave node.
For example, in accordance with a data write request, the slave node of the generation subunit 3021 generates a data synchronization request, the slave node of the transmission subunit 3022 transmits the data synchronization request to the master node, the slave node of the reception subunit 3023 receives the data number when the master node returns the data number, and the slave node of the return subunit 3024 returns notification information of a data write failure to the terminal when the master node does not return the data number.
(3) A transmitting unit 303;
a sending unit 303, configured to send the write data and the data number as synchronous data to other nodes of the distributed data system from the node, and receive feedback information sent by other nodes.
For example, the sending unit 303 may be specifically configured to send, by the node, the written data and the data number as the synchronization data to other nodes of the distributed data system, where when the other nodes successfully receive the synchronization data, the other nodes send feedback information that the receiving is successful to the slave node, and when the other nodes do not successfully receive the synchronization data, the other nodes do not return the feedback information, and receive, from the slave node, the feedback information sent by the other nodes that successfully receive the synchronization data.
(4) A determination unit 304;
a determining unit 304, configured to determine, according to the feedback information, a first number of nodes of other nodes that successfully receive the synchronization data from the node.
For example, the determining unit 304 may be specifically configured to, when receiving the feedback information sent by the other node, determine that the other node successfully receives the synchronization data, and therefore, only by analyzing the number of the received feedback information, may determine the first number of nodes of the other node that successfully receives the synchronization data.
(5) A storage unit 305;
a storage unit 305 for storing the write data based on the first number of nodes from the nodes.
For example, the storage unit 305 may be specifically configured to compare the first number of nodes of the other nodes that successfully receive the synchronization data with half of the total number of nodes of the distributed data system, and when the first number of nodes exceeds half of the total number of nodes of the distributed data system, the slave node writes the synchronization data into the local database of the slave node and sends a data saving request to the other nodes to save the synchronization data to the local databases of the other nodes; when the number of the first nodes does not exceed half of the total number of the nodes, the slave nodes discard the written data and send data discarding requests to other nodes so as to discard the synchronous data received by other nodes.
Optionally, the data storage device may further include a writing unit 306, as shown in fig. 13, which is as follows:
a write unit 306, configured to receive a data write request sent by a terminal, where the master node generates a data number of the write data according to the data write request, the master node sends the write data and the data number to slave nodes of the distributed data system as synchronous data, and receives feedback information sent by the slave nodes, the master node determines, according to the feedback information, the number of second nodes of the slave nodes that successfully receive the synchronous data, when the number of the second nodes exceeds half of the total number of the nodes of the distributed data system, the master node stores the write data in a local database of the master node, and sends a data storage request to the slave nodes to store the synchronous data in the local database of the slave nodes, and when the number of the second nodes does not exceed half of the total number of the nodes of the distributed data system, the master node discards the write data and sends a data discard request to the slave nodes, so that the slave node discards the synchronization data.
In a specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and the specific implementation of the above units may refer to the foregoing method embodiments, which are not described herein again.
As can be seen from the above, in this embodiment, after the slave node of the receiving unit 301 receives a data write request sent by a terminal, where the data write request carries write data, the data write request is used to write the write data into a node of a distributed data system, the slave node of the obtaining unit 302 obtains, according to the data write request, a data number of the write data from a master node, where the data number is a sequence identifier of the write data in the distributed data system, the slave node of the sending unit 303 sends the write data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information sent by the other nodes, the slave node of the determining unit 304 determines, according to the feedback information, a first number of nodes of the other nodes that successfully receive the synchronous data, and the slave node of the storing unit 305 determines, based on the first number of nodes, storing the written data; according to the scheme, the slave nodes can receive the written data, the master nodes are used for numbering the written data, the slave nodes synchronize the data numbers and the written data to other nodes of the distributed data system as synchronous data, the plurality of nodes are used for receiving the written data, the master nodes are operated for numbering, a plurality of nodes can receive more written data, ordered synchronization of the written data can be guaranteed, and therefore the data storage efficiency can be greatly improved.
An embodiment of the present invention further provides an electronic device, as shown in fig. 14, which shows a schematic structural diagram of the electronic device according to the embodiment of the present invention, specifically:
the electronic device may include components such as a processor 401 of one or more processing cores, memory 402 of one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 14 does not constitute a limitation of the electronic device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:
the processor 401 is a control center of the electronic device, connects various parts of the whole electronic device by various interfaces and lines, performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the electronic device. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401.
The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by operating the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the electronic device, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 access to the memory 402.
The electronic device further comprises a power supply 403 for supplying power to the various components, and preferably, the power supply 403 is logically connected to the processor 401 through a power management system, so that functions of managing charging, discharging, and power consumption are realized through the power management system. The power supply 403 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.
The electronic device may further include an input unit 404, and the input unit 404 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
Although not shown, the electronic device may further include a display unit and the like, which are not described in detail herein. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application program stored in the memory 402, thereby implementing various functions as follows:
the method comprises the steps that after a slave node receives a data writing request sent by a terminal, the data writing request carries writing data, the data writing request is used for writing the writing data into nodes of a distributed data system, the slave node obtains a data number of the writing data from a master node according to the data writing request, the data number is a sequential identifier of the writing data in the distributed data system, the slave node sends the writing data and the data number to other nodes of the distributed data system as synchronous data and receives feedback information sent by the other nodes, the slave node determines the number of first nodes of the other nodes which successfully receive the synchronous data according to the feedback information, and the slave node stores the writing data based on the number of the first nodes
For example, the slave node receives a data write request sent by the terminal, generates a data synchronization request according to the data write request, sends the data synchronization request to the master node, receives the data number when the master node returns the data number, and returns a prompt message of data write failure to the terminal when the master node does not return the data number. When receiving a data number from a node, the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system, receives feedback information sent by the other nodes, determines the first node number of the other nodes which successfully receive the synchronous data according to the feedback information, writes the synchronous data into a local database of the slave node by the slave node when the first node number exceeds half of the total number of the nodes, and sends a data storage request to the other nodes so as to store the synchronous data into the local database of the other nodes, and discards the written data by the slave node and sends a data discarding request to the other nodes so as to discard the synchronous data received by the other nodes when the first node number does not exceed half of the total number of the nodes.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
As can be seen from the above, in the embodiment of the present invention, after a slave node receives a data write request sent by a terminal, the data write request carries write data, the data write request is used to write the write data into nodes of a distributed data system, the slave node obtains a data number of the write data from a master node according to the data write request, the data number is a sequential identifier of the write data in the distributed data system, the slave node sends the write data and the data number to other nodes of the distributed data system as synchronous data and receives feedback information sent by the other nodes, the slave node determines, according to the feedback information, a first node number of the other nodes that successfully receive the synchronous data, and the slave node stores the write data based on the first node number; according to the scheme, the slave nodes can receive the written data, the master nodes are used for numbering the written data, the slave nodes synchronize the data numbers and the written data to other nodes of the distributed data system as synchronous data, the plurality of nodes are used for receiving the written data, the master nodes are operated for numbering, a plurality of nodes can receive more written data, ordered synchronization of the written data can be guaranteed, and therefore the data storage efficiency can be greatly improved.
It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.
To this end, the embodiment of the present invention provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any data storage method provided by the embodiment of the present invention. For example, the instructions may perform the steps of:
the method comprises the steps that after a slave node receives a data writing request sent by a terminal, the data writing request carries writing data, the data writing request is used for writing the writing data into nodes of a distributed data system, the slave node obtains a data number of the writing data from a master node according to the data writing request, the data number is a sequential identifier of the writing data in the distributed data system, the slave node sends the writing data and the data number to other nodes of the distributed data system as synchronous data and receives feedback information sent by the other nodes, the slave node determines the number of first nodes of the other nodes which successfully receive the synchronous data according to the feedback information, and the slave node stores the writing data based on the number of the first nodes
For example, the slave node receives a data write request sent by the terminal, generates a data synchronization request according to the data write request, sends the data synchronization request to the master node, receives the data number when the master node returns the data number, and returns a prompt message of data write failure to the terminal when the master node does not return the data number. When receiving a data number from a node, the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system, receives feedback information sent by the other nodes, determines the first node number of the other nodes which successfully receive the synchronous data according to the feedback information, writes the synchronous data into a local database of the slave node by the slave node when the first node number exceeds half of the total number of the nodes, and sends a data storage request to the other nodes so as to store the synchronous data into the local database of the other nodes, and discards the written data by the slave node and sends a data discarding request to the other nodes so as to discard the synchronous data received by the other nodes when the first node number does not exceed half of the total number of the nodes.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
Wherein the computer-readable storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
Since the instructions stored in the computer-readable storage medium can execute the steps in any data storage method provided by the embodiment of the present invention, the beneficial effects that can be achieved by any data storage method provided by the embodiment of the present invention can be achieved, which are detailed in the foregoing embodiments and will not be described again here.
The data storage method, the data storage device, and the computer-readable storage medium according to the embodiments of the present invention are described in detail, and the principles and embodiments of the present invention are described herein by using specific examples, and the descriptions of the above embodiments are only used to help understand the method and the core ideas of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (15)

1. A data storage method for data write operations in a distributed data system, wherein the distributed data system includes a master node and at least two slave nodes, comprising:
the slave node receives a data writing request sent by a terminal, wherein the data writing request carries writing data, and the data writing request is used for writing the writing data into the nodes of the distributed data system;
the slave node acquires a data number of the written data from the master node according to the data writing request, wherein the data number is a sequential identifier of the written data in the distributed data system;
the slave node sends the written data and the data number as synchronous data to other nodes of the distributed data system, and receives feedback information sent by the other nodes;
the slave node determines the number of first nodes of other nodes which successfully receive the synchronous data according to the feedback information;
the slave node stores the write data based on the first number of nodes.
2. The data storage method of claim 1, wherein the slave node stores the write data based on the first number of nodes, comprising:
when the number of the first nodes exceeds half of the total number of the nodes, the slave nodes write the synchronous data into the local databases of the slave nodes and send data storage requests to other nodes so as to store the synchronous data into the local databases of other nodes;
when the number of the first nodes does not exceed half of the total number of the nodes, the slave nodes discard the write data and send data discarding requests to the other nodes so as to discard the synchronous data received by the other nodes.
3. The data storage method according to claim 1, wherein the obtaining, by the slave node, the data number of the write data from the master node according to the data write request includes:
according to the data writing request, the slave node generates a data synchronization request;
the slave node sends the data synchronization request to the master node;
when the master node returns the data number, the slave node receives the data number;
and when the master node does not return the data number, the slave node returns prompt information of data writing failure to the terminal.
4. The data storage method of claim 3, wherein before the slave node sends the data synchronization request to the master node, the method further comprises:
the slave node sends a test request to the master node, wherein the test request is used for testing the state of the master node;
the slave node receives feedback information of the test request and determines whether the master node is in a normal state according to the feedback information;
when the feedback information indicates that the master node is in an abnormal state, the slave node determines that the master node is in the abnormal state, and the slave node triggers a new master node determination process;
the slave node sends the data synchronization request to the master node, and the data synchronization request comprises: and when the main node is in a normal state, the slave node sends the synchronous data request to the main node.
5. The data storage method of claim 4, wherein the new master node determination process comprises:
the slave node sends a voting request for selecting a new master node to other slave nodes;
the slave node receives the voting results of the responses of the other slave nodes to the voting requests, wherein the voting results comprise that the slave node which sends out the voting requests is a new master node and the slave node which does not send out the voting requests is a new master node;
and when the voting result is that the number of the slave nodes which agree to send out the voting request to be the new master node exceeds half of the total number of the slave nodes, selecting the slave nodes as the new master nodes.
6. The data storage method according to claim 5, wherein the voting request carries a first data number with a largest data number of the slave node, and after the slave node sends a voting request for selecting a new master node to another slave node, the method further comprises:
the other slave nodes receive the voting requests sent by the slave nodes;
the other slave nodes compare the first data number with the current maximum second data number of the other slave nodes;
when the second data number does not exceed the first data number, the other slave nodes send voting results which enable the slave node sending the voting request to be a new master node to the slave nodes;
and when the second data number exceeds the first data number, the other slave nodes send voting results which do not agree that the slave node sending the voting request is a new master node to the slave nodes.
7. The data storage method of claim 6, further comprising:
and when the second data number does not exceed the first data number and the other slave nodes receive at least two voting requests within a preset time interval, the other slave nodes send voting results which enable the slave node sending the voting request to be a new master node to the slave node sending the voting request for the first time.
8. The data storage method of claim 1, further comprising:
the slave node sends a data detection request to the master node, wherein the data detection request is used for acquiring the maximum target data number in the data numbers of the master node;
the slave node receives the target data number sent by the master node;
and when the target data number exceeds the maximum current data number in the data numbers of the slave nodes, the slave nodes determine that the data of the slave nodes are missing, and acquire the missing data from the master node.
9. The data storage method according to claim 8, wherein the determining, by the slave node, that there is a missing data of the slave node and obtaining the missing data from the master node includes:
the slave node determines that data of the slave node is missing;
the slave node sends a first data acquisition request to the master node, wherein the first data acquisition request carries the current data number of the slave node;
and the slave node receives the data which is sent by the master node according to the current data number and is missing from the slave node.
10. The data storage method of claim 1, wherein when the distributed data system comprises at least two master nodes, the method further comprises:
when the main node receives a data writing request, the main node sends the synchronous data to the other nodes;
when other main nodes in other nodes receive the synchronous data, the main node determines the identities of the other main nodes and acquires the identity identifications of the other main nodes.
11. The data storage method of claim 10, further comprising:
and when the master node receives the synchronous data sent by the other master nodes, the master node determines the identities of the other master nodes and acquires the identity identifications of the other master nodes.
12. The data storage method according to claim 10 or 11, wherein after the master node determines the identities of other master nodes and obtains the identities of the other master nodes, the method further comprises:
the main node sends the data comparison request to other main nodes according to the identity marks of the other main nodes, wherein the data comparison request is used for acquiring a first target data number with the largest data number of the other main nodes;
the main node receives first target data numbers sent by other main nodes;
and when the first target data number exceeds a second target data number which is the largest in the data numbers of the main nodes, the main nodes become slave nodes, and the data which are missing from the main nodes are obtained from other main nodes.
13. The data storage method of claim 12, wherein the master node obtaining the data missing from the other master nodes comprises:
the main node determines that data is missing;
the main node sends a second data acquisition request to the other main nodes, wherein the second data acquisition request carries a second target data number of the main node;
and the master node receives the data which are lost by the master node and are sent by the other master nodes according to the second target data number.
14. The data storage method of claim 1, further comprising:
the main node receives a data writing request sent by the terminal;
the main node generates a data number of the written data according to the data writing request;
the master node sends the written data and the data number to slave nodes of the distributed data system as synchronous data and receives feedback information sent by the slave nodes;
the master node determines the number of second nodes of the slave nodes which successfully receive the synchronous data according to the feedback information;
when the number of the second nodes exceeds half of the total number of the nodes of the distributed data system, the master node stores the synchronous data to a local database of the master node and sends the data storage request to the slave node so as to store the synchronous data to the local database of the slave node;
when the second number of nodes does not exceed half of the total number of nodes of the distributed data system, the master node discards the write data and sends the data discard request to the slave node, so that the slave node discards the synchronization data.
15. A method of storing data, comprising:
a receiving unit, configured to receive, by the slave node, a data write request sent by a terminal, where the data write request carries write data, and the data write request is used to write the write data into a node of the distributed data system;
an obtaining unit, configured to obtain, by the slave node, a data number of the write data from the master node according to the data write request, where the data number is a sequential identifier of the write data in the distributed data system;
the sending unit is used for sending the written data and the data number as synchronous data to other nodes of the distributed data system by the slave node and receiving feedback information sent by the other nodes;
a determining unit, configured to determine, by the slave node, a first node number of other nodes that successfully receive the synchronization data according to the feedback information;
and the storage unit is used for storing the write data by the slave node based on the first node number.
CN202010004352.XA 2020-01-03 2020-01-03 Data storage method, device and computer readable storage medium Active CN111198662B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010004352.XA CN111198662B (en) 2020-01-03 2020-01-03 Data storage method, device and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010004352.XA CN111198662B (en) 2020-01-03 2020-01-03 Data storage method, device and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN111198662A true CN111198662A (en) 2020-05-26
CN111198662B CN111198662B (en) 2023-07-14

Family

ID=70746679

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010004352.XA Active CN111198662B (en) 2020-01-03 2020-01-03 Data storage method, device and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN111198662B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111625601A (en) * 2020-05-27 2020-09-04 北京字节跳动网络技术有限公司 Data writing method and device, computer equipment and storage medium
CN112068997A (en) * 2020-09-09 2020-12-11 恒生电子股份有限公司 Data backup method, device, equipment and storage medium
CN112527901A (en) * 2020-12-10 2021-03-19 杭州比智科技有限公司 Data storage system, method, computing device and computer storage medium
CN113282334A (en) * 2021-06-07 2021-08-20 深圳华锐金融技术股份有限公司 Method and device for recovering software defects, computer equipment and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104933132A (en) * 2015-06-12 2015-09-23 广州巨杉软件开发有限公司 Distributed database weighted voting method based on operating sequence number
US20170168756A1 (en) * 2014-07-29 2017-06-15 Hewlett Packard Enterprise Development Lp Storage transactions
CN107295080A (en) * 2017-06-19 2017-10-24 北京百度网讯科技有限公司 Date storage method and server applied to distributed server cluster
JP2017224253A (en) * 2016-06-17 2017-12-21 富士通株式会社 Parallel processor and memory cache control method
US20190042659A1 (en) * 2016-02-17 2019-02-07 Hangzhou Hikvision Technology Co. Ltd Data writing and reading and apparatus and cloud storage system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170168756A1 (en) * 2014-07-29 2017-06-15 Hewlett Packard Enterprise Development Lp Storage transactions
CN104933132A (en) * 2015-06-12 2015-09-23 广州巨杉软件开发有限公司 Distributed database weighted voting method based on operating sequence number
US20190042659A1 (en) * 2016-02-17 2019-02-07 Hangzhou Hikvision Technology Co. Ltd Data writing and reading and apparatus and cloud storage system
JP2017224253A (en) * 2016-06-17 2017-12-21 富士通株式会社 Parallel processor and memory cache control method
CN107295080A (en) * 2017-06-19 2017-10-24 北京百度网讯科技有限公司 Date storage method and server applied to distributed server cluster

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111625601A (en) * 2020-05-27 2020-09-04 北京字节跳动网络技术有限公司 Data writing method and device, computer equipment and storage medium
CN111625601B (en) * 2020-05-27 2023-03-14 北京火山引擎科技有限公司 Data writing method and device, computer equipment and storage medium
CN112068997A (en) * 2020-09-09 2020-12-11 恒生电子股份有限公司 Data backup method, device, equipment and storage medium
CN112068997B (en) * 2020-09-09 2023-12-19 恒生电子股份有限公司 Data backup method, device, equipment and storage medium
CN112527901A (en) * 2020-12-10 2021-03-19 杭州比智科技有限公司 Data storage system, method, computing device and computer storage medium
CN113282334A (en) * 2021-06-07 2021-08-20 深圳华锐金融技术股份有限公司 Method and device for recovering software defects, computer equipment and storage medium

Also Published As

Publication number Publication date
CN111198662B (en) 2023-07-14

Similar Documents

Publication Publication Date Title
CN111198662A (en) Data storage method and device and computer readable storage medium
US11360854B2 (en) Storage cluster configuration change method, storage cluster, and computer system
CN108847982B (en) Distributed storage cluster and node fault switching method and device thereof
EP2691859B1 (en) Fault detection and recovery as a service
US10331625B2 (en) Managing sequential data store
US7539150B2 (en) Node discovery and communications in a network
CN106960060B (en) Database cluster management method and device
WO2016183967A1 (en) Failure alarm method and apparatus for key component, and big data management system
CN112506702B (en) Disaster recovery method, device, equipment and storage medium for data center
CN111541762B (en) Data processing method, management server, device and storage medium
CN110351313B (en) Data caching method, device, equipment and storage medium
CN108140035B (en) Database replication method and device for distributed system
CN114238495A (en) Method and device for switching main cluster and standby cluster of database, computer equipment and storage medium
CN112000735A (en) Data processing method, device and system
CN113342893B (en) Node synchronization method and device based on block chain, storage medium and server
CN111651320A (en) High-concurrency connection method and system
CN105323271B (en) Cloud computing system and processing method and device thereof
CN113765690A (en) Cluster switching method, system, device, terminal, server and storage medium
CN111092956A (en) Resource synchronization method, device, storage medium and equipment
CN114564458B (en) Method, device, equipment and storage medium for synchronizing data among clusters
CN112685486B (en) Data management method and device for database cluster, electronic equipment and storage medium
Lin et al. ReHRS: A hybrid redundant system for improving MapReduce reliability and availability
US11381642B2 (en) Distributed storage system suitable for sensor data
CN113364874A (en) Node synchronization method and device based on block chain, storage medium and server
CN117395263B (en) Data synchronization method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20201010

Address after: Room 2012, building 12, 13, 15, fangmaoyuan (phase II), 1177 Huanhu Road, Tianding street, Yuelu District, Changsha City, Hunan Province

Applicant after: Tencent cloud computing (Changsha) Co.,Ltd.

Address before: 518057 Nanshan District science and technology zone, Guangdong, Zhejiang Province, science and technology in the Tencent Building on the 1st floor of the 35 layer

Applicant before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd.

GR01 Patent grant
GR01 Patent grant