CN114090551A

CN114090551A - Method, device and system for storing data

Info

Publication number: CN114090551A
Application number: CN202010747359.0A
Authority: CN
Inventors: 刘强
Original assignee: Huawei Cloud Computing Technologies Co Ltd
Current assignee: Huawei Cloud Computing Technologies Co Ltd
Priority date: 2020-07-29
Filing date: 2020-07-29
Publication date: 2022-02-25

Abstract

The embodiment of the application discloses a method, a device and a system for storing data, relates to the field of computers, and reduces the storage cost of the system on the premise of ensuring the availability of a distributed data service system. The specific scheme is as follows: configuring a permanent server and a temporary server for a replication group, wherein the permanent server is used for storing data for a long time, the temporary server is used for temporarily storing data, and the replication group is used for writing the same data into the servers contained in the replication group by adopting a replication protocol; each persistent server in the replication group respectively executes updating operation, and updates the storage data meeting the first condition in the storage data of the server into a union set of the data meeting the first condition and stored in all the servers in the replication group; after each temporary server in the replication group respectively determines that the data meeting the second condition in the stored data of the temporary server per se is stored in the persistent server in the replication group, each temporary server respectively executes deletion operation to delete the data meeting the second condition.

Description

Method, device and system for storing data

Technical Field

The embodiment of the application relates to the technical field of information, in particular to a method, a device and a system for storing data.

Background

With the development of internet technology and storage technology, distributed data service systems are widely used. A distributed data services system consists of a distributed deployment of multiple servers (which may also be referred to as nodes). A user communicates with the nodes through the client, writes data to the nodes, and reads data from each node in the replica group.

In the distributed data service system, the availability is an important index for measuring the reliability of the distributed data service system, and the availability is the probability of successful data access, and determines the number of fault nodes tolerated when the system can successfully access. To avoid partial node failure leading to unavailability of the entire system, a replication protocol is typically employed to replicate data to multiple nodes, which may be referred to as a replication group. Thus, even if part of nodes in the replication group fail, the rest nodes can still continue to provide services, and the high availability of the system is ensured.

Currently common replication protocols include a master replication protocol and a masterless replication protocol. The master replication protocol scheme is that a master node is selected from a replication group, other nodes in the replication group are used as slave nodes, data are written into the master node when the data are written, the master node sequentially sends the data to the slave nodes for writing, the slave nodes which are successfully written send writing success responses to the master node, the master node counts the number of the nodes which are successfully written, and when the number is larger than a first threshold value, the writing is considered to be successful. The nodes included in the replication group of the system without the main replication protocol have the same functions, when data are written, a user sends the data to the nodes in the replication group through a client to write, the nodes with successful writing send writing success responses to the client, the client counts the number of the nodes with successful writing, and when the number is larger than a first threshold value, the writing is considered to be successful.

When the replication protocol stores data, all nodes in the replication group are used for storing data input by a user, and in order to ensure complete storage of the user data, a large-capacity server needs to be configured as a node in the replication group, so that although the availability of the distributed data service system is improved, the equipment cost of the large-capacity server is high, which results in high storage cost of the distributed data service system. When the system availability requirement is high, the number of nodes in the copy group will be large, and the storage cost of the system will be huge.

Disclosure of Invention

The application provides a method, a device and a system for storing data, which reduce the storage cost of a distributed data service system on the premise of ensuring the availability of the distributed data service system.

In order to achieve the purpose, the following technical scheme is adopted in the application:

in a first aspect, a method for storing data is provided, which is applied to a replication group comprising one or more persistent servers and one or more temporary servers; the system comprises a persistent server, a temporary server and a replication group, wherein the persistent server is used for storing data for a long time, the temporary server is used for temporarily storing data, and the replication group is used for writing the same data into a server contained in the replication group by adopting a replication protocol; the method can comprise the following steps: each persistent server respectively executes updating operation, and updates the stored data meeting the first condition in the stored data of the server into a union set of the data meeting the first condition and stored in all servers in the replication group; after each temporary server respectively determines that the data meeting the second condition in the stored data of the temporary server is stored in the persistent server in the replication group, each temporary server respectively executes deletion operation to delete the data meeting the second condition.

According to the data storage method provided by the application, a persistent server for storing data for a long time and a temporary server for temporarily storing the data are configured in the replication group, after the data are successfully written in or periodically, data synchronization is carried out on the persistent server, the data in the persistent server comprise the stored data of all servers in the replication group to which the persistent server belongs, and then the data in the temporary server are deleted. Therefore, the temporary servers in the copy group are only used for temporarily storing data, and a small-capacity server can be configured to be used as the temporary server, so that the equipment cost is reduced on the premise of ensuring the availability of the system. Even if the system availability demand is further increased, the system availability demand is secured by increasing the number of temporary servers without greatly increasing the equipment cost.

The first condition and the second condition may be configured according to actual requirements, which is not specifically limited in the embodiments of the present application.

With reference to the first aspect, in one possible implementation manner, the first condition may include: the data written in the previous N times, wherein N is greater than or equal to 1; or, the data written in the first preset time before the current time. In the possible implementation mode, the persistent server can update data under different first conditions, so that the flexibility of updating operation and the flexibility of data storage are improved.

With reference to the first aspect and the foregoing one possible implementation manner, in another possible implementation manner, in one possible implementation manner, the second condition may include: data written for the first M times, wherein M is greater than or equal to 1; or, the data written in the second preset time before the current time. In this possible implementation manner, the temporary server may delete data under a different second condition, which improves flexibility of the deletion operation and flexibility of data storage.

With reference to the first aspect and any one of the foregoing possible implementation manners, in another possible implementation manner, a replication group uses a master replication protocol, for a first persistent server, the first persistent service is any one persistent server in the replication group, the first persistent server is a slave node, and the first persistent server performs an update operation to update storage data meeting a first condition in storage data of the first persistent server to a union of data meeting the first condition and stored in all servers in the replication group, including: if the master nodes are in normal communication, the first persistent server executes updating operation, and updates the storage data meeting the first condition in the storage data of the first persistent server into the data meeting the first condition stored in the master nodes in the copy group; or if the main node fails, the first persistent server executes updating operation, and writes data which are not stored in the first persistent server in the data which are stored in all the servers in the copy group and meet the first condition; or if the master node fails and the newly selected master node is the first persistent server, the first persistent server executes updating operation and writes data which are not stored in the first persistent server in the data which are stored in all the servers in the copy group and meet the first condition; the first persistent server sends an update completion instruction to other persistent servers in the replication group; the update completion instruction is used for indicating the main node to complete the update operation; or if the master node is failed and the newly selected master node is a persistent server other than the first persistent server, the first persistent server updates the stored data meeting the first condition in the stored data of the first persistent server to the data meeting the first condition stored in the newly selected master node after the newly selected master node executes the updating operation. In this possible implementation manner, a method for updating data by a persistent server under different conditions of the master replication protocol is specifically described, and different updating methods are adopted for different conditions, so that the reliability of stored data is improved.

With reference to the first aspect and any one of the foregoing possible implementations, in another possible implementation, the number of servers in a replication group may be determined by a system availability, and the number of persistent servers in the replication group may be determined by a system persistence. In the possible implementation mode, the number of the persistent servers and the number of the temporary servers in the copy group are determined by the availability index and the persistence index of the system, so that the reliability of storage is not influenced while the equipment cost is reduced.

With reference to the first aspect and any one of the foregoing possible implementation manners, in another possible implementation manner, the persistent server may perform an update operation after the write operation; alternatively, the persistent server periodically performs the update operation. In the possible implementation mode, different modes for executing the updating operation by the persistent server are configured, so that the flexibility of data storage is improved.

With reference to the first aspect and any one of the foregoing possible implementation manners, in another possible implementation manner, if a persistent server in a replication group fails, the method may further include: adding a new persistent server to the replication group; the new persistent server writes the union of the data stored in all servers in the replica group. In the possible implementation mode, a processing method for the failure of the persistent server is described, and the reliability of the stored data is improved.

In a second aspect, an apparatus for storing data is provided, which may be deployed in a persistent server in a replication group, where the replication group is configured to write the same data to servers included in the replication group by using a replication protocol, and the persistent server is configured to store data for a long time; the apparatus may include: a storage unit, a write unit and an update unit. Wherein:

and the storage unit is used for storing the data written into the persistent server.

And the writing unit is used for writing the data written into the persistent server into the storage unit of the persistent server.

And the updating unit is used for executing updating operation and updating the storage data meeting the first condition in the storage data in the storage unit into a union set of the data meeting the first condition and stored in all the servers in the copy group.

According to the data storage device, a persistent server for storing data for a long time and a temporary server for temporarily storing the data are configured in the replication group, after the data are successfully written or periodically, data synchronization is carried out on the persistent server, the data in the persistent server comprise the stored data of all servers in the replication group to which the persistent server belongs, and then the data in the temporary server are deleted. Therefore, the temporary servers in the copy group are only used for temporarily storing data, and a small-capacity server can be configured to be used as the temporary server, so that the equipment cost is reduced on the premise of ensuring the availability of the system. Even if the system availability demand is further increased, the system availability demand is secured by increasing the number of temporary servers without greatly increasing the equipment cost.

With reference to the second aspect, in one possible implementation manner, the first condition may include: the data written in the previous N times, wherein N is greater than or equal to 1; or, the data written in the first preset time before the current time.

In a third aspect, another apparatus for storing data is provided, where the apparatus is deployed in a temporary server in a replication group, the replication group may further include one or more persistent servers, the persistent servers are configured to store data for a long time, the temporary server is configured to store data temporarily, and the replication group is configured to write the same data to servers included in the replication group by using a replication protocol; the apparatus may include: the device comprises a storage unit, a writing unit and a processing unit. Wherein:

and the storage unit is used for storing the data written into the temporary server.

And the writing unit is used for writing the data written into the temporary server into the storage unit of the temporary server.

And the processing unit is used for executing deletion operation and deleting the data meeting the second condition after determining that the data meeting the second condition in the storage data of the storage unit is stored in the persistent server in the copy group.

With reference to the third aspect, in one possible implementation manner, the second condition may include: data written for the first M times, wherein M is greater than or equal to 1; or, the data written in the second preset time before the current time.

In a fourth aspect, a system for storing data is provided, the system may include a replication group, the replication group including one or more persistent servers and one or more temporary servers; the system comprises a persistent server, a temporary server and a replication group, wherein the persistent server is used for storing data for a long time, the temporary server is used for temporarily storing data, and the replication group is used for writing the same data into a server contained in the replication group by adopting a replication protocol; wherein:

the persistent server is to: and executing an updating operation, and updating the storage data meeting the first condition in the storage data of the server into a union set of the data meeting the first condition and stored in all the servers in the copy group.

The temporary server is used for: and after determining that the data meeting the second condition in the self storage data is stored in the persistent server in the copy group, each temporary server respectively executes deletion operation to delete the data meeting the second condition.

According to the data storage system, the persistent server for storing data for a long time and the temporary server for temporarily storing the data are configured in the replication group, after the data are successfully written or periodically, data synchronization is carried out on the persistent server, the data in the persistent server comprise the stored data of all servers in the replication group to which the persistent server belongs, and then the data in the temporary server are deleted. Therefore, the temporary servers in the copy group are only used for temporarily storing data, and a small-capacity server can be configured to be used as the temporary server, so that the equipment cost is reduced on the premise of ensuring the availability of the system. Even if the system availability demand is further increased, the system availability demand is secured by increasing the number of temporary servers without greatly increasing the equipment cost.

In a fifth aspect, a server is provided, which may comprise a processor and a memory coupled to the processor, the memory being connected to the processor, the memory being configured to store a computer program, which when executed by the processor, performs the actions of the persistent server of the first aspect or any of the possible implementations of the first aspect.

In a sixth aspect, a server is provided, which may comprise a processor and a memory coupled to the processor, the memory being connected to the processor, the memory being configured to store a computer program, which when executed by the processor, performs the actions of the temporary server in the first aspect or any one of the possible implementations of the first aspect.

In a seventh aspect, an embodiment of the present application provides a chip system, where the chip system includes a processor and may further include a memory, and is configured to implement the function of the persistent server in the foregoing method. The chip system may be formed by a chip, and may also include a chip and other discrete devices.

In an eighth aspect, an embodiment of the present application provides a chip system, where the chip system includes a processor and may further include a memory, and is configured to implement the function of the temporary server in the foregoing method. The chip system may be formed by a chip, and may also include a chip and other discrete devices.

In a ninth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes instructions, when executed on a computer, causing the computer to perform the functions of the persistent server or the temporary server in the method for storing data in any one of the above aspects or any one of the possible implementations.

In a tenth aspect, an embodiment of the present application further provides a computer program product, which when run on a computer, causes the computer to perform the functions of the persistent server or the temporary server in the method for storing data in any one of the above aspects or any one of the possible implementations.

It should be noted that, on the premise of not contradicting the scheme, various possible implementation manners of any one of the above aspects may be combined.

Drawings

Fig. 1 is a schematic structural diagram of a distributed data service system provided in the prior art;

fig. 2 is a schematic structural diagram of a distributed data service system according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of a server according to an embodiment of the present application;

fig. 4 is a schematic flowchart of a method for storing data according to an embodiment of the present application;

FIG. 5 is a schematic flow chart illustrating another method for storing data according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of another distributed data service system provided in an embodiment of the present application;

fig. 7 is a schematic structural diagram of an apparatus for storing data according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of another server provided in the embodiment of the present application;

FIG. 9 is a schematic structural diagram of another apparatus for storing data according to an embodiment of the present disclosure;

fig. 10 is a schematic structural diagram of another server according to an embodiment of the present application.

Detailed Description

The terms "first," "second," and "third," etc. in the description and claims of this application and the above-described drawings are used for distinguishing between different objects and not for limiting a particular order.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present relevant concepts in a concrete fashion for ease of understanding.

In the description of the present application, a "/" indicates a relationship in which the objects associated before and after are an "or", for example, a/B may indicate a or B; in the present application, "and/or" is only an association relationship describing an associated object, and means that there may be three relationships, for example, a and/or B, and may mean: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. Also, in the description of the present application, "a plurality" means two or more than two unless otherwise specified. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or multiple.

In the embodiments of the present application, at least one may also be described as one or more, and a plurality may be two, three, four or more, which is not limited in the present application.

For the sake of understanding, the technical terms related to the present application are explained first.

A replica group, may refer to a collection of multiple servers deployed in a distributed data services system to store the same data.

Persistent server, may refer to a server in a replication group that is used for long-term storage of data. A large capacity server may be configured as a persistent server.

A temporary server may refer to a server in a replication group that is used for short-term storage of data. A small capacity server may be configured as a temporary server.

The storage scheme of the master replication protocol and the storage scheme without the master replication protocol, which are commonly used in the distributed data service system, are briefly introduced. As shown in FIG. 1, a distributed data services system may include one or more clients, as well as a plurality of servers. Wherein, a plurality of servers can be used as a copy group to write and read data.

First, a procedure of configuring a copy group is described: the user calculates the number of servers that a replication group in the system can include according to the availability index of the distributed data service system, then configures the number of servers for each replication group in the system, and configures the configuration information of the replication group according to the identification of the number of servers. For the masterless replication protocol, the centralized control unit (which may be deployed at any node) obtains the identities of the servers, and stores the identities of the servers as configuration information of the replication group in a preset location of each server in the replication group. For the master replication protocol, the centralized control unit selects one node from a plurality of nodes included in the replication group as a master node, and other nodes as slave nodes; the master node acquires the identifiers of all the nodes in the copy group, and stores the identifiers of the nodes included in the copy group and the states of the nodes (master nodes or slave nodes) as configuration information of the copy group in a preset position of each node in the copy group.

For example, the configuration information of the replication group in the masterless replication protocol may be: server 1, server 2, server 3, server 4, and server 5.

For another example, the configuration information of the replication group in the master replication protocol may be: the master node includes a server 1; the slave nodes include server 2, server 3, server 4, and server 5.

The writing process of the data may include: the user sends a writing request and an identifier of data to be written to a centralized control unit of the distributed data service system through a client, the centralized control unit receives the writing request, allocates a replication group for the data to be written, sends the allocated replication group identifier to the client, and the client receives the allocated replication group identifier and writes the data to be written into the replication group indicated by the identifier. Specifically, for a master replication protocol, a client sends data to be written to a master node, the master node receives and writes the data to be written, then the master node sends the data to be written to other slave nodes in the replication group, the slave nodes receive and write the data, the slave nodes which are successfully written send write success responses to the master node, the master node counts the number of the nodes which are successfully written, when the number is greater than a first threshold value, it is determined that the data is successfully written, and the master node sends a write success response to the client. Specifically, for the non-master replication protocol, the client directly sends data to be written to all nodes in the replication group, all nodes in the replication group receive and write the data to be written, the nodes with successful writing send write success responses to the client, the client counts the number of nodes with successful writing, and when the number is greater than a first threshold value, it is determined that the data is successfully written.

The reading process of the data may include: the user sends a reading request and an identifier of data to be read to a centralized control unit of the distributed data service system through the client, the centralized control unit receives the reading request, searches a copy group to which the reading request belongs, sends the copy group identifier to which the reading request belongs to the client, and the client receives the copy group identifier and reads the data to be read from the copy group indicated by the identifier. Specifically, for the master replication protocol, the client sends an identifier of data to be read to the master node in the replication group, the master node receives the identifier of the data to be read and sends the data to be read to the client, and if the client receives the data to be read sent by the master node, the client considers that the reading is successful. For the masterless replication protocol, the client sends the identifier of the data to be read to all the nodes in the replication group, the nodes receive the identifier and search the data to be read indicated by the identifier, the data to be read searched by the nodes is sent to the client, the client counts the number of the received data to be read, and if the number is greater than a second threshold value, the reading is considered to be successful.

If a node in the replica group fails, the configuration information of the replica group stored in each node needs to be updated. Specifically, all servers in the replication group detect whether the communication between the nodes is normal through a heartbeat mechanism, and for a non-master replication protocol, if a node a in the replication group does not receive a heartbeat test signal sent by a node B within a preset time period, it is determined that the node B fails, the node a adds a new node C to the replication group, updates the configuration information of the replication group, and the updated configuration information includes an identifier of the node C and does not include an identifier of the node B.

For a master replication protocol, if a slave node D in a replication group detects that a slave node E in the replication group fails, the slave node D sends a failure request to a master node, the request comprises an identifier of the node E, the master node receives the request, configures a new slave node F for the replication group, and updates configuration information of the replication group, wherein the updated configuration information comprises the identifier of the slave node F and does not comprise the identifier of the slave node E; if the master node in the replication group detects that the slave node E in the replication group fails, the master node configures a new node F for the replication group, and updates the configuration information of the replication group, wherein the updated configuration information contains the identifier of the slave node F and does not contain the identifier of the slave node E; if the slave node D in the replication group detects that the master node in the replication group is failed, the slave node D initiates voting to other slave nodes in the replication group, all the slave nodes in the replication group determine a new master node according to a voting mechanism, then the new master node adds a new slave node F to the replication group, configuration information of the replication group is updated, and the updated configuration information contains an identifier of the new master node, an identifier of the new slave node F and an identifier of the failed master node.

However, when the above replication protocol stores data, all nodes in the replication group are used for storing data input by a user, and in order to ensure complete storage of user data, a large-capacity server needs to be configured as a node in the replication group, so that although the availability of the distributed data service system is improved, the cost of equipment of the large-capacity server is high, which results in high storage cost of the distributed data service system. When the system availability requirement is high, the number of nodes in the copy group will be large, and the storage cost of the system will be huge.

Based on the above, the present application provides a method for storing data, where a persistent server for storing data for a long time and a temporary server for temporarily storing data are configured in a replication group, and after data is successfully written in or periodically, data synchronization is performed on the persistent server, so that the data in the persistent server includes the stored data of all servers in the replication group to which the persistent server belongs, and then the data in the temporary server is deleted. Therefore, the temporary servers in the copy group are only used for temporarily storing data, and a small-capacity server can be configured to be used as the temporary server, so that the equipment cost is reduced on the premise of ensuring the availability of the system. Even if the system availability demand is further increased, the system availability demand is secured by increasing the number of temporary servers without greatly increasing the equipment cost.

Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The method for storing data provided by the embodiment of the present application may be applied to the distributed data service system 20 shown in fig. 2, and the distributed data service system 20 may include one or more clients 201, one or more replication groups 202, and a centralized management device 203. One replication group 202 may include one or more persistent servers 2021, and one or more transient servers 2022.

It should be noted that the servers included in the replication group 202 may be servers in a distributed deployment.

The client 201 may be configured to receive input data of a user and transmit the received input data to a server included in the replication group 202 assigned by the central management apparatus 203. The input data may be an identifier of data to be read or data to be written, which is input by a user.

Illustratively, the client 201 may be a notebook, server, tablet, or the like having associated input and transmission functionality.

The centralized management device 203 may be used to configure one or more persistent servers 2021 and one or more temporary servers 2022 for the replication group 202.

The centralized management device 203 may also be used to assign a replica group 202 to the input data. For example, when the input data is an identifier of data to be read, the centralized management device 203 is configured to search the copy group 202 to which the data to be read indicated by the identifier belongs. For another example, when the input data is data to be written, the centralized management device 203 is configured to allocate a copy group 202 for storing the data to be written to the data to be written.

Specifically, the centralized management device 203 may be or be deployed in any designated server in the distributed data service system 20, or the centralized management device 203 may also be deployed alone, which is not specifically limited in this embodiment of the present application.

Persistent server 2021 may be used to receive input data and provide specific services based on the input data. For example, when the input data is data to be written, the persistent server 2021 may be configured to receive the data to be written and write the data to be written into the persistent server for long-term storage. For another example, when the input data is an identifier of data to be read, the persistent server 2021 may be configured to receive the identifier of the data to be read, search for the data to be read indicated by the identifier, and send the data to be read to the client 201.

Persistent server 2021 may also be used to perform update operations.

Illustratively, the persistent server 2021 may be a server having a large storage space.

The temporary server 2022 may be used to receive input data and provide specific services based on the input data. For example, when the input data is data to be written, the temporary server 2022 may be configured to receive the data to be written, write the data to be written therein, and temporarily store the data.

The temporary server 2022 may also be used to perform delete operations. Specifically, after the temporary server 2022 determines that data satisfying the second condition among its own stored data has been stored in the persistent server 2021 in the copy group 202, the temporary server 2022 performs a deletion operation.

Specifically, the temporary server 2022 may be a small capacity server.

Now, taking the replication group 202 as an example, the configuration of the replication group under different replication protocols will be described.

The configuration process of the copy group comprises the following steps: first, the centralized control unit 203 configures a plurality of persistent servers 2021 and a plurality of temporary servers 2022 to the copy group 202. The centralized control unit 203 then configures the configuration information of the copy group based on the identities of the plurality of persistent servers 2021, the identities of the plurality of temporary servers 2022, and the modality of each server (master or slave node or node).

For the copy group 202, the data writing process is similar to the data writing process, and the data reading process is similar to the data reading process, and the data reading and writing are specifically implemented by referring to the data reading and writing, which is not described herein again.

The following describes a method and an apparatus for storing data according to an embodiment of the present application with reference to the accompanying drawings.

In one aspect, the embodiment of the present application provides a server 30 for executing the method for storing data provided by the present application, where the server 30 may be a persistent server or a temporary server in the replication group 202 illustrated in fig. 2.

As shown in fig. 3, the server 30 may include a processor 301, a memory 302, and a transceiver 303.

The following describes each component of the server 30 in detail with reference to fig. 3:

the memory 302 may be a volatile memory (volatile memory), such as a random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory (flash memory), a Hard Disk Drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, for storing program code, configuration files, or other content that may implement the methods of the present application.

Processor 301 is the control center for server 30. For example, the processor 301 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as: one or more microprocessors (digital signal processors, DSPs), or one or more Field Programmable Gate Arrays (FPGAs).

The transceiver 303 is used to communicate with other devices. The transceiver 303 may be a communication port or the like.

In one possible implementation, where the server 30 is a persistent server, the processor 301 performs the following functions by running or executing software programs and/or modules stored in the memory 302 and calling stored data stored in the memory 302:

and updating the storage data meeting the first condition in the storage data of the server into a union set of the data meeting the first condition and stored in all the servers in the copy group.

In another possible implementation, the server 30 is a temporary server, and the processor 301 executes or executes the software programs and/or modules stored in the memory 302 and calls the stored data stored in the memory 302 to perform the following functions:

and after determining that the data meeting the second condition in the self storage data is stored in the persistent server in the copy group, the temporary server executes deletion operation to delete the data meeting the second condition.

On the other hand, the method for storing data provided by the embodiment of the application can be applied to a replication group comprising one or more permanent servers and one or more temporary servers.

It should be noted that the method for storing data provided by the embodiment of the present application may be applied to any replication group of the distributed data service system. The implementation process of storing data in each copy group is similar, and a copy group is taken as an example to describe the method for storing data provided in the embodiment of the present application, and the rest is not described again.

Before storing data, a copy set needs to be configured first. The configuration process will now be described in detail.

The user system is pre-configured with a certain number of persistent servers and a certain number of temporary servers. And the user determines the number P of the persistent servers and the number Q of the temporary servers in the copy group according to the parameter index which needs to be met by the system. In specific configuration, the centralized control unit selects P persistent servers and Q temporary servers meeting a first rule as a replication group from a certain number of persistent servers and a certain number of temporary servers configured in advance by the system, and then the centralized control unit acquires configuration information when the replication group adopts different replication protocols and stores the configuration information in a fixed position of each server included in the replication group.

The first rule may be configured according to actual requirements, which is not limited in the embodiment of the present application. Specifically, the first rule may be statically configured in the design stage of the copy group, or the first rule may be set as configurable content, and the configuration may be modified by a user, an administrator, or other personnel in the use stage of the copy group.

In one possible implementation, the first rule may include: the remaining storage space is greater than a third threshold.

In another possible implementation manner, the first rule may include: the remaining storage space is greater than a third threshold and the remaining storage space is in the first interval.

The third threshold and the first interval may be configured according to actual requirements, which is not specifically limited in the embodiment of the present application.

Specific values of P and Q may be configured according to actual requirements, which is not limited in the embodiments of the present application. Specifically, P is greater than or equal to 1 and Q is greater than or equal to 1.

In one possible implementation, the number of servers in the replication group, O, is determined by the system availability, and the number of persistent servers in the replication group, P, is determined by the system persistence. Specifically, the user can calculate, according to the index of the persistence that the system needs to satisfy, the number of nodes that should be successfully written in when the index of the persistence is satisfied, which is taken as the number P of persistent servers in the replication group. The user can calculate the number of nodes that should be written in at least when the availability index is satisfied, as the total number O of servers in the replication group, according to the availability index that needs to be satisfied. The number of temporary servers in the replica group Q is made by subtracting the number of persistent servers P from the total number of servers in the replica group O.

Of course, the user may further determine the number of persistent servers and the number of temporary servers in the replication group according to other requirements or indexes that the system should meet, which is not limited in this embodiment of the present application.

The obtaining of the configuration information of the replication group using different replication protocols by the centralized control unit may be implemented as: if the replication group adopts a master replication protocol, the centralized control unit selects one persistent server from the P persistent servers as a master node, and the other persistent servers and the Q temporary servers as slave nodes, and then takes the identifications of the P persistent servers, the identifications of the Q temporary servers, the identifications of the servers included in the master node and the identifications of the servers included in the slave nodes as configuration information of the replication group. If the replication group adopts a master-less replication protocol, the centralized control unit takes the identifications of the P persistent servers and the identifications of the Q temporary servers as the configuration information of the replication group.

For example, the configuration information of the replication group of the master replication protocol may be: the persistent server includes a1, B1, and C1; the temporary server includes a1 and b 1; the master node includes a 1; the slave nodes include B1, C1, a1, and B1.

The configuration information of the copy group without the master copy protocol may be: the persistent server includes a2, B2, and C2; the temporary server includes a2 and b 2.

The method for storing data provided by the embodiment of the application is specifically applied to the method for storing data provided by the embodiment of the application after the data is written into the copy group, namely after the data is written into the copy group.

In a possible implementation manner, the method for storing data provided by the embodiment of the present application may be performed after each write operation.

In another possible implementation manner, the method for storing data provided in the embodiment of the present application may be performed after a preset number of write operations.

The preset times may be configured according to actual requirements, which is not limited in the embodiments of the present application.

For example, the preset number of times may be determined according to the frequency of data writing. For example, the larger the frequency of data writing, the smaller the preset number of times may be; the smaller the frequency of data writing, the larger the preset number of times may be.

In another possible implementation manner, the method for storing data provided by the embodiment of the present application may be periodically performed.

The period duration may be configured according to actual requirements, which is not limited in the embodiment of the present application.

For example, the duration of the period may be determined according to the frequency of data writing. For example, the greater the frequency of data writing, the smaller the duration of the period may be; the smaller the frequency of data writing, the larger the duration of the period may be.

The process of data writing may refer to the specific implementation process of data writing, which is not described herein any more.

As shown in fig. 4, a method for storing data provided in an embodiment of the present application may include:

s401, each persistent server respectively executes updating operation, and updates the storage data meeting the first condition in the storage data of each persistent server into a union set of the data meeting the first condition and stored in all the servers in the copy group.

The first condition may be configured according to actual requirements, which is not specifically limited in the embodiments of the present application. Specifically, the first condition may be statically configured in the design stage of the copy group, or the first condition may be set as configurable content, and the configuration may be modified by a user or an administrator or other personnel in the use stage of the copy group.

In one possible implementation, the first condition may include: data written the first N times.

Wherein N is greater than or equal to 1.

In another possible implementation manner, the first condition may include: and data written in a first preset time before the current time.

The first preset time period may be configured according to actual requirements, which is not specifically limited in the embodiment of the present application. Specifically, the first preset duration may be statically configured in a design stage of the copy group, or the first preset duration may also be set as configurable content, and the configuration is modified by a user, an administrator, or other personnel in a use stage of the copy group.

For example, the user may configure the first preset time period according to a writing frequency of data or a size of written data. For example, the higher the writing frequency of the data, the smaller the first preset time period.

In the specific implementation of S401, each persistent server in the replication group respectively executes an update operation, and the update process of each persistent server is similar, the implementation of S401 is now described by taking the first persistent server as an example, and the implementation of other persistent servers is not described in detail. Specifically, for the first persistent server, the implementation of S401 may include, but is not limited to, any of cases 1 to 5 described below.

In case 1, the replication group adopts a master replication protocol, the first persistent server is a slave node, and the master node normally communicates, and for the first persistent server, S401 may be implemented as: the first persistent server updates the storage data meeting the first condition in the storage data of the first persistent server into the data meeting the first condition and stored in the main node in the copy group; or the first persistent server writes data which is not stored in the first persistent server in the data which is stored in the main node in the copy group and meets the first condition.

Illustratively, a first persistent server detects that an identifier of a slave node in configuration information includes an identifier of the first persistent server by acquiring configuration information of a replication group stored in a fixed position of the first persistent server, and determines that the replication group adopts a master replication protocol, and the first persistent server is the slave node; if the first persistent server is in a preset time period, the heartbeat test information sent by the main node can be normally received, and the main node is determined to be in normal communication; then, the first persistent server acquires data which is stored in the main node in the copy group and meets the first condition, and updates the stored data which meets the first condition in the stored data of the first persistent server into the data which meets the first condition and is stored in the main node in the copy group; or the first persistent server acquires the data meeting the first condition and stored in the main node in the copy group, and writes the data which is not stored in the data meeting the first condition and stored in the main node in the copy group into the data.

It should be noted that, if the replication group adopts a master replication protocol, the first persistent server is a master node, and the master node communicates normally, and the master node may not perform the update operation.

In case 2, the replication group adopts a master replication protocol, the first persistent server is a slave node, and the master node fails, and for the first persistent server, S401 may be implemented as: the method comprises the steps that a first persistent server obtains data meeting a first condition and stored in all servers in a copy group, and data which are not stored in the first persistent server in the data meeting the first condition and stored in all the servers in the copy group are written in; or the first persistent server acquires a union set of the data meeting the first condition and stored in all the servers in the copy group, and the first persistent server updates the stored data meeting the first condition in the stored data of the first persistent server into the union set of the data meeting the first condition and stored in all the servers in the copy group.

Illustratively, a first persistent server detects that an identifier of a slave node in configuration information includes an identifier of the first persistent server by acquiring configuration information of a replication group stored in a fixed position of the first persistent server, and determines that the replication group adopts a master replication protocol, and the first persistent server is the slave node; if one or more servers in the replication group detect that the heartbeat test information sent by the main node cannot be normally received within a preset time period, determining that the main node fails; the method comprises the steps that a first persistent server writes data which are not stored in the first persistent server in data which are stored in all servers in a copy group and meet a first condition; or the first persistent server updates the storage data meeting the first condition in the storage data of the first persistent server to a union set of the data meeting the first condition stored in all the servers in the copy group.

In case 3, the replication group adopts a master replication protocol, the first persistent server is a slave node, the master node fails, and the newly selected master node is the first persistent server, and for the first persistent server, S401 may be implemented as: the first persistent server executes updating operation, and writes data which are not stored in the first persistent server in the data which are stored in all the servers in the copy group and meet the first condition; then, the first persistent server sends an update completion instruction to other persistent servers in the replication group to instruct the master node to complete the update operation, and the other persistent servers can execute the update operation.

Illustratively, a first persistent server detects that an identifier of a slave node in configuration information includes an identifier of the first persistent server by acquiring configuration information of a replication group stored in a fixed position of the first persistent server, and determines that the replication group adopts a master replication protocol, and the first persistent server is the slave node; if one or more servers in the replication group detect that heartbeat test information sent by the main node cannot be normally received within a preset time period, determining a communication fault between the main nodes; and then, the first persistent server initiates voting, voting information is sent to other slave nodes, the other slave nodes perform voting according to the voting information, a voting result is sent to the first persistent server, the first persistent server counts the voting result, the number of votes voting for the first persistent server as a main node is determined to be greater than a preset threshold value, and the newly selected main node is determined to be the first persistent server. At the moment, the first persistent server writes data which are not stored in the first persistent server in the data which are stored in all the servers in the copy group and meet the first condition; or the first persistent server updates the storage data meeting the first condition in the storage data of the first persistent server to a union set of the data meeting the first condition stored in all the servers in the copy group. Then, the first persistent server sends an update completion instruction to other persistent servers in the replication group to instruct the master node to complete the update operation, and the other persistent servers can execute the update operation.

In case 4, when the replication group adopts a master replication protocol, the first persistent server is a slave node, the master node fails, and the newly selected master node is a persistent server in the replication group other than the first persistent server, S401 may be implemented as: after the newly selected main node executes the updating operation, the first persistent server executes the updating operation, and the first persistent server updates the stored data meeting the first condition in the stored data of the first persistent server into the data meeting the first condition and stored in the newly selected main node; or the first persistent server writes data which are not stored in the first persistent server in the newly selected data which are stored in the main node and meet the first condition.

Illustratively, a first persistent server detects that an identifier of a slave node in configuration information includes an identifier of the first persistent server by acquiring configuration information of a replication group stored in a fixed position of the first persistent server, and determines that the replication group adopts a master replication protocol, and the first persistent server is the slave node; if one or more servers in the replication group cannot normally receive heartbeat test information sent by the main node within a preset time period, determining that the main node fails; and then, other nodes in the replication group initiate voting, and the newly selected main node is determined to be the other persistent server in the replication group except the first persistent server according to a voting mechanism. At the moment, after receiving an update completion instruction sent by a newly selected main node, the first persistent server determines that the newly selected main node executes an update operation, and updates the stored data meeting the first condition in the stored data of the first persistent server into the data meeting the first condition and stored in the newly selected main node; or the first persistent server writes data which are not stored in the first persistent server in the newly selected data which are stored in the main node and meet the first condition.

In case 5, the replication group adopts a master-less replication protocol, and for the first persistent server, S401 may be implemented as: the first persistent server updates the storage data meeting the first condition in the storage data of the first persistent server to a union set of the data meeting the first condition and stored in all the servers in the copy group, or the first persistent server writes the data which is not stored in the first persistent server in the data meeting the first condition and stored in all the servers in the copy group.

Illustratively, a first persistent server detects that server identifiers of a master node and a slave node are not included in configuration information by acquiring the configuration information of a copy group stored in a fixed position of the first persistent server, determines that the copy group adopts a master-free copy protocol, then the first persistent server acquires a union set of data which is stored in all servers in the copy group and meets a first condition, and updates the stored data which meets the first condition in the stored data of the first persistent server into the union set of the data which meets the first condition and is stored in all servers in the copy group; or the first storage server respectively acquires the data meeting the first condition and stored in each server in the copy group, and writes the data which is not stored in the first persistent server in the data meeting the first condition and stored in each server.

S402, after each temporary server respectively determines that the data meeting the second condition in the stored data of the temporary server is stored in the persistent server in the replication group, each temporary server respectively executes deletion operation to delete the data meeting the second condition.

The second condition may be configured according to actual requirements, which is not specifically limited in the embodiments of the present application. Specifically, the second condition may be statically configured in the design stage of the copy group, or the second condition may be set as configurable content, and the configuration may be modified by a user or an administrator or other personnel in the use stage of the copy group.

In one possible implementation, the second condition may include: data written M times before.

Wherein M is greater than or equal to 1.

In this possible implementation, M may be equal to N, and at this time, the frequency of performing the update operation may be the same as the frequency of performing the delete operation.

In this possible implementation, M may also be greater than N, i.e., the frequency of performing update operations may also be greater than the frequency of performing delete operations.

In another possible implementation, the second condition may include: and data written in a second preset time before the current time.

The second preset time period may be configured according to actual requirements, which is not specifically limited in the embodiment of the present application. Specifically, the second preset time period may be statically configured in the design stage of the copy group, or the second preset time period may also be set as configurable content, and the configuration is modified by a user, an administrator, or other personnel in the use stage of the copy group.

For example, the user may configure the second preset time period according to a writing frequency of data or a size of the written data. For example, the higher the writing frequency of the data, the smaller the second preset time period.

In a specific implementation of S402, each temporary server in the replication group executes a deletion operation after determining that all persistent servers in the replication group execute the update operation. And the implementation process of each temporary server is similar, the implementation of S402 is now described by taking the first temporary server as an example, and the implementation of other temporary servers is not described in detail.

Specifically, for the first temporary server, the implementation of S402 may include, but is not limited to, any one of the following modes 1 to 3.

In the mode 1, after the first temporary server determines that the data meeting the second condition in the stored data of the first temporary server is stored in all the persistent servers in the replication group, the first temporary server executes deletion operation to delete the data meeting the second condition.

In the mode 2, after the first temporary server determines that the data meeting the second condition in the stored data of the first temporary server is stored in at least one persistent server in the replication group, the first temporary server executes deletion operation to delete the data meeting the second condition.

And in the mode 3, the first temporary server determines the data meeting the second condition in the stored data of the first temporary server, and after the number of the persistent servers stored in the replication group is greater than a preset threshold, the first temporary server executes deletion operation to delete the data meeting the second condition.

The preset threshold may be configured according to actual requirements, which is not specifically limited in the embodiments of the present application.

Illustratively, the first temporary server obtains the storage data of each persistent server in the replication group, determines whether each persistent server stores data meeting the second condition in the first temporary server, and if all persistent servers in the replication group store data meeting the second condition in the first temporary server, the first temporary server determines that the data meeting the second condition in the storage data of the first temporary server has been stored in the persistent server in the replication group, and then the first temporary server performs a deletion operation to delete the data meeting the second condition.

Further, as shown in fig. 5, if the persistent server in the copy group fails, the method for storing data provided by the present application may further include S403 and S404.

And S403, if the persistent server in the replication group fails, adding a new persistent server to the replication group.

All the servers in the replication group periodically send heartbeat test information to other servers to judge whether the servers in the replication group normally communicate. The processing manner when each persistent server in the replication group fails is similar, and now the implementation process of S403 is described by taking a failure of a second persistent server as an example, where the second persistent server is any one of the persistent servers in the replication.

For a second persistent server failure, S403 may be implemented as: and when one or more servers in the copy group do not normally receive the heartbeat test information sent by the second persistent server within a preset time period, determining that the second persistent server fails, and adding a new persistent server to the copy group according to a second rule.

The second rule may be configured according to actual requirements, which is not specifically limited in the embodiment of the present application.

In one possible implementation, the new persistent server is a persistent server with the largest remaining storage space among persistent servers not configured to the replica group.

In another possible implementation manner, the new persistent server is a persistent server that is not configured to the replication group and has a difference in the remaining storage space from other persistent servers in the replication group in the second interval.

Specifically, the implementation of S403 may include, but is not limited to, any one of scheme 1 to scheme 3 below.

In scheme 1, the replication group uses a master replication protocol, and the second persistent server is a master node, for example, S403 may be implemented as: if one or more servers in the replication group do not normally receive heartbeat test information sent by the master node within a preset time period, determining that the master node fails, selecting one node from the slave nodes included in the replication group as a new master node by other slave nodes in the replication group according to a voting mechanism, adding a new persistent server to the replication group by the new master node according to a second rule to serve as a new slave node, deleting the identifier of the failed node from the server identifiers included in the slave nodes in the configuration information of the replication group by the new master node, and adding the identifier of the new slave node; and updating the identification of the master node with the original fault in the configuration information of the copy group into the identification of the new master node.

In case of the scheme 2, the replication group adopts a master replication protocol, and the second persistent server is a slave node, for example, S403 may be implemented as: if one or more servers in the replica group do not normally receive heartbeat test information sent by the second persistent server within a preset time period, determining that the second persistent server fails, then adding a new persistent server to the replica group by the master node according to a second rule to serve as a new slave node, and updating the identifier of the failed node to the identifier of the new slave node in the server identifiers included in the slave nodes in the configuration information of the replica group.

In case of scenario 3, the replication group employs a master-less replication protocol, and for the second persistent server failure, for example, S403 may be implemented as: if one or more servers in the replica group do not normally receive heartbeat test information sent by the second persistent server within a preset time period, determining that the second persistent server fails, then adding a new persistent server to the replica group by the third persistent server according to a second rule, and updating the identifier of the failed persistent server in the configuration information of the replica group to the identifier of the new persistent server.

The third persistent server may be any one of the persistent servers in the replication group that has not failed, or the third server may be a persistent server selected by all the servers in the replication group that have not failed according to the third rule.

The user may configure the third rule according to actual requirements, which is not specifically limited in the embodiment of the present application.

Illustratively, the third rule may be: the persistent server that detected the failed persistent server first.

S404, the new persistent server writes the union of the data stored in all the servers in the copy group.

In one possible implementation manner, the replication group uses a masterless replication protocol, and S404 may be implemented as: and the new persistent server acquires the union set of the data stored in all the servers in the copy group and writes the union set. Or, the new persistent server first obtains the data stored by one persistent server in the copy group and then writes the data, which is not stored by the persistent server of the information, in the data stored by other servers in the copy group.

In another possible implementation manner, the replication group employs a primary replication protocol, and S404 may be implemented as: the new persistent server writes the data stored by the master nodes in the replica group or writes the data stored by the new master node.

Further, as shown in fig. 5, if the temporary server in the replication group fails, the method for storing data provided by the present application may further include S403A.

S403A, if the temporary server in the replication group fails, adding a new temporary server to the replication group.

Here, the processing manner of all the temporary servers in the replication group is similar, and now the implementation process of S403 is described by taking a second temporary server failure as an example, where the second temporary server is any one temporary server in the replication.

For a second persistent server failure, S403 may be implemented as: and if one or more servers in the copy group do not normally receive the heartbeat test information sent by the second temporary server within a preset time period, determining that the second temporary server fails, and adding a new persistent server to the copy group according to a fourth rule.

The fourth rule may be configured according to actual requirements, which is not specifically limited in the embodiment of the present application.

In one possible implementation, the new temporary server is any one of the temporary servers that is not configured to the replica group.

In another possible implementation manner, the new temporary server is a temporary server that is not configured to the replication group and has a difference in the remaining storage space from other temporary servers in the replication group in the third interval.

Specifically, the implementation of S403A may include, but is not limited to, scheme a or scheme b below.

In the scheme a, the replication group adopts a primary replication protocol, and for the second temporary server, S403A may be implemented as: and if one or more servers in the replication group do not normally receive the heartbeat test information sent by the second temporary server within a preset time period, determining that the second temporary server fails, and adding a new temporary server to the replication group by the main node according to a third rule.

Example a, if one or more servers in the replica group do not normally receive heartbeat test information sent by the second temporary server within a preset time period, it is determined that the second temporary server fails, then the master node adds a new temporary server to the replica group according to a third rule, and uses the new temporary server as a new slave node, and updates an identifier of the failed node to an identifier of the new slave node in server identifiers included in the slave nodes in the configuration information of the replica group.

In the scheme b, the replication group adopts the masterless replication protocol, and for the second temporary server, S403A may be implemented as: and if one or more servers in the replication group do not normally receive the heartbeat test information sent by the second temporary server within a preset time period, determining that the second temporary server fails, and adding a new temporary server to the replication group by the third persistent server according to a third rule.

Example B, if one or more servers in the replication group do not normally receive the heartbeat test information sent by the second temporary server, it is determined that the second temporary server fails, then the third persistent server adds a new temporary server to the replication group according to the second rule, and updates the identifier of the failed temporary server in the configuration information of the replication group to the identifier of the new temporary server.

The method for storing data provided by the present application is described below by taking a storage process of a distributed data service system as an example.

As shown in fig. 6, a distributed data services system client, a plurality of persistent servers, and a plurality of temporary servers. Wherein persistent server A1, persistent server B1, persistent server C1, temporary server a1, and temporary server B1 are taken as replication group 1; persistent server A2, persistent server B2, persistent server C2, temporary server a2, and temporary server B2 are considered copy group 2.

Wherein, the replication group 1 adopts a main replication protocol, and the configuration information of the replication group 1 is: the persistent server includes a1, B1, and C1; the temporary server includes a1 and b 1; the master node includes a 1; the slave nodes include B1, C1, a1, and B1.

The user writes the data M to the copy group 1 through the client, specifically, successfully writes the data M to A1, B1, a1 and B1; after 3 minutes, the user writes data N again to copy group 1 via the client, specifically, data N was successfully written to A1, C1, and b 1. Then, B1 acquires data M and data N in a1, and writes data N; c1 acquires the data M and the data N in A1 and writes the data M; a1 detects that the data M is stored in A1, B1 and C1, and then the data M is deleted; b1 detects that data M and data N are stored in A1, B1 and C1, and then deletes the data M and the data N.

At a certain moment, if B1 detects that A1 has a fault, B1, C1, a1 and B1 select B1 as a new master node according to a voting mechanism, and B1 adds a new persistent server D1 as a slave node to the replication group 1. D1 acquires the stored data in B1 and writes the data M and the data N. B1 updates the configuration information of copy group 1 as: the persistent server includes B1, C1, and D1; the temporary server includes a1 and b 1; the master node includes B1; the slave nodes include C1, D1, a1, and b 1.

At another time, if a1 detects a B1 failure, a1 adds a new persistent server D1 to copy group 1 as a slave node. D1 acquires the storage data in A1 and writes the data M and the data N. A1 updates the configuration information of copy group 1 as: the persistent server includes a1, D1, and C1; the temporary server includes a1 and b 1; the master node includes a 1; the slave nodes include D1, C1, a1, and b 1.

At another time, if a1 detects a1 failure, a1 adds a new temporary server c1 to copy group 1 as a slave node. A1 updates the configuration information of copy group 1 as: the persistent server includes a1, B1, and C1; the temporary server includes c1 and b 1; the master node includes a 1; the slave nodes include B1, C1, C1, and B1.

Wherein, the replication group 2 adopts a master-less replication protocol, and the configuration information of the replication group 2 is: the persistent server includes a2, B2, and C2; the temporary server includes a2 and b 2.

The user writes data J to the replication group 2 through the client, specifically, successfully writes the data J to B2, a2 and B2; after 3 minutes, the user writes data K to copy group 2 again through the client, specifically, data K was successfully written to a2, C2, and b 2. Then, a2 obtains data J and data K in B2, C2, a2 and B2, writes the data J; b2 acquires data J and data K in A2, C2, a2 and B2 and writes the data K; c2 acquires data J and data K in A2, B2, a2 and B2 and writes the data J; a2 detects that data J is stored in A2, B2 and C2, and then the data J is deleted; b2 detects that data J and data K are stored in A2, B2 and C2, and then deletes the data J and data K.

At some point, if B2 detects an A2 failure, B2 adds a new persistent server D2 to replica group 2. D2 obtains the stored data in B2, C2, a2 and B2, and writes data J and data K. B2 updates the configuration information of copy group 2 as: the persistent server includes D2, B2, and C2; the temporary server includes a2 and b 2.

At another time, if B2 detects a failure of a2, B2 adds a new temporary server c2 to copy group 2. B2 updates the configuration information of copy group 2 as: the persistent server includes a2, B2, and C2; the temporary server includes c2 and b 2.

The above description has introduced the solution provided by the embodiments of the present invention mainly from the point of view of the working principle of the servers in a replica group of a system for storing data. It is understood that each device for storing data, such as a persistent server, a temporary server, etc., contains corresponding hardware structures and/or software modules for performing each function in order to realize the above-mentioned functions. Those of skill in the art will readily appreciate that the present invention can be implemented in hardware or a combination of hardware and computer software, with the exemplary elements and algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the embodiment of the present invention, functional modules may be divided according to the above method examples, for example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, the division of the modules in the embodiment of the present invention is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

In the case of dividing each functional module according to each function, fig. 7 illustrates an apparatus 70 for storing data provided by the embodiment of the present application, which is used for implementing the function of the persistent server in the above method. The means for storing data 70 may be a persistent server or the means for storing data 70 may be deployed to a persistent server. As shown in fig. 7, the means 70 for storing data may include: a memory unit 701, a write unit 702, and an update unit 703. The storage unit 701 is used for storing data written into the persistent server; the writing unit 702 is configured to write the data written in the persistent server into the storage unit of the persistent server, and the updating unit 703 is configured to execute S401 in fig. 4 or fig. 5. All relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.

In the case of an integrated unit, fig. 8 shows a server 80 provided in the embodiment of the present application, for implementing the functions of the persistent server in the above method. The server 80 may be a persistent server, and the server 80 includes at least one processing module 801 for implementing the functions of the persistent server in the method provided by the embodiment of the present application. For example, the processing module 801 may be configured to execute the process S401 in fig. 4, or the processes S401, S403, and S404 in fig. 5, which refer to the detailed description in the method example and are not described herein again.

Server 80 may also include at least one memory module 802 for storing program instructions and/or data. The memory module 802 is coupled with the processing module 801. The coupling in the embodiments of the present application is an indirect coupling or a communication connection between devices, units or modules, and may be an electrical, mechanical or other form for information interaction between the devices, units or modules. The processing module 801 may cooperate with the memory module 802. The processing module 801 may execute program instructions stored in the memory module 802. At least one of the at least one memory module may be included in the processing module.

The server 80 may also include a communication module 803 for communicating with other devices over the transmission medium, the communication module 803 for communicating with other devices. For example, the processing module 801 may execute the procedure S401 in fig. 4 or fig. 5 by using the communication module 803.

When the processing module 801 is a processor, the storage module 802 is a memory, and the communication module 803 is a transceiver, the server 80 in the embodiment of fig. 8 may be the server 30 shown in fig. 3.

As mentioned above, the apparatus 70 or the server 80 for storing data provided by the embodiments of the present application can be used to implement the function of the persistent server in the method implemented by the embodiments of the present application, and for convenience of description, only the part related to the embodiments of the present application is shown, and details of the specific technology are not disclosed, please refer to the embodiments of the present application.

In the case of dividing each functional module corresponding to each function, as shown in fig. 9, a device 90 for storing data provided in the embodiment of the present application is used to implement the function of the temporary server in the above method. The means for storing data 90 may be a temporary server or the means for storing data 90 may be deployed at a temporary server. As shown in fig. 9, the apparatus 90 for storing data may include: a storage unit 901, a writing unit 902 and a processing unit 903. The storage unit 901 is used for storing data written in the temporary server; the writing unit 902 is configured to write the data written to the temporary server into the storage unit of the temporary server; the processing unit 903 is configured to execute S402 in fig. 4 or fig. 5. All relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.

In the case of using an integrated unit, as shown in fig. 10, a server 100 provided in the embodiment of the present application is used to implement the function of the temporary server in the above method. The server 100 may be a temporary server. The server 100 includes at least one processing module 1001 for implementing the functions of the temporary server in the method provided by the embodiment of the present application. For example, the processing module 1001 may be configured to execute S402 in fig. 4 or fig. 5, which specifically refers to the detailed description in the method example and is not described herein again.

The server 100 may also include at least one storage module 1002 for storing program instructions and/or data. The storage module 1002 is coupled with the processing module 1001. The coupling in the embodiments of the present application is an indirect coupling or a communication connection between devices, units or modules, and may be an electrical, mechanical or other form for information interaction between the devices, units or modules. The processing module 1001 may cooperate with the storage module 1002. The processing module 1001 may execute program instructions stored in the storage module 1002. At least one of the at least one memory module may be included in the processing module.

The server 100 may also include a communication module 1003 for communicating with other devices over a transmission medium for determining that the apparatus in the server 100 may communicate with other devices. The communication module 1003 is used for the device to communicate with other devices.

When the processing module 1001 is a processor, the storage module 1002 is a memory, and the communication module 1003 is a transceiver, the server 100 according to the embodiment of the present application, shown in fig. 10, may be the server 30 shown in fig. 3.

As described above, the apparatus 90 or the server 100 for storing data provided in the embodiments of the present application may be used to implement the function of the temporary server in the method implemented in the embodiments of the present application, and for convenience of description, only the part related to the embodiments of the present application is shown, and details of the specific technology are not disclosed, please refer to the embodiments of the present application.

Other embodiments of the present application provide a system for storing data, which may include a first means for storing data that may function as a persistent server and a second means for storing data that may function as a temporary server. For example, the first device for storing data is a persistent server described in the embodiments of the present application, and the second device for storing data is a temporary server described in the embodiments of the present application.

Other embodiments of the present application provide a chip system, which includes a processor and may further include a memory, and is configured to implement the functions of the persistent server in the embodiments shown in fig. 4 or fig. 5. The chip system may be formed by a chip, and may also include a chip and other discrete devices.

Other embodiments of the present application provide a chip system, which includes a processor and may further include a memory, and is configured to implement the functions of the temporary server in the embodiments shown in fig. 4 or fig. 5. The chip system may be formed by a chip, and may also include a chip and other discrete devices.

Further embodiments of the present application also provide a computer readable storage medium, which may include a computer program, which when run on a computer, causes the computer to perform the steps of the persistent server or the transient server of the embodiments shown in fig. 4 or fig. 5 described above.

Further embodiments of the present application also provide a computer program product comprising a computer program that, when run on a computer, causes the computer to perform the steps of the persistent server or the temporary server of the embodiments shown in fig. 4 or fig. 5 described above.

Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partially contributed to by the prior art, or all or part of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only an embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of storing data, applied to a replication group comprising one or more persistent servers and one or more temporary servers; the persistent server is used for storing data for a long time, the temporary server is used for temporarily storing data, and the replication group is used for writing the same data into the servers contained in the replication group by adopting a replication protocol; the method comprises the following steps:

each persistent server respectively executes updating operation, and updates the stored data meeting a first condition in the stored data of the server into a union set of the data meeting the first condition and stored in all servers in the copy group;

after each temporary server respectively determines that the data meeting the second condition in the stored data of the temporary server is stored in the persistent server in the replication group, each temporary server respectively executes deletion operation to delete the data meeting the second condition.

2. The method of claim 1, wherein the first condition comprises:

data written for the first N times, wherein N is greater than or equal to 1;

alternatively, the first and second electrodes may be,

and data written in a first preset time before the current time.

3. The method of claim 1 or 2, wherein the second condition comprises:

data written for the first M times, wherein M is greater than or equal to 1;

alternatively, the first and second electrodes may be,

and data written in a second preset time before the current time.

4. The method according to any one of claims 1 to 3, wherein the replica group employs a master replication protocol, and for a first persistent server, the first persistent service is any persistent server in the replica group, the first persistent server is a slave node, and the first persistent server performs an update operation to update stored data meeting a first condition in its stored data to a union of data meeting the first condition stored in all servers in the replica group, including:

if the master nodes are in normal communication, the first persistent server executes updating operation, and updates the storage data meeting a first condition in the storage data of the first persistent server into the data meeting the first condition and stored in the master nodes in the copy group;

alternatively, the first and second electrodes may be,

if the master node fails, the first persistent server executes updating operation, and writes data which are not stored in the first persistent server in the data which are stored in all the servers in the copy group and meet the first condition;

alternatively, the first and second electrodes may be,

if the master node fails and the newly selected master node is a first persistent server, the first persistent server executes updating operation and writes data which are not stored in the first persistent server in the data which are stored in all the servers in the copy group and meet the first condition; the first persistent server sends an update completion instruction to other persistent servers in the copy group; the update completion instruction is used for indicating the master node to complete the update operation;

alternatively, the first and second electrodes may be,

if the main node is in failure, and the newly selected main node is a persistent server other than the first persistent server, the first persistent server updates the stored data meeting the first condition in the stored data of the first persistent server into the data meeting the first condition stored in the newly selected main node after the newly selected main node executes the updating operation.

5. The method of any of claims 1-4, wherein the number of servers in the replica group is determined by system availability and the number of persistent servers in the replica group is determined by system persistence.

6. The method according to any one of claims 1 to 5,

after the write operation, the persistent server executes the update operation;

alternatively, the first and second electrodes may be,

the persistent server periodically performs the update operation.

7. The method according to any one of claims 1-3, wherein if a persistent server in the replica group fails, the method further comprises:

adding a new persistent server to the replication group;

and the new persistent server writes the union of the data stored in all the servers in the copy group.

8. An apparatus for storing data, characterized by persistent servers deployed in a replication group, the replication group being configured to write identical data to servers it contains using a replication protocol, the persistent servers being configured to store data for long periods of time; the device comprises:

the storage unit is used for storing the data written into the persistent server;

the writing unit is used for writing the data written into the persistent server into a storage unit of the persistent server;

and the updating unit is used for executing updating operation, and updating the storage data meeting the first condition in the storage data in the storage unit into a union set of the data meeting the first condition and stored in all the servers in the copy group.

9. The apparatus of claim 8, wherein the first condition comprises:

data written for the first N times, wherein N is greater than or equal to 1;

alternatively, the first and second electrodes may be,

and data written in a first preset time before the current time.

10. An apparatus for storing data, wherein the apparatus is characterized in that temporary servers are deployed in a replication group, the replication group further comprises one or more permanent servers, the permanent servers are used for storing data for a long time, the temporary servers are used for temporarily storing data, and the replication group is used for writing the same data to the servers contained in the replication group by adopting a replication protocol; the device comprises:

a storage unit for storing the data written in the temporary server;

a writing unit configured to write the data written in the temporary server into a storage unit of the temporary server;

and the processing unit is used for executing deletion operation after determining that the data meeting the second condition in the storage data of the storage unit is stored in the persistent server in the copy group, and deleting the data meeting the second condition.

11. The apparatus of claim 10, wherein the second condition comprises:

data written for the first M times, wherein M is greater than or equal to 1;

alternatively, the first and second electrodes may be,

and data written in a second preset time before the current time.

12. A system for storing data, the system comprising a replication group, the replication group comprising one or more persistent servers and one or more temporary servers; the persistent server is used for storing data for a long time, the temporary server is used for temporarily storing data, and the replication group is used for writing the same data into the servers contained in the replication group by adopting a replication protocol; wherein:

the persistent server is to: executing an updating operation, and updating the storage data meeting a first condition in the storage data of the server into a union set of the data meeting the first condition and stored in all servers in the copy group;

the temporary server is configured to: after determining that the data meeting the second condition in the stored data of the temporary servers are stored in the persistent servers in the replication group, each temporary server executes deletion operation respectively to delete the data meeting the second condition.

13. The system of claim 12, wherein the first condition comprises:

data written for the first N times, wherein N is greater than or equal to 1;

alternatively, the first and second electrodes may be,

and data written in a first preset time before the current time.

14. The system of claim 12 or 13, wherein the second condition comprises:

data written for the first M times, wherein M is greater than or equal to 1;

alternatively, the first and second electrodes may be,

and data written in a second preset time before the current time.

15. The system according to any one of claims 12 to 14, wherein the replication group employs a master replication protocol, and for a first persistent server, the first persistent service is any persistent server in the replication group, the first persistent server is a slave node, and when the first persistent server performs an update operation, the method specifically includes:

alternatively, the first and second electrodes may be,

16. The system of any of claims 12-15, wherein the number of servers in the replication group is determined by system availability and the number of persistent servers in the replication group is determined by system persistence.

17. The system according to any one of claims 12-16,

the persistent server is specifically configured to: after the write operation, the persistent server executes the update operation;

alternatively, the first and second electrodes may be,

the persistent server is specifically configured to: the persistent server periodically performs the update operation.

18. The system according to any of claims 12-14, wherein if a persistent server in the replica group fails, the persistent server is further configured to:

adding a new persistent server to the replication group;

19. A server, characterized in that the server comprises: a processor and a memory;

the memory is connected with the processor; the memory is used for storing computer instructions which, when executed by the processor, cause the server to perform the functions of a persistent server or a temporary server in the method of storing data according to any one of claims 1 to 7.

20. A computer-readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the functions of a persistent server or a transient server in the method of storing data of any of claims 1 to 7.

21. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the functions of a persistent server or a temporary server in a method of storing data according to any one of claims 1 to 7.