CN112148798A - Data processing method and device applied to distributed system - Google Patents

Abstract

The embodiment of the application provides a data processing method and a data processing device applied to a distributed system, wherein the distributed system comprises a main node and a plurality of slave nodes, and the data processing method comprises the following steps: acquiring a state identifier of each slave node from the master node, wherein the state identifier is used for indicating whether the slave node is in an available state or not, and the slave node in the available state stores a copy of data in the master node; determining whether a target slave node in an available state exists in the distributed system according to the state identification of each slave node; if the target slave node exists, sending the data reading request to the target slave node, and receiving response data which is returned by the target slave node and aims at the data reading request; and if the target slave node does not exist, sending the data reading request to the master node, and receiving response data which is returned by the master node and aims at the data reading request. According to the technical scheme of the embodiment of the application, the data reading processing is realized, and the requirements of high availability, high consistency and high performance of data processing can be met.

Description

Data processing method and device applied to distributed system

Technical Field

The present application relates to the field of computer and communication technologies, and in particular, to a data processing method and apparatus applied to a distributed system.

Background

A non-relational database (NoSQL) provides a user with very rich parameters and modes to control the read-write strategy of the database, and the user can configure a desired database read-write model through the combination of the parameters.

However, the read-write strategy of the existing database makes the user have to make trade-offs and tradeoffs among high availability, high performance and strong consistency, for example, to ensure the strong consistency of data read-write, the read master node is selected, and at this time, the performance advantages of multiple nodes are sacrificed; to take advantage of the multi-node, selecting a read slave node can only sacrifice consistency or availability. On the whole, the read-write strategy of the existing database is difficult to simultaneously ensure the high availability, the high performance and the strong consistency, thereby causing that the requirements of part of services in the actual scene cannot be met.

Disclosure of Invention

The embodiment of the application provides a data processing method and device applied to a distributed system, and further can better meet the requirements of high availability, strong consistency and high performance of part of services in an actual environment to at least a certain extent.

Other features and advantages of the present application will be apparent from the following detailed description, or may be learned by practice of the application.

According to an aspect of the embodiments of the present application, there is provided a data processing method applied to a distributed system, where the distributed system includes a master node and a plurality of slave nodes, the method including: acquiring a state identifier of each slave node from the master node, wherein the state identifier is used for indicating whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node; determining whether a target slave node in an available state exists in the distributed system according to the state identification of each slave node; if the target slave node exists, sending a data reading request to the target slave node, and receiving response data which is returned by the target slave node and aims at the data reading request; and if the target slave node does not exist, sending the data reading request to the master node, and receiving response data which is returned by the master node and aims at the data reading request.

According to an aspect of the embodiments of the present application, there is provided a data processing method applied to a distributed system, where the distributed system includes a master node and a plurality of slave nodes, the method including: sending a state identifier of each slave node to terminal equipment, wherein the state identifier is used for indicating whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node; receiving a data reading request sent by the terminal device, wherein the data reading request is sent by the terminal device when the terminal device determines that no target slave node in an available state exists in the distributed system according to the state identification of each slave node; and responding to the data reading request, and returning response data aiming at the data reading request to the terminal equipment.

According to an aspect of the embodiments of the present application, there is provided a data processing apparatus applied to a distributed system including a master node and a plurality of slave nodes, the apparatus including: the obtaining unit is configured to obtain a state identifier of each slave node from the master node, where the state identifier is used to indicate whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node; a determining unit, configured to determine whether there is a target slave node in an available state in the distributed system according to the state identifier of each slave node; the first sending unit is configured to send a data reading request to the target slave node if the target slave node exists, and receive response data which is returned by the target slave node and aims at the data reading request; and the second sending unit is configured to send the data reading request to the master node and receive response data returned by the master node for the data reading request if the target slave node does not exist.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: a write request sending unit configured to send a data write request to the master node so that the master node writes data specified by the data write request; and the notification receiving unit is configured to receive notification information sent by the master node, wherein the notification information is generated after the master node receives a write copy completion message sent by a slave node in an available state.

In some embodiments of the present application, based on the foregoing scheme, the obtaining unit is configured to: acquiring a node state view from the master node, wherein the node state view comprises state identifiers of all slave nodes; and determining the state identification of each slave node according to the node state view.

According to an aspect of the embodiments of the present application, there is provided a data processing apparatus applied to a distributed system including a master node and a plurality of slave nodes, the apparatus including: a third sending unit, configured to send a status identifier of each slave node to the terminal device, where the status identifier is used to indicate whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node; a receiving unit, configured to receive a data read request sent by the terminal device, where the data read request is sent by the terminal device when it is determined that there is no target slave node in an available state in the distributed system according to the state identifier of each slave node; and the response unit is configured to respond to the data reading request and return response data aiming at the data reading request to the terminal equipment.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: a write request receiving unit configured to receive a data write request sent by the terminal device; and the writing unit is configured to respond to the data writing request, write data specified to be written by the data writing request, and generate an operation log, wherein the operation log is used for enabling each slave node to determine the data needing to be written.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: and the updating unit is configured to update the state identifier of each slave node according to the delay time returned after the operation log is pulled by each slave node, so as to obtain a new state identifier of each slave node.

In some embodiments of the present application, based on the foregoing solution, the apparatus further includes: a notification generating unit configured to generate notification information if a write-copy completion message sent by a slave node in an available state is received, where the slave node in the available state is determined according to the new state identifier of each slave node; a notification transmission unit configured to transmit the notification information to the terminal device.

In some embodiments of the present application, based on the foregoing solution, the updating unit is configured to: if the delay time is not received, determining that the slave node which does not send the delay time is in an unavailable state; and if the delay time is received and is greater than the preset delay time, determining that the slave node sending the delay time is in an unavailable state.

According to an aspect of embodiments of the present application, there is provided a computer-readable medium on which a computer program is stored, the computer program, when executed by a processor, implementing the data processing method applied to a distributed system as described in the above embodiments.

According to an aspect of an embodiment of the present application, there is provided an electronic device including: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the data processing method applied to the distributed system as described in the above embodiments.

According to an aspect of embodiments herein, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data processing method applied to the distributed system provided in the various optional embodiments.

In the technical solution provided by some embodiments of the present application, a target slave node in an available state in a distributed system is determined by a state identifier of each slave node, then a data read request is sent to the target slave node, and after the data read request is submitted to the target slave node, response data for the data read request returned from the target slave node is received. And if the target slave node in the available state does not exist in the distributed system, sending the data reading request to the target slave node, and then receiving response data returned by the master node for the data reading request. The technical scheme of the embodiment of the application has at least the following beneficial effects:

(1) according to the technical scheme of the embodiment of the application, the state identification is used for indicating whether the slave node is in the available state, the slave node in the available state stores the copy of the data in the master node, and the data writing request only needs to be written and confirmed in the slave node in the available state, in other words, the data writing request is not blocked due to the fact that the slave node in the unavailable state exists in the distributed system, and therefore high availability is guaranteed;

(2) in addition, when a target slave node in an available state exists, the data reading request is sent to the target slave node, and the slave node in the available state stores the copy of the data in the master node, so that the data reading request is sent to the target slave node, the data written before can be guaranteed to be read, and the strong consistency of reading after writing is guaranteed;

(3) moreover, even if the nodes in the unavailable state exist in the distributed system, the data can be read from the slave nodes or the master nodes in the available state, so that the performance advantages of the rest nodes are exerted, and high performance is ensured.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:

FIG. 1 is a diagram illustrating an exemplary system architecture to which aspects of embodiments of the present application may be applied;

FIG. 2 shows a flow diagram of a data processing method applied to a distributed system according to one embodiment of the present application;

FIG. 3 shows a flow diagram of a data processing method applied to a distributed system according to one embodiment of the present application;

FIG. 4 shows a flow diagram of a data processing method applied to a distributed system according to one embodiment of the present application;

FIG. 5 shows a flow diagram of a data processing method applied to a distributed system according to one embodiment of the present application;

FIG. 6 shows a block diagram of a data processing apparatus applied to a distributed system according to an embodiment of the present application;

FIG. 7 shows a block diagram of a data processing apparatus applied to a distributed system according to an embodiment of the present application;

FIG. 8 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the subject matter of the present application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the application.

It is to be noted that the terms used in the specification and claims of the present application and the above-described drawings are only for describing the embodiments and are not intended to limit the scope of the present application. It will be understood that the terms "comprises," "comprising," "includes," "including," "has," "having," and the like, when used herein, specify the presence of stated features, integers, steps, operations, elements, components, and/or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It will be further understood that, although the terms first, second, third, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element without departing from the scope of the present invention. Similarly, a second element may be termed a first element. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

It should be understood that in the present application, "at least one" means one or more, "a plurality" means two or more.

Before further detailed description of the embodiments of the present application, terms and expressions referred to in the embodiments of the present application will be described, and the terms and expressions referred to in the embodiments of the present application will be explained below.

Cloud computing (cloud computing): the method refers to a delivery and use mode of an IT infrastructure, and refers to acquiring required resources in an on-demand and easily-extensible mode through a network; the generalized cloud computing refers to a delivery and use mode of a service, and refers to obtaining a required service in an on-demand and easily-extensible manner through a network. Such services may be IT and software, internet related, or other services. Cloud Computing is a product of development and fusion of traditional computers and Network Technologies, such as Grid Computing (Grid Computing), distributed Computing (distributed Computing), Parallel Computing (Parallel Computing), Utility Computing (Utility Computing), Network Storage (Network Storage Technologies), Virtualization (Virtualization), Load balancing (Load Balance), and the like.

Cloud storage (cloud storage): the distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system which integrates a large number of storage devices (storage devices are also referred to as storage nodes) of different types in a network through application software or application interfaces to cooperatively work through functions of cluster application, grid technology, distributed storage file systems and the like, and provides data storage and service access functions to the outside.

Non-relational database: the data storage mode is used for storing massive non-relational data in the Web2.0 era and meets the requirements on concurrent access performance, storage capacity, expansibility and reliability.

MongoDB: the method is an open-source, mode-free and document storage-oriented high-performance distributed NoSQL database. The copy Set (Replica Set) of the MongoDB includes a plurality of instances, and the instances may be in a master (Primary) state, a slave (Secondary) state, and the like. Data can be written in through a main node, and then a replication set protocol based on Raft-like is used for actively and synchronously operating logs (oplog) to other nodes, so that the consistency of all node data in a replication set is ensured, and the nodes form replicas mutually; the read operation can be selectively distributed on all the nodes, and the performance of database query is effectively improved.

MongoDB as a very flexible and widely applied distributed NoSQL database product provides very rich parameters and modes for users to control the read-write strategies of the database, namely read attention, write attention and read preference. The user can configure a desired database read-write model through a combination of these parameters.

MongoDB read-write model control is related to three parameters in total, which are respectively: read focus (Read focus), Write focus (Write focus), Read Preference (Read Preference). The user can adjust the consistency and availability guarantee levels and establish corresponding read-write strategies by effectively controlling the three parameters, such as providing stronger consistency guarantees for the database or providing weaker consistency guarantees but higher availability guarantees for the database. These three parameters are briefly described below:

first, read attention

The level of read attention may be as follows:

(1) local grade: any data of the current node can be read.

(2) Major rating level: only "data successfully written to most nodes" can be read.

(3) Linear level: linear level, all successful write operations completed before the operation and acknowledged by most nodes can be read.

(4) Available level: the availability level provides greater partition tolerance in a sharded cluster scenario, and is consistent with the Local level in a replica cluster scenario.

(5) Snapshot level: snapshot level for multi-document transaction scenarios.

The original intention of the reading attention is to solve the relatively common problem of 'dirty reading' in the database, for example, a user reads a certain piece of data from the main node of the MongoDB, but the piece of data is not synchronized to most nodes, then the main node fails, and after the data is restored, the main node rolls back the data which is not synchronized to most nodes, so that the user reads the 'dirty data'. When the designated reading attention is at the major level, the read data can be ensured to be synchronized to most nodes, and the rolling back is certainly avoided, so that the problem of 'dirty reading' is avoided. It is worth noting that the priority level does not guarantee that the read data is up to date.

Second, write attention

Specific options for writing attention include:

1. w: how many nodes the data is written into is returned to the client

(1) { w:0 }: the method is applicable to the scenes with high requirement on writing performance and inferior requirement on data security (data may be lost).

(2) { w:1 }: indicating that the write operation is acknowledged only on the current master node. w is designated as N and is confirmed by the number of members in the replica set. For example, when there are 3 nodes in the copy set, w:2 indicates that the write operation needs to be confirmed on two nodes to return; w:3 then indicates that the write operation requires acknowledgements from all nodes to return.

(3) { w: majauthority }: the method represents that the write operation is confirmed on most nodes of the copy set, and is suitable for a scene with high requirement on data security and second-order write performance.

(4) Custom copy assurance rules: it needs to be used together with "tag". For example, tag of a high performance node is designated as a Solid State Drive (SSD), and the node returns after confirming the write operation.

2. j: returning to the client after j ournal persistence during writing

(1) { j: true }: when the representation is written, the journal is persisted and then returned to the client.

(2) { j: false }: and the representation does not return to the client after j ournal persistence when being written.

3. wtimeout: indicating a write timeout time.

When the designated w is greater than 1 or majpriority, if there is a node failure, it may cause the write acknowledge condition to be unsatisfied all the time, thereby blocking the write request. In this case, this problem can be avoided by setting the write timeout time.

Third, read preference

The reading preference describes how the MongoDB client sends a reading request to the master node or the slave node in the replica set, and specific options of the reading preference include:

(1) primary: read-only master node, default mode.

(2) Primary Preferred: and preferentially reading the master node, and reading the slave node when the master node is unavailable.

(3) Second: read-only slave nodes.

(4) Secondary Preferred: the slave node is read preferentially, and the slave node cannot read the master node.

(5) Nerest: the node with the lowest network latency is read.

(6) Self-defining rules: and customizing the nodes to be read according to the attributes such as the positions or the performances of the nodes in the replica set. Nerest can be considered a generic, custom rule.

For various reasons (e.g., network congestion, low disk throughput, long time high load operation, etc.), a slave node in a replica set may be far behind the master node and may read too old data if the user chooses to read the slave. In later versions 3.4, therefore, montodb introduced a new control parameter for read preferences: maxsalensseconds, which the user may specify to avoid reading over-old data when setting the read preference to Secondary Preferred.

Fourth, causal consistency

When an operation is logically dependent on one of the previous operations, a causal relationship exists between the two operations. For example, an insert operation of a document and an update operation of the document have a causal relationship, and if the insert operation is not completed, the update operation cannot be performed. What is meant by causal consistency is that operations with causal relationships meet their strict sequence requirements. In the MongoDB, by specifying specific read attention and write attention, operations can be performed in an order that respects the causal relationship, and the user can also observe the result that conforms to the causal consistency constraint.

The use mode is simple and roughly as follows:

a. the client starts a new causal consistency client session and specifies that both read and write concerns are "maj authority";

b. a series of read-write operations are executed, and the MongoDB returns a global logic clock;

c. the client session will keep track of this one clock, ensuring causal consistency.

Causal consistency is achieved based on a global logic clock "ClusterTime" whose timing is roughly as follows for two causal operations:

1. the client sends a write-in operation to the master node, wherein the w:1 represents that the master node does not wait for the write-in operation to be copied to other slave nodes and then returns, but returns immediately after submitting to an own operation log;

2. the main node calculates a global logic clock ClusterTime and increases the ClusterTime incrementally;

3. the master node returns the result containing the global logic clock (present in the "OpTime" field) to the client;

4. the client conditionally updates the view about "lastOperationTime" according to the received return value;

5. the client sends a read operation to the slave node, and in order to ensure that the data written by the client can be read (namely to ensure causality), the client assigns "afterclusterime" to represent the data after the time of the read operation;

6. the slave node checks whether the expected read data is in its own operation log, if not, it will block until the condition is fulfilled;

7. the result is returned to the client;

to make MongoDB meet the requirements of causal consistency, the following conditions need to be met: 1) designating the client session as of a causal consistent category; 2) specifying that both read and write concerns are "major"; 3) all operations must be completed within the same client session, and if not, an "afterClusterTime" value (which may be obtained from other causally consistent sessions) needs to be manually set.

In MongoDB, the default read and write concerns are configured as "local" and "{ w:1 }". Under such a read-write strategy, causal consistency of the database cannot be guaranteed, and if the read preference is read-first, old data, dirty data and the like may be read.

The MongoDB selection completely throws users away from the trade-off problem between availability, consistency, and performance. The user can set the own database into a desired read-write model by selecting different read concerns, write concerns and read preferences. The common read-write model of the business can be summarized as shown in the following table 1:

TABLE 1

As can be seen from table 1, these conventional read-write models either fail to take advantage of the slave nodes, or have the possibility of reading old data, or have some availability problems (a down node may cause a write operation to fail). The existing MongoDB read-write model, including a newly introduced causal consistency model, still has the defect that the requirements of high availability, strong consistency and high performance of part of users in a complex scene cannot be met.

In view of the above, in an embodiment of the present application, a data processing method applied to a distributed system is provided, where the distributed system includes a master node and a plurality of slave nodes, a terminal device obtains a status identifier of each slave node from the master node, the status identifier is used to indicate whether the slave node is in an available state, the slave node in the available state stores a copy of data in the master node, and then the terminal device can determine whether a target slave node in the available state exists in the distributed system according to the state identifier of each slave node, and if the target slave node exists, sending the data reading request to the target slave node, and receiving response data which is returned by the target slave node and aims at the data reading request, if the target slave node does not exist, sending the data reading request to the main node and receiving response data which is returned by the main node and aims at the data reading request. The technical scheme of the embodiment of the application can better meet the requirements of high availability, strong consistency and high performance of part of services in the actual environment:

firstly, the technical scheme of the embodiment of the application indicates whether the slave node is in an available state by using the state identifier, and the slave node in the available state stores the copy of the data in the master node, so that the data writing request only needs to be written and confirmed in the slave node in the available state, in other words, the data writing request is not blocked by the slave node in the unavailable state in the distributed system, and high availability is ensured;

in addition, when a target slave node in an available state exists, the data reading request is sent to the target slave node, and the slave node in the available state stores the copy of the data in the master node, so that the data reading request is sent to the target slave node, the data written before can be guaranteed to be read, and the strong consistency of reading after writing is guaranteed;

moreover, even if the nodes in the unavailable state exist in the distributed system, the data can be read from the slave nodes or the master nodes in the available state, so that the performance advantages of the rest nodes are exerted, and high performance is ensured.

Fig. 1 shows a schematic diagram of an exemplary system architecture to which the technical solution of the embodiment of the present application may be applied, and the system architecture 100 may include a terminal device 101 and a distributed system 102.

The terminal device 101 may be an actual hardware device, such as a general-purpose computer device, or may be a virtual software application, such as a client running on a computer device, where the client may be a virtual machine, virtual machine software, or database software, etc.

In a specific implementation of the distributed system 102 in this application, the distributed system may include servers, which may be used to provide storage, computation and network resources for the distributed system, and the servers may be referred to as nodes, and the distributed system 102 may select one of the servers as a master node, and set one or more other servers to store the same data as the master node, that is, to serve as a copy of the master node, and the copy serving as a backup may be referred to as a slave node. For example, in fig. 1, distributed system 102 includes a master node 102a and three slave nodes 102b, 102c, and 102 d.

It should be noted that the three slave nodes 102b, 102c, and 102d shown in fig. 1 do not constitute a limitation on the slave nodes in the distributed system, and it is understood that the distributed system 102 may include a greater number of slave nodes.

It should be further noted that the servers included in the distributed system 102 may be independent physical servers, and may also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, web services, cloud communications, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms.

The terminal device 101 may initiate a service request including a write request, a read request, or a data synchronization request to a node in the distributed system 102. In an embodiment of the present application, the terminal device 101 may obtain, through a driver, a status identifier of each slave node in the distributed system 102 from the master node 102a, where the status identifier is used to indicate whether the slave node is in an available state, and the slave node in the available state stores a copy of master node data, after obtaining the status identifier of each slave node, the terminal device 101 may determine whether a target slave node in the available state exists in the distributed system 102 according to the status identifier of each slave node, and if the target slave node exists, the terminal device 101 may send a data read request to the target slave node, and conversely, if the target slave node does not exist, the terminal device 101 may send a data read request to the master node 102 a.

In an embodiment of the present application, the terminal device 101 may further send a data write request to the master node 102a, so that the master node 102a writes data specified by the data write request, writes the data in the master node 102a, and updates the operation log, so that the slave node may determine data to be written by pulling the operation log, and after determining the data to be written, the slave node may write a copy of the data locally.

In an embodiment of the present application, after the slave node in the available state locally writes a copy of data, it needs to send a write copy completion message to the master node 102a, and after receiving the write copy completion message returned by the slave node in the available state, the master node 102a may generate a response message for the data write request and send the response message for the data write request to the terminal device 101.

The implementation details of the technical solution of the embodiment of the present application are set forth in detail below:

fig. 2 shows a flowchart of a data processing method applied to a distributed system according to an embodiment of the present application, which may be performed by a terminal device, which may be the terminal device 101 shown in fig. 1. Referring to fig. 2, the method includes:

step S210, obtaining a state identifier of each slave node from the master node, wherein the state identifier is used for indicating whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node;

step S220, determining whether a target slave node in an available state exists in the distributed system according to the state identification of each slave node;

step S230, if the target slave node exists, sending a data read request to the target slave node, and receiving response data for the data read request returned by the target slave node;

step S240, if the target slave node does not exist, sending the data read request to the master node, and receiving response data for the data read request returned by the master node.

These steps are described in detail below.

In step S210, a status identifier of each slave node is obtained from the master node, where the status identifier is used to indicate whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node.

Since the conventional database read-write model has the possibility of reading old data or has certain usability problems, the read mode in which the write focus is { w: maj ority } plus the read is a compromise scheme, but the read mode still lacks finer-grained control over consistency and high performance.

Furthermore, the introduction of the causal consistency model effectively strengthens the alternative read-write model and the consistency assurance level, but still has a number of drawbacks, such as: 1) the read slave may block, resulting in unexpected read performance degradation; 2) the read/write attention needs to be appointed to be 'majpriority', and the write performance is not improved compared with { w: majpriority }; 3) the application program must ensure that only one thread performs read-write operation in the client session with consistent cause and effect at a time, and if the cause and effect consistency is to be ensured in a plurality of client sessions, a global logic clock needs to be transmitted in the sessions, which is complicated; 4) a driver is required to be used that supports causal consistent client sessions.

Therefore, the existing read-write model still cannot meet the requirements of high availability, strong consistency and high performance of part of customers in complex scenes.

In order to solve the technical problem, in this embodiment, first, information of a status identifier of each slave node may be maintained at a master node in a distributed system, where the status identifier of each slave node is used to identify whether the slave node is in an available state, and the status identifier may effectively distinguish the slave node in the available state from the slave node in an unavailable state.

In an embodiment of the present application, since the status identifier of each slave node is maintained on the master node, if the master node fails, a master-slave switch may be performed to switch one slave node to a new master node, in which case the status identifier of each slave node is maintained on the new master node.

It will be appreciated that a slave node in an available state may be used for data processing, while a slave node in an unavailable state may not be used for data processing. For example, a user's data read request may be selected to be sent to a node in an available state, and not to be sent to a node in an unavailable state; the data writing request of the user only needs to be confirmed at the slave node in the available state, but does not need to be confirmed at the slave node in the unavailable state, that is, the data writing request of the user is sent to the master node, and after the data is written by the master node, the master node can return notification information to the terminal equipment as long as the master node receives a copy writing completion message of the slave node in the available state.

Just because a data write request from a user only needs to be acknowledged at a slave node in an available state, but not at a slave node in an unavailable state, in this embodiment, the slave node in the available state is a slave node that stores a copy of data in the master node.

When a user has a requirement for reading data, a data reading request can be sent to the distributed system through the terminal device, and the data reading request is mainly used for requesting to read data from nodes in the distributed system. Before the terminal device sends a data reading request, the terminal device may obtain the state identifier of each slave node from the master node of the distributed system through the driver.

In one embodiment of the present application, the master node may maintain the status identifiers of the slave nodes in a node status view, where the view is actually a virtual table, and the view is represented in a manner similar to a real data table, and includes a series of named columns and rows, but the view does not exist in the database in the form of a stored data value set. Therefore, the maintenance cost for maintaining the state identification of each slave node in the node state view mode is lower.

In this embodiment, the terminal device may obtain a node state view, where the node state view includes the state identifier of each slave node, and then may determine the state identifier of each slave node according to the node state view, that is, determine whether each slave node is in an available state.

Step S220, determining whether there is a target slave node in an available state in the distributed system according to the state identifier of each slave node.

After the state identifier of each slave node is acquired from the master node, the terminal device can determine whether a target slave node in an available state exists in the distributed system according to the state identifier of each slave node.

Step S230, if the target slave node exists, sending a data read request to the target slave node, and receiving response data for the data read request returned by the target slave node.

In this embodiment, if the terminal device determines that the target slave node exists in the distributed system, the data read request may be sent to the target slave node. After submitting the data read request to the target slave node, response data for the data read request returned from the target slave node is received.

Step S240, if the target slave node does not exist, sending the data read request to the master node, and receiving response data for the data read request returned by the master node.

On the contrary, if the terminal device determines that there is no target slave node, that is, there is no slave node in an available state in the distributed system, in this case, the terminal device may send a data read request to the master node, and the master node returns response data for the data read request to the terminal device after receiving the data read request.

It should be noted that, in this embodiment, the master node is always in an available state, and therefore, when there is no target slave node in an available state, the master node may respond to the data read request, and then, the master node returns response data for the data read request to the terminal device.

Based on the technical solution of the above embodiment, the status identifier indicates whether the slave node is in an available status, and the slave node in the available status stores the copy of the data in the master node, which indicates that the data write request only needs to be written and confirmed in the slave node in the available status, in other words, indicates that the data write request is not blocked by the slave node in the unavailable status in the distributed system, thereby ensuring high availability; in addition, the data reading request is sent to the target slave node in an available state, so that the data written before can be read, and the strong consistency of reading after writing is ensured; moreover, even if the nodes in the unavailable state exist, the data can be read from the slave nodes or the master nodes in the available state, so that the performance advantages of the rest nodes are exerted, and high performance is ensured.

In an embodiment of the present application, in addition to the data read request, the user may also have a requirement for data writing, and in this embodiment, as shown in fig. 3, the data processing method further includes steps S310 to S320, which are described in detail as follows:

step S310, sending a data write request to the master node, so that the master node writes in data specified by the data write request.

When the data writing method is specifically implemented, a user can send a data writing request to the main node through the terminal device, wherein the data writing request is mainly used for writing data which are specified to be written into the main node of the distributed data system, and the data which are specified to be written can be video data, audio data, image data, text data and the like.

In some embodiments, the terminal device may directly carry data to be written in the data writing request, and the host node receives the data writing request and writes the data to be written carried in the data writing request. Certainly, in other embodiments, the terminal device may further store the data to be written in the third-party database, and store the data to be written in the storage address of the third-party database in the data writing request to the master node, after receiving the data writing request, the master node obtains the storage address of the data to be written in the third-party database according to the data writing request, then the master node obtains the data in the third-party database according to the storage address, and after obtaining the data, the master node may further send a prompt message to the terminal device to prompt the terminal device to successfully obtain the data.

Step S320, receiving notification information sent by the master node, where the notification information is generated after the master node receives a write-copy completion message sent by a slave node in an available state.

The master node, upon receiving the data write request, may write data locally at the master node in response to the data write request.

In one embodiment, after writing data, the master node may generate an operation log, which is a record of completed processing, that is, the master node may encapsulate the written data into the operation log. The slave node in the available state can perform data synchronization by pulling the operation log, and finally, the data consistency of the master node and the slave node in the available state in the distributed system is ensured.

The slave node in the available state may send a write-copy completion message to the master node after locally writing a copy of data, and the master node may generate a response message for the data write request after receiving the write-copy completion message returned by the slave node in the available state, and send the response message for the data write request to the terminal device.

In this embodiment, each data write request only needs to be confirmed on a node in an available state, which is more flexible than { w: all } and { w: majority } in write attention because the number of nodes that need to be confirmed may dynamically change according to whether the node is in the available state, and when there is only one node in the available state, that is, after the master node receives a write copy completion message sent by the node, notification information may be returned to the terminal device, which is equal to { w:1 }; when all the slave nodes in the distributed system are in an available state, that is, the master node needs to return notification information to the terminal device after receiving the write-copy completion messages sent by all the slave nodes, which is equal to { w: all }.

Fig. 4 shows a flowchart of a data processing method applied to a distributed system according to an embodiment of the present application, where the data processing method may be performed by a master node in the distributed system, which may be the distributed system 102 shown in fig. 1, and the master node may be the master node 102a shown in fig. 1. Referring to fig. 4, the data processing method specifically includes:

step S410, sending a state identifier of each slave node to the terminal equipment, wherein the state identifier is used for indicating whether the slave node is in an available state or not, and the slave node in the available state stores a copy of data in the master node;

step S420, receiving a data reading request sent by the terminal device, where the data reading request is sent by the terminal device when it is determined that there is no target slave node in an available state in the distributed system according to the state identifier of each slave node;

step S430, responding to the data read request, and returning response data for the data read request to the terminal device.

These steps are described in detail below:

in step S410, a status identifier of each slave node is sent to the terminal device, where the status identifier is used to indicate whether the slave node is in an available state, and the slave node in the available state stores a copy of the data in the master node.

In specific implementation, the master node may send the state identifier of each slave node to the terminal device under the effect of the terminal device driver, where the state identifier of each slave node is used to identify whether the slave node is in an available state, and the state identifier of each slave node may effectively distinguish the slave node in the available state from the slave node in the unavailable state. Wherein the slave node in the available state stores a copy of the data in the master node.

In step S420, a data read request sent by the terminal device is received, where the data read request is sent by the terminal device when it is determined that there is no target slave node in an available state in the distributed system according to the state identifier of each slave node.

When the terminal device learns that no target slave node in an available state exists in the distributed system according to the state identifier of each slave node, the terminal device can send a data reading request to the master node, and the master node receives the data reading request. The data reading request is mainly used for requesting to read data stored in the distributed system.

In step S430, responding to the data read request, and returning response data for the data read request to the terminal device.

And after receiving the data reading request of the terminal equipment, the main node responds to the data reading request and returns response data aiming at the data reading request to the terminal equipment.

Referring to fig. 5, in an embodiment of the present application, when a terminal device initiates a data write request, the terminal device may first request a master node, and in this embodiment, the method specifically includes steps S510 to S520, which is described in detail as follows:

step S510, receiving a data writing request sent by the terminal device.

The data write request, that is, the request for writing data, may be a write operation requesting to change existing data, or a write operation requesting to add new data.

When a user has a data writing requirement, a data writing request can be sent to the host node through the terminal device, and it can be understood that the data writing request sent to the host node by the terminal device may directly carry data to be written, or may only carry the data to be written and store the data in a storage address of a third-party database, so that the host node can obtain the data to be written according to the storage address.

Step S520, responding to the data write request, writing data specified by the data write request, and generating an operation log, where the operation log is used to enable each slave node to determine data that needs to be written.

After receiving the data write request, the master node may respond to the data write request, write data specified by the data write request in the local of the master node, and may generate an operation log, where the operation log includes the data written by the master node, so that each slave node in the distributed system may determine the data to be written by pulling the operation log.

In an embodiment of the present application, the master node may update the state identifier of each slave node according to the delay time returned after each slave node pulls the operation log, so as to obtain a new state identifier of each slave node.

In an embodiment of the present application, the updating, by the master node, the state identifier of each slave node according to the delay time returned after the slave node pulls the operation log may specifically include: if the master node does not receive the delay time, the slave node which does not send the delay time can be determined to be in an unavailable state, and if the slave node which is maintained on the master node before is in an available state, the state identification of the slave node can be updated to be in an unavailable state at the moment; if the master node receives the delay time and the received delay time is greater than the preset delay time, it indicates that the slave node lags behind the master node more, so the master node may also determine that the slave node sending the delay time is in an unavailable state, and if the slave node maintained on the master node before is in an available state, the state identifier of the slave node may be updated to be in an unavailable state at this time.

In an embodiment of the present application, the master node may update the state identifier of each slave node according to a delay time returned after the slave node pulls the operation log, and after the state identifier of each slave node is updated, the master node may know which slave nodes are in an available state at this time, and therefore, if a write-copy completion message sent by the slave node in the available state is received, the master node may generate notification information and return the notification information to the terminal device.

In this embodiment, each data write request only needs to be confirmed on a node in an available state, which is more flexible than { w: all } and { w: majority } in write attention because the number of nodes that need to be confirmed may dynamically change according to whether the node is in the available state, and when there is only one node in the available state, that is, after the master node receives a write copy completion message sent by the node, notification information may be returned to the terminal device, which is equal to { w:1 }; when all the slave nodes in the distributed system are in an available state, that is, the master node needs to return notification information to the terminal device after receiving the write-copy completion messages sent by all the slave nodes, which is equal to { w: all }.

The following describes an embodiment of an apparatus of the present application, which may be used to execute a data processing method applied to a distributed system in the foregoing embodiment of the present application. For details that are not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the data processing method applied to the distributed system described above in the present application.

Fig. 6 is a block diagram of a data processing apparatus applied to a distributed system according to an embodiment of the present application, and referring to fig. 6, a data processing apparatus 600 applied to a distributed system including a master node and a plurality of slave nodes according to an embodiment of the present application includes: an obtaining unit 602, a determining unit 604, a first sending unit 606 and a second sending unit 608.

The obtaining unit 602 is configured to obtain, from the master node, a status identifier of each slave node, where the status identifier is used to indicate whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node; a determining unit 604, configured to determine whether there is a target slave node in an available state in the distributed system according to the state identifier of each slave node; a first sending unit 606, configured to send a data read request to the target slave node if the target slave node exists, and receive response data returned by the target slave node for the data read request; a second sending unit 608, configured to send the data read request to the master node if the target slave node does not exist, and receive response data returned by the master node for the data read request.

In some embodiments of the present application, the apparatus further comprises: a write request sending unit configured to send a data write request to the master node so that the master node writes data specified by the data write request; and the notification receiving unit is configured to receive notification information sent by the master node, wherein the notification information is generated after the master node receives a write copy completion message sent by a slave node in an available state.

In some embodiments of the present application, the obtaining unit 602 is configured to: acquiring a node state view from the master node, wherein the node state view comprises state identifiers of all slave nodes; and determining the state identification of each slave node according to the node state view.

FIG. 7 shows a block diagram of a data processing apparatus applied to a distributed system according to one embodiment of the present application.

Referring to fig. 7, a data processing apparatus 700 for a distributed system including a master node and a plurality of slave nodes according to an embodiment of the present application includes: a third transmitting unit 702, a receiving unit 704 and a responding unit 706.

A third sending unit 702, configured to send a status identifier of each slave node to the terminal device, where the status identifier is used to indicate whether the slave node is in an available state, where the slave node in the available state stores a copy of data in the master node; a receiving unit 704, configured to receive a data read request sent by the terminal device, where the data read request is sent by the terminal device when it is determined that there is no target slave node in an available state in the distributed system according to the state identifier of each slave node; a response unit 706 configured to respond to the data read request and return response data for the data read request to the terminal device.

In some embodiments of the present application, the apparatus further comprises: a write request receiving unit configured to receive a data write request sent by the terminal device; and the writing unit is configured to respond to the data writing request, write data specified to be written by the data writing request, and generate an operation log, wherein the operation log is used for enabling each slave node to determine the data needing to be written.

In some embodiments of the present application, the apparatus further comprises: and the updating unit is configured to update the state identifier of each slave node according to the delay time returned after the operation log is pulled by each slave node, so as to obtain a new state identifier of each slave node.

In some embodiments of the present application, the apparatus further comprises: a notification generating unit configured to generate notification information if a write-copy completion message sent by a slave node in an available state is received, where the slave node in the available state is determined according to the new state identifier of each slave node; a notification transmission unit configured to transmit the notification information to the terminal device.

In some embodiments of the present application, the update unit is configured to: if the delay time is not received, determining that the slave node which does not send the delay time is in an unavailable state; and if the delay time is received and is greater than the preset delay time, determining that the slave node sending the delay time is in an unavailable state.

FIG. 8 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

It should be noted that the computer system 800 of the electronic device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 8, a computer system 800 includes a Central Processing Unit (CPU)801 that can perform various appropriate actions and processes, such as performing the methods described in the above embodiments, according to a program stored in a Read-Only Memory (ROM) 802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for system operation are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An Input/Output (I/O) interface 805 is also connected to bus 804.

The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.

In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. When the computer program is executed by the Central Processing Unit (CPU)801, various functions defined in the system of the present application are executed.

It should be noted that the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with a computer program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method described in the above embodiments.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the application. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal device, or a network device, etc.) to execute the method according to the embodiments of the present application.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (10)

1. A data processing method applied to a distributed system, wherein the distributed system comprises a master node and a plurality of slave nodes, the method comprising:

acquiring a state identifier of each slave node from the master node, wherein the state identifier is used for indicating whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node;

determining whether a target slave node in an available state exists in the distributed system according to the state identification of each slave node;

if the target slave node exists, sending a data reading request to the target slave node, and receiving response data which is returned by the target slave node and aims at the data reading request;

and if the target slave node does not exist, sending the data reading request to the master node, and receiving response data which is returned by the master node and aims at the data reading request.

2. The method of claim 1, further comprising:

sending a data writing request to the main node so that the main node writes data specified to be written by the data writing request;

and receiving notification information sent by the master node, wherein the notification information is generated after the master node receives a write-in copy completion message sent by the slave node in an available state.

3. The method of claim 1, wherein obtaining the status identifier of each slave node from the master node comprises:

acquiring a node state view from the master node, wherein the node state view comprises state identifiers of all slave nodes;

and determining the state identification of each slave node according to the node state view.

4. A data processing method applied to a distributed system, wherein the distributed system comprises a master node and a plurality of slave nodes, the method comprising:

sending a state identifier of each slave node to terminal equipment, wherein the state identifier is used for indicating whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node;

receiving a data reading request sent by the terminal device, wherein the data reading request is sent by the terminal device when the terminal device determines that no target slave node in an available state exists in the distributed system according to the state identification of each slave node;

and responding to the data reading request, and returning response data aiming at the data reading request to the terminal equipment.

5. The method of claim 4, further comprising:

receiving a data writing request sent by the terminal equipment;

and responding to the data writing request, writing data specified to be written by the data writing request, and generating an operation log, wherein the operation log is used for enabling each slave node to determine the data needing to be written.

6. The method of claim 5, further comprising:

and updating the state identifier of each slave node according to the delay time returned after the operation log is pulled by each slave node, so as to obtain a new state identifier of each slave node.

7. The method of claim 6, further comprising:

if a write-in copy completion message sent by the slave node in the available state is received, generating notification information, wherein the slave node in the available state is determined according to the new state identification of each slave node;

and sending the notification information to the terminal equipment.

8. The method according to claim 6, wherein updating the state identifier of each slave node according to a delay time returned after the operation log is pulled by each slave node, to obtain a new state identifier of each slave node, includes:

if the delay time is not received, determining that the slave node which does not send the delay time is in an unavailable state;

and if the delay time is received and is greater than the preset delay time, determining that the slave node sending the delay time is in an unavailable state.

9. A data processing apparatus for use in a distributed system having a master node and a plurality of slave nodes, the apparatus comprising:

the obtaining unit is configured to obtain a state identifier of each slave node from the master node, where the state identifier is used to indicate whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node;

a determining unit, configured to determine whether there is a target slave node in an available state in the distributed system according to the state identifier of each slave node;

the first sending unit is configured to send a data reading request to the target slave node if the target slave node exists, and receive response data which is returned by the target slave node and aims at the data reading request;

and the second sending unit is configured to send the data reading request to the master node and receive response data returned by the master node for the data reading request if the target slave node does not exist.

10. A data processing apparatus for use in a distributed system having a master node and a plurality of slave nodes, the apparatus comprising:

a third sending unit, configured to send a status identifier of each slave node to the terminal device, where the status identifier is used to indicate whether the slave node is in an available state, and the slave node in the available state stores a copy of data in the master node;

a receiving unit, configured to receive a data read request sent by the terminal device, where the data read request is sent by the terminal device when it is determined that there is no target slave node in an available state in the distributed system according to the state identifier of each slave node;

and the response unit is configured to respond to the data reading request and return response data aiming at the data reading request to the terminal equipment.

Images