CN115357600B

CN115357600B - Data consensus processing method, system, device, equipment and readable storage medium

Info

Publication number: CN115357600B
Application number: CN202211290793.6A
Authority: CN
Inventors: 张宾; 易简; 李清; 江勇; 张伟哲; 张宇
Original assignee: Peng Cheng Laboratory
Current assignee: Peng Cheng Laboratory
Priority date: 2022-10-21
Filing date: 2022-10-21
Publication date: 2023-02-03
Anticipated expiration: 2042-10-21
Also published as: CN115357600A

Abstract

The application discloses a data consensus processing method, a system, a device, equipment and a readable storage medium, wherein the method comprises the following steps: outputting a reading request to a target copy in a multi-copy cluster, and outputting an acquisition guide request to other copies in the multi-copy cluster; the multiple copy clusters are copy clusters of multiple key value intervals after the key value storage space and the operation log table are partitioned, wherein each copy of the key value interval is responsible for responding to an external request; after the target copy responds to the read request, receiving a response result returned by the target copy to finish the operation of the read request; after the other copies respond to the acquisition guide request, receiving guide manuals returned by the other copies; and if the number of the guide manuals with the same content is larger than the preset number, determining that the current multiple copy scenes meet the consistency requirement of the consensus protocol, and determining that the read request is valid. The application realizes a novel data consensus processing method.

Description

Data consensus processing method, system, device, equipment and readable storage medium

Technical Field

The present application relates to the field of data processing, and in particular, to a method, a system, an apparatus, a device, and a readable storage medium for data consensus processing.

Background

Multiple copy key-value stores need to provide strong consistency of data and high availability of client access. To achieve these goals, conventional multi-copy key-value stores are built using a common consensus protocol known as the raft protocol. And when the wooden raft protocol is used, one copy in the cluster is elected as a leader node, and the node is responsible for broadcasting the operation request sent by the client and adjusting the content sent to the client at the next time according to the feedback of other followers. When the leader finds enough follower copies to have recorded the current operation, the leader performs the operation and returns the results to the client.

However, under a typical three-copy configuration, the leader needs to process two follower messages for each client message received. Depending on the processing effect in an actual system, the time spent by the leader in processing one client message and one other duplicate message is similar, so in a three-copy configuration, only one third of the leader's processing power contributes to the client request, and the remaining two thirds are used to process the follower message. Therefore, the throughput performance of the raft protocol is poor in the current scenario where multi-copy key value storage is used. Where throughput is defined herein as the number of requests per second processed and responded to by the client.

Disclosure of Invention

In view of this, the present application provides a data consensus processing method, system, device, apparatus, and readable storage medium, which aim to improve throughput performance of processing client requests by multiple key value storage terminals.

In order to achieve the above object, the present application provides a data consensus processing method, including the following steps:

outputting a reading request to a target copy in a multi-copy cluster, and outputting an acquisition guide request to other copies in the multi-copy cluster; the multiple copy clusters are copy clusters of a plurality of key value intervals after key value storage space and an operation log table are partitioned, wherein the copy of each key value interval is responsible for responding to an external request;

after the target copy responds to the read request, receiving a response result returned by the target copy to finish the operation of the read request;

after the other copies respond to the acquisition guide request, receiving guide manuals returned by the other copies;

and if the number of the guide manuals with the same content is larger than the preset number, determining that the current multiple copy scenes meet the consistency requirement of the consensus protocol, and determining that the read request is valid.

Illustratively, the outputting the read request to the target copy in the multi-copy cluster and outputting the get boot request to the other copies in the multi-copy cluster includes:

outputting a get boot request to each copy;

receiving a guide manual returned when each copy responds to the acquisition guide request; the guide manual consists of a mapping relation from a key value interval to a copy identifier and chronological value data, wherein the copy identifier is obtained from a key partition identifier and a partition mapping relation of the guide manual, and the key partition identifier is obtained by modulo the partition number of the guide manual by the key identifier;

and if the number of the received guide manuals is larger than the preset number, determining that the access connection is successful, outputting a reading request to a target copy in the multi-copy cluster, and outputting an acquisition guide request to other copies in the multi-copy cluster.

Illustratively, the outputting the read request to the target copy in the multi-copy cluster includes:

after determining that the access connection is successful, determining a key of a key value interval of the read request to be received;

based on the guide manual, calculating a copy identifier corresponding to the key;

and taking the copy corresponding to the copy identification as a target copy.

Illustratively, after receiving the guidance manual returned by each copy in response to the get guidance request, the method includes:

traversing the age value data in each guide manual;

determining the latest updated age value data;

determining the latest version of the guide manual based on the latest updated age value data;

the latest version of the guidance manual is appended to the read request for subsequent use.

Illustratively, the response result includes a guidance manual of the target copy, and after the target copy responds to the read request, receiving a response result returned by the target copy includes:

identifying first age value data of a guide book of the target copy and identifying second age value data of the guide book of the latest version;

if the first-generation value data and the second-generation value data have difference in updating times, marking the target copy as a problem copy;

and after receiving the guide manual returned by the other copies, if the guide manual returned by the other copies is the same as the guide manual of the target copy, canceling the mark of the question copy.

Illustratively, to achieve the above object, the present application further provides a data consensus processing system, where the data consensus processing system includes a client and a plurality of copy clusters:

the client is used for outputting a reading request to a target copy in a multi-copy cluster and outputting an acquisition guide request to other copies in the multi-copy cluster, receiving a response result returned by the target copy after the multi-copy cluster responds to the consensus request so as to complete the operation of the reading request, receiving the number of guide manuals with the same content returned by the other copies, wherein the number of the guide manuals is greater than a preset number, determining that the current multi-copy scene meets the consistency requirement of the consensus protocol, and determining that the reading request is valid;

the multi-copy cluster is used for receiving the read request and the acquisition guide request and responding to the read request and the acquisition guide request, wherein any copy meets the effect of simultaneously responding to the read request and the acquisition guide request, and the request is processed in parallel in two different event cycles, wherein the read request is processed by using a main event cycle, and the acquisition guide request is processed by using a special event cycle.

Illustratively, the multi-replica cluster includes a plurality of copies of key-value intervals, any copy of the multi-replica cluster:

broadcasting the self-stored guide manual to other copies except the self-stored guide manual within a preset period;

if the offline copies which are not broadcasted to other copies within the preset time length exist, the offline copies are removed from the guidance manual;

and broadcasting the updated guide manual to the other copies for the other copies to update the guide manual of the other copies.

Illustratively, to achieve the above object, the present application further provides a data consensus processing apparatus, including:

the data consensus processing device comprises:

the output module is used for outputting a reading request to a target copy in the multi-copy cluster and outputting an acquisition guide request to other copies in the multi-copy cluster; the multiple copy clusters are copy clusters of a plurality of key value intervals after key value storage space and an operation log table are partitioned, wherein the copy of each key value interval is responsible for responding to an external request;

a first receiving module, configured to receive a response result returned by the target copy after the target copy responds to the read request, so as to complete an operation of the read request;

the second receiving module is used for receiving the guide manuals returned by the other copies after the other copies respond to the acquisition guide request;

and the judging module is used for determining that the current multiple copy scenes meet the consistency requirement of the consensus protocol and determining that the reading request is effective if the number of the guide manuals with the same content is larger than the preset number.

Illustratively, to achieve the above object, the present application further provides a data consensus processing device, including: the data consensus processing method comprises the steps of a memory, a processor and a data consensus processing program stored on the memory and executable on the processor, the data consensus processing program being configured to implement the steps of the data consensus processing method as described above.

Illustratively, to achieve the above object, the present application further provides a computer readable storage medium, on which a data consensus processing program is stored, which when executed by a processor implements the steps of the data consensus processing method as described above.

Compared with the prior art that when a client accesses a multi-copy key value storage end, the client needs to send a read request to one leader copy, the leader copy copies the request to other follower copies and collects voting information of the follower copies, so that the throughput performance of the multi-copy key value storage end for processing the request of the client per second is poor. The client side directly and simultaneously carries out request interaction with all the copies in the multi-copy cluster, directly obtains a response result of the target copy to the read request and the obtained guide manuals of other copies, and determines that the current multi-copy cluster meets the consistency requirement of the consensus protocol and the current read request is effective after the number of the guide manuals with the same content is larger than the preset number. Therefore, in the application, the client directly performs request interaction with all the copies in the multi-copy cluster, and the consistency requirement and the effectiveness of the read request are directly determined according to the guide manuals returned by other copies, so that the throughput performance of the multi-copy cluster for processing the client request per second is improved.

Drawings

FIG. 1 is a schematic flow chart diagram illustrating a first embodiment of a data consensus processing method according to the present application;

fig. 2 is a schematic diagram of a single consensus communication process of a conventional client accessing multiple key value storage nodes in the present application;

FIG. 3 is a flow diagram of a novel data read mechanism of the present application;

FIG. 4 is a client message structure in the present application;

fig. 5 is a schematic structural diagram of a hardware operating environment according to an embodiment of the present application.

The implementation, functional features and advantages of the object of the present application will be further explained with reference to the embodiments, and with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.

Referring to fig. 1, fig. 1 is a schematic flow chart of a first embodiment of the data consensus processing method.

The embodiments of the present application provide an embodiment of a data consensus processing method, and it should be noted that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in an order different from the order shown or described here. For convenience of description, the following omits to perform various steps of a subject description data consensus processing method including:

step S110: outputting a reading request to a target copy in a multi-copy cluster, and outputting an acquisition guide request to other copies in the multi-copy cluster; the multiple copy clusters are copy clusters of a plurality of key value intervals after key value storage space and an operation log table are partitioned, wherein the copy of each key value interval is responsible for responding to an external request;

when a traditional client accesses a multi-copy key value storage end, the client reads one leader copy in the multi-copy key value storage, the leader copy interacts with a follower copy, and after a consistency requirement is met, the leader copy returns a read operation result and consistency judgment to the client, so that the throughput performance is poor when the client accesses the multi-copy key value storage, and the throughput is defined as the number of requests processed and responded by the client per second.

Referring to fig. 2 and fig. 2 are schematic diagrams of a single consensus communication process in which a conventional client accesses a multi-copy key value storage, when the client in fig. 2 sends a request to the multi-copy key value storage, the request is sent to a leader copy only, the leader copy copies the request and sends the request to other follower copies, voting information of the follower copies is collected, whether the consensus requirement of consensus processing is met is determined according to the voting information, after the leader collects votes, the client normally responds to the request of the client and returns to the client if the consensus requirement is met, and an error message is returned to the client if the consensus requirement is not met.

According to the conventional consensus communication flow in fig. 2, the client interacts only with the leader copy, and the information processing capability of the leader copy is mostly used for the process of copying the request and collecting votes with the follower copy, and thus, the throughput performance in the conventional consensus communication flow is poor.

In this embodiment, an access mode in which a conventional client sends a request to one leader copy is changed, a read request is output to a target copy in the multi-copy cluster, and meanwhile, a get boot request is output to other copies in the multi-copy cluster. The effect of sending the request to a plurality of copies at the same time is achieved, and the number of the plurality of copy key value storage ends which process and respond to the clients per second is increased.

The multiple copy clusters are copy clusters of multiple key value intervals after key value storage space and an operation log table are partitioned, only one leader copy exists in the original multiple copy clusters, and other copies are all used as follower copies of the leader copy.

When the key value intervals of the key value storage space are divided, the key value storage space of one or more server ends can be divided, each obtained key value interval has a part of leaders to take charge of responding to external requests, and meanwhile, the granularity of the divided key value storage space is the same as the granularity of the division of the operation log table, so that the one-to-one correspondence of the key value storage space and the operation log table is ensured.

In order to increase the number of client requests processed by multiple copy key value storage terminals per second, so that each copy in a multiple copy cluster processes and responds to the client requests, some leader-free protocols (such as EPaxos, consistency algorithm, where any copy can submit logs) imply that the above purpose is achieved by using a "per-operation leader" protocol (per-operation leader, action leader, operation leader), but these protocols have high communication overhead. In practice, however, the same effect can be achieved for key-value stores using a "per-key-value leader" protocol. Therefore, in the present embodiment, the "per-key value leader" is used to achieve the effect of no leader, so that each copy processes and responds to the client request, that is, the copy of each key value interval is responsible for responding to the external request.

Illustratively, when a read request and a boot acquisition request are output to a multi-copy cluster, different requests are output to different copies according to the purpose of the different requests.

The current client needs to read the key value of a certain copy in the multi-copy cluster, the copy is used as a target copy, a read request is output to the target copy, and the client can obtain the key value of the target copy after the target copy responds to the read request.

The current client needs to judge the validity of the operation of the read request, and then consistency judgment between the copies needs to be made according to the condition that other copies of the target copy are removed from the multi-copy cluster, so that whether the size of the key value obtained by the read request of the current client is valid is determined, at this time, an acquisition guide request is output to the other copies, data for judging consistency is obtained from the other copies, and the consistency and the validity are judged according to the data. The acquisition guide request is a request for acquiring the judgment consistency data in other copies.

a, step a: outputting a get boot request to each copy;

before starting the client, the data stored by the client needs to be updated to ensure that the situation of consensus consistency is directly determined by the client when the client accesses the multi-copy cluster, so that the requirement of collecting votes by a leader copy to judge consistency is avoided, and the throughput performance is further improved, therefore, the update state of the data used for judging the consistency requirement at the client needs to be ensured, and the data can change along with the update of the copies in the multi-copy cluster, so the update process of the data is as follows: the client collects the corresponding data in each copy, and determines the data with the latest update time from the received corresponding data, namely the data with the latest content.

Therefore, an acquisition guide request for acquiring corresponding data is formulated, the acquisition guide request is output to each copy, and when the copy responds to the acquisition guide request, the corresponding data is returned to the client, so that the effect of collecting data is achieved.

Step b: receiving a guide manual returned when each copy responds to the acquisition guide request; the guide manual consists of a mapping relation from a key value interval to a copy identifier and chronological value data, wherein the copy identifier is obtained from a key partition identifier and a partition mapping relation of the guide manual, and the key partition identifier is obtained by modulo the partition number of the guide manual by the key identifier;

after the acquisition guide request of the client is received by the replica, the acquisition guide request is processed in a special event loop and is responded, the acquisition guide request is a request that the client wants to acquire corresponding data from the multi-replica cluster, therefore, when the acquisition guide request is responded by the replica, special data used for judging the consistency requirement is returned, and the returned special data is a guide manual.

Illustratively, there is one guide book per copy, which contains two parts: the key value subarea of each copy corresponds to the mapping relation of the copy identification and the age value data of the guide manual, and a novel data structure is formed by the two data contents, and the data structure is the guide manual.

The copy identification is obtained by the key partition identification and the partition mapping relation of the guide manual, and the key partition identification is obtained by the key identification modulo the partition number of the guide manual.

The value of the age value is used as a mark when the guide manual is updated and iterated, a new age value must be assigned when the copy is modified and updated each time, the age value must be larger than the known value in the previous copy, so that the effect of the age value for updating and iterating is ensured, the conflict of the age value data is avoided, and meanwhile, the value obtained by modulo the number of all copies of the age value must be equal to the number of the copy. The number of the copy itself is that when a plurality of copy clusters are obtained after the key value storage interval is divided, each copy is sequentially numbered, for example, in the case of three copies, the number is a, b, c or 1,2,3.

Illustratively, the guide manual only allows the copy to be created and modified when in use, and secondly, the synchronization of the guide manual is performed between the copies through a special interaction protocol, namely, a timeout mechanism similar to that in a raft protocol is used, so that the guide manual stored in each copy is synchronously updated.

The reason why the copy modifies the guide manual is that some mapping relations must be removed or modified from the guide manual in order to achieve fault tolerance, so that the mapping relations held by the guide manual are all correct, and the client finds the copy which is not in error by means of the acquired guide manual of the latest version and judges that the copy has an error report condition.

The special interactive protocol content among the copies mainly ensures the state of the copies contained in the guide manual, in the process, the survival state is declared among the copies through periodically sending information, and the overtime copies in the guide manual can be correspondingly modified when the copies find that other copies are overtime.

Step c: and if the number of the received guide manuals is larger than the preset number, determining that the access connection is successful, outputting a reading request to a target copy in the multi-copy cluster, and outputting an acquisition guide request to other copies in the multi-copy cluster.

After the client is started, a guiding request is sent to the multi-cluster-copy cluster, and the multi-cluster-copy is not determined to respond to the client currently, so that when the multi-cluster-copy responds to the guiding request and returns a guiding manual to the client, the number of guiding manuals, namely the number of copies responding to the client, is judged, and when the number is larger than the preset number, the client and the multi-cluster-copy are determined to be successfully connected, namely the client is successfully accessed and connected.

The preset number may be half of the number of copies in the multi-copy cluster, and more than half, that is, the preset number is greater than or equal to half of the number of copies in the multi-copy cluster. Therefore, according to the preset number as a measurement standard, after the number of the obtained guidance manuals is larger than half of the number of the copies in the multi-copy cluster, the current client can be determined to be successfully connected with the multi-copy cluster.

And after the client is successfully connected with the multiple copy clusters, outputting a reading request to the target copy, and outputting an acquisition guide request to other copies in the multiple copy clusters.

The read request comprises a guide manual attached from a client, when the read request is output to the target copy, the copy can process and respond to the read request in a main event cycle, and meanwhile, any copy in the multi-copy cluster can process and respond to the read request.

In addition, any copy processes the read request in the main event cycle, and any copy processes the acquisition guide request in the special event cycle, namely any copy uses different event cycles when processing different requests, so that the effect that any copy can process the read request and the acquisition guide request simultaneously is achieved, and the efficiency of processing the request is improved.

step d: after determining that the access connection is successful, determining a key of a key value interval of the read request to be received;

after the access connection is determined to be successful, the key value of the key value interval to be read in the multi-copy cluster needs to be determined, the key value comprises a relation table corresponding to the key and the key value, namely the key value of the key value interval to receive the read request is determined, and the key of the key value interval to receive the read request needs to be determined first.

Step e: based on the guide manual, calculating a copy identification corresponding to the key;

step f: and taking the copy corresponding to the copy identification as a target copy.

From the above description, the guidance manual is composed of the year value data and the mapping relationship from a key value interval to a copy identifier, where the copy identifier is obtained from the partition mapping relationship between the key partition identifier and the guidance manual, and the key partition identifier is obtained by modulo the partition number of the guidance manual by the key identifier, so that, according to the key of the key value interval and the guidance manual, the key is equivalent to the key partition identifier, thereby calculating the copy identifier corresponding to the key.

Furthermore, the corresponding copy can be determined according to the copy identifier, that is, the copy of the read request to be received is determined from the multiple-copy cluster, the copy is used as the target copy, the copies except the target copy in the multiple-copy cluster are used as other copies, and the client outputs the acquisition guide request to the other copies.

step g: traversing the age value data in each guide manual;

when the client outputs the reading request to the target copy, the reading request is accompanied by a guide manual held by the client, so that a novel data reading mechanism based on the guide manual is realized.

When the client side is not in access connection with the multiple copy cluster sides, the multiple copy cluster sides can update in real time, and when the copies are updated, the updated or modified data contents are recorded in the guidance manual in a chronological value data mode, so that when the client side is in access connection with the multiple copy cluster sides, the client side needs to update the own guidance manual.

Therefore, when the client is started, the acquisition guide request is output to other copies, the guide manuals of the other copies are acquired, and the guide manuals of the latest version are determined from the guide manuals, so that the novel data reading mechanism is guaranteed to be normally implemented.

To determine the latest version in the guide book, the chronological data of the received guide book is traversed.

Step h: determining the latest updated age value data;

step i: determining the latest version of the guide manual based on the latest updated age value data;

and after traversing the age value data of the guide manual, determining the latest updated age value data.

According to the characteristics of the age value data, the age value data only produces an updating effect after the guidance manual is updated or modified, and the value of each time the age value data is updated is larger than the previous value, for example, a copy is updated nine times currently, the age value data can be marked as nine, and the copy is updated to ten times further currently.

Therefore, the latest update chronological value data is determined as long as the chronological value data having the largest value among the chronological value data is determined.

After the latest updating data is determined, the latest guiding manual, namely the latest guiding manual, can be directly determined.

Step j: the latest version of the guidance manual is appended to the read request for subsequent use.

After the latest guide manual is determined, the guide manual is attached to a read request used subsequently, namely the guide manual is attached to the read request output to the target copy, so that a novel data reading mechanism is guaranteed.

Referring to fig. 3, fig. 3 is a flow chart of a novel data reading mechanism.

Currently, two clients and three duplicate terminals are taken as examples to describe an interaction flow of a novel data reading mechanism, and the chronological value data of a guide manual of the current client and the chronological value data of a guide manual of the duplicate terminal are both 5.

When the client 2 performs a write operation on the copy 2, an effect that 7 is given to x is generated in the copy 2, that is, the copy 2 is modified once, at this time, the copy 2 copies the modification of itself, and the write operation that 7 is given to x is broadcasted to the copy 1 and the copy 3 in a broadcast manner, so that it is ensured that the copy 1 and the copy 3 are both in an updated state.

When the client 1 outputs a read request to the copy 1, it simultaneously outputs an acquisition guidance request (read guidance manual) to the other copies (copy 2 and copy 3).

Referring to fig. 4, fig. 4 is a client message structure. The message structure content of the client in fig. 4 corresponds to the flow content in fig. 3.

Therefore, in the above-mentioned interaction flow in fig. 3, the write request of the client obtains the chronological value data, the copy flag (request id), the key, and the operation result of the guidance manual;

the read request of the client acquires the related data of the target copy: chronological value data, copy labels and keys of the guidance manual;

the acquisition guide request of the client acquires the copy marks of the guide manuals of other copies;

and determining whether the current request is successful and effective according to the judgment consistency requirement.

Step S120: after the target copy responds to the read request, receiving a response result returned by the target copy to finish the operation of the read request;

after the target copy receives the read request of the client segment, the read request is processed in the main event cycle, and a response result is returned to the client, so that the operation of the read request is completed.

In this process, it is first determined whether the operation of the read request is successful, that is, whether the key value of the target copy is successfully read, and it is subsequently determined whether the current read request is valid.

step k: identifying first age value data of a guide book of the target copy and identifying second age value data of the guide book of the latest version;

when the guidance manual attached to the read request is compared with the guidance manual of the target copy, the chronological value data in the guidance manual is compared, so that whether the guidance manual of the target copy and the guidance manual of the client are matched can be determined.

Therefore, it is necessary to identify the age value data of the guide book in the read request and the target copy, where the age value data of the guide book of the target copy is the first age value data, and where the age value data of the guide book of the latest version attached to the read request is the second age value data.

Step l: if the first-generation value data and the second-generation value data have difference in updating times, marking the target copy as a problem copy;

when comparing first-generation data and second-generation data, there are two situations, one is: the first generation value data and the second generation value data are consistent, namely, the target copy returns a response result containing the key value and the guidance manual. The other is as follows: the first-generation value data and the second-generation value data do not coincide with each other, that is, the number of times the first-generation value data is updated is greater than or less than that of the second-generation value data.

When the first generation value data and the second generation value data are inconsistent, the response result returned by the target copy only contains the guide manual, so that when the two generation value data are inconsistent, the situation that the client fails to perform the read request on the target copy is determined to exist, at this time, the target copy with the failed read request needs to be subjected to error reporting and marking, and the target copy is marked as a problem copy.

Step m: and after receiving the guide manuals returned by the other copies, if the guide manuals returned by the other copies are the same as the guide manuals of the target copies, canceling the mark of the question copy.

And after the problem copy is marked, the problem copy is put aside, and after a certain time, whether the problem copy has a real problem or not is judged again. The problems include network failures, outage areas, and the like.

Therefore, when the problem copy is put on hold, the guidance manual returned by the other copy is normally acquired, the current guidance manual of the client is updated according to the age value data of the guidance manual of the other copy, and the problem copy is judged again whether the guidance manual is consistent or not by using the updated guidance manual.

Illustratively, the guidance manual of the current client is updated ten times, the guidance manual of the target copy is updated eleven times, the current target copy is marked as the problem copy according to the logic for marking the problem copy, and the guidance manual of the other copies which is updated ten times and eleven times exists, then the client will regard the guidance manual which is updated eleven times as the latest guidance manual, and compare the updated guidance manual with the guidance manual of the target copy, and both are updated eleven times, so the marking of the problem copy is cancelled. Namely, when the updated guide manual is used for completing the reading request, the problem copy is proved to have no problem, the reading request operation is directly completed, and the key value of the target copy is obtained.

When the problem copy is put aside, the client continuously collects the guide manuals of other copies, updates the guide manuals stored by the client according to the received guide manuals of other copies to ensure that the guide manuals of the client keep the latest state, and simultaneously ensures that the number of the guide manuals collected by the client for updating exceeds the preset number due to the fact that the initial client outputs the guide acquisition request to the other copies with the number exceeding the preset number in the updating process, so that the client can continuously collect the guide manuals for updating to update the latest state through the logic of gradual convergence after continuously collecting the guide manuals.

If the problem copy is placed and the placing time exceeds a certain time, and the problem copy is checked for consistency, the guide manual of the problem copy is not consistent with the guide manual of the client, the client removes the problem copy, and modifies the guide manual of other copies through an interaction protocol among the copies, namely removes the problem copy contained in the guide manual of other copies, and broadcasts the modified guide manual to all other copies, so that the guide manual of the copies in the whole multi-copy cluster is in the latest state.

After the target copy processes the read request, a corresponding response action is performed on the read request, a response result is returned to the client, and the client receives the response result returned by the target copy at this time, so that the operation of the read request is completed.

The return result comprises a guidance manual of the target copy and a key value of the target copy.

Illustratively, the client outputs a read request to the target copy, and after the target copy processes the read request, there are two results, one is to normally execute the read request operation, and the other is to abnormally execute the read request operation.

When the target copy normally executes the read operation request, the response result returned by the target copy at this time includes the key value of the target copy and the guidance manual of the target copy.

When the target copy abnormally executes the read operation request, the response result returned by the target copy at this time only comprises the guidance manual of the target copy.

And when judging whether the target copy can normally execute the read request operation, comparing whether a guide manual from the client side attached in the read request is consistent with a guide manual in the target copy, if so, normally executing the read request by the target copy, wherein a response result returned to the client side comprises a key value of the target copy and the guide manual of the target copy, if not, abnormally executing the read request by the target copy, and a response result returned to the client side only comprises the guide manual of the target copy.

Step S130: after the other copies respond to the acquisition guide request, receiving guide manuals returned by the other copies;

after the other copies respond to the acquisition guide request, the other copies process the acquisition guide request in a special event loop and return guide manuals held by the other copies to the client.

In the current state, the target copy has completed the response to the read request, that is, the target copy has completed the return of the response result, at this time, it can be determined whether the read request of the current client is completed according to the response result, further, it needs to determine whether the current multi-copy cluster is consistent through a consensus protocol, that is, it is determined whether the current multi-copy scene meets the requirement of consistency, and if the requirement of consistency is met, it is determined that the current read request is valid.

In the consistency judgment process, the consistency judgment is carried out according to the consistency of the guide manuals in other copies.

Step S140: and if the number of the guide manuals with the same content is larger than the preset number, determining that the current multiple copy scenes meet the consistency requirement of the consensus protocol, and determining that the read request is effective.

The predetermined number is set forth above, i.e., the predetermined number is any number greater than half of the total number of copies in the multi-copy cluster.

When the obtained guide manuals of other copies have the same content and the number of the guide manuals with the same content is larger than the preset number, the consistency requirement of the consensus protocol in the current multiple copy scenes can be judged, the judging process is similar to the stated judgment on whether the connection between the client side and the multiple copy cluster terminal is successful or not, when the number of the guide manuals with the same content is larger than the preset number, the updating states of most of the guide manuals of the copies in the current multiple copy cluster can be judged to be the same, the consistency requirement of the consensus protocol is further determined, and the validity of the current reading request is determined.

Illustratively, based on the first embodiment of the data consensus processing method of the present application, a second embodiment is provided, where the method further includes:

step n: broadcasting the self-stored guide manual to other copies except the self-stored guide manual within a preset period; the copy time is announced to be alive by periodically sending information, and the copy can modify the guidance manual only when other copies are found to be out of time. The process of fault discovery resembles the timeout mechanism of the "raft protocol".

This message is named Gossip message (the message is used for the interaction protocol between the copies), and each copy broadcasts its own guidance manual to the other copies in a random 150ms to 300ms time.

Wherein the time period of 150ms to 300ms is a predetermined period.

In a preset period, the guidance manual stored by the user is broadcasted to other copies except the user, namely, the state of the copy is informed to other copies through the process of broadcasting messages, so that the copy with problems can be detected.

Step o: if the offline copies which are not broadcasted to other copies within the preset time length exist, the offline copies are removed from the guidance manual;

step p: and broadcasting the updated guide manual to the other copies for the other copies to update the guide manual of the other copies.

Illustratively, when a copy A finds that some other copy B does not send the message to itself within 1500ms, A judges that B is off-line, modifies its own guidance manual, removes B from the guidance manual, and broadcasts a new guidance manual. After receiving the new boot manual, other copies replace themselves locally, so that the boot manual of the whole cluster can be updated.

The preset duration is 1500ms, and the duration can be adjusted accordingly according to actual conditions.

Wherein, the copy B is an offline copy.

In this embodiment, the update state of the guidance manual held by the copies in the multiple copy clusters affects a read operation mechanism of the client, and after the client is successfully connected to the multiple copy clusters, the client needs to invoke the latest version of the guidance manual to complete subsequent key value reading of the target copy, that is, it needs to be ensured that most of the copies of the guidance manual in the multiple copy clusters are in the latest state, so that the guidance manual held by the client is broadcast to other copies except the client within a preset period, and if there is an offline copy that is not broadcast to other copies within a preset time period, the offline copy is removed from the guidance manual, and the updated guidance manual is broadcast to other copies to update the guidance manual of the client, so that most of the guidance manual held by the copies in the multiple copy clusters are in the latest state.

In addition, the present application also provides a data consensus processing apparatus, which includes:

and the judging module is used for determining that the current multiple copy scenes meet the consistency requirement of the consensus protocol and determining that the reading request is valid if the number of the guide manuals with the same content is larger than the preset number.

Illustratively, the output module includes:

the output submodule is used for outputting an acquisition guide request to each copy;

the receiving submodule is used for receiving a guide manual returned when each copy responds to the acquisition guide request; the guide manual consists of a mapping relation from a key value interval to a copy identifier and chronological value data, wherein the copy identifier is obtained from a key partition identifier and a partition mapping relation of the guide manual, and the key partition identifier is obtained by modulo the partition number of the guide manual by the key identifier;

the first judgment sub-module is used for determining that the access connection is successful if the number of the received guide manuals is larger than the preset number, outputting a reading request to a target copy in the multi-copy cluster, and outputting an acquisition guide request to other copies in the multi-copy cluster;

the first determining submodule is used for determining a key of a key value interval of the read request to be received after the access connection is determined to be successful;

the calculation submodule is used for calculating the copy identification corresponding to the key based on the guide manual;

and the second determining submodule is used for taking the copy corresponding to the copy identification as a target copy.

Illustratively, the receiving sub-module includes:

the traversing unit is used for traversing the chronological value data in each guide manual;

a first determination unit configured to determine latest updated chronological value data;

a second determination unit configured to determine a latest version of the guidance manual based on the latest updated chronological value data;

and the adding unit is used for adding the guide manual of the latest version to the read request used subsequently.

Illustratively, the receiving submodule includes:

an identification submodule for identifying first-generation value data of the guide manual of the target copy and identifying second-generation value data of the guide manual of the latest version;

the second judgment sub-module is used for marking the target copy as a problem copy if the first generation value data and the second generation value data have different updating times;

and the third judgment submodule is used for canceling the mark of the question copy if the guide manual returned by the other copies is the same as the guide manual of the target copy after receiving the guide manual returned by the other copies.

The specific implementation of the data consensus processing apparatus is substantially the same as that of the embodiments of the data consensus processing method, and is not described herein again.

In addition, the present application further provides a data consensus processing system, where the data consensus processing system includes a client and multiple copy clusters:

Illustratively, any replica in the multi-replica cluster:

if an offline copy which is not broadcasted to other copies within a preset time length exists, the offline copy is removed from the guide manual;

In addition, the application also provides data consensus processing equipment. As shown in fig. 5, fig. 5 is a schematic structural diagram of a hardware operating environment according to an embodiment of the present application.

For example, fig. 5 is a schematic structural diagram of a hardware operating environment of the data consensus processing device.

As shown in fig. 5, the data consensus processing device may comprise a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 are in communication with each other through the communication bus 504, and the memory 503 is used for storing computer programs; the processor 501 is configured to implement the steps of the data consensus processing method when executing the program stored in the memory 503.

The communication bus 504 mentioned above as the data consensus processing device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus 504 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface 502 is used for communication between the data consensus processing device and other devices.

The Memory 503 may include a Random Access Memory (RMD) and a Non-Volatile Memory (NM), such as at least one disk Memory. Optionally, the memory 503 may also be at least one storage device located remotely from the processor 501.

The Processor 501 may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

In addition, an embodiment of the present application also provides a computer-readable storage medium, where a data consensus processing program is stored on the computer-readable storage medium, and when the data consensus processing program is executed by a processor, the data consensus processing method implements the steps of the data consensus processing method described above.

The specific implementation of the computer-readable storage medium of the present application is substantially the same as the embodiments of the data consensus processing method described above, and is not described herein again.

It should be noted that, in this document, the terms include, or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that includes a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, the inclusion of a term comprising an element defined by … … does not preclude the presence of additional like elements in a process, method, article, or system that comprises the element.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present application may be substantially or partially embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.

The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.

Claims

1. A data consensus processing method is characterized by comprising the following steps:

the outputting a read request to a target replica in a multi-replica cluster and outputting an acquire steering request to other replicas in the multi-replica cluster includes:

outputting a get boot request to each copy;

receiving a guide manual returned when each copy responds to the acquisition guide request; the guidance manual consists of a year value data and a mapping relation from a key value interval to a copy identifier, wherein the copy identifier is obtained from a key partition identifier and a partition mapping relation of the guidance manual, and the key partition identifier is obtained by modulo the partition number of the guidance manual by the key identifier;

if the number of the received guide manuals is larger than the preset number, determining that the access connection is successful, outputting a reading request to a target copy in a multi-copy cluster, and outputting an acquisition guide request to other copies in the multi-copy cluster;

and if the number of the guide manuals with the same content is larger than the preset number, determining that the current multiple copy scenes meet the consistency requirement of the consensus protocol, and determining that the read request is effective.

2. The data consensus processing method of claim 1, wherein outputting the read request to a target replica in a multi-replica cluster comprises:

and taking the copy corresponding to the copy identification as a target copy.

3. The data consensus processing method of claim 2, wherein after receiving the guidance manual returned by each of the copies in response to the get guidance request, comprising:

traversing the age value data in each guide manual;

determining the latest updated age value data;

4. The data consensus processing method of claim 3, wherein the response result comprises a guidance manual of the target copy, and receiving the response result returned by the target copy after the target copy responds to the read request comprises:

5. A data consensus processing system, characterized in that the data consensus processing system comprises a client and a multi-copy cluster:

the client is further used for outputting the acquisition guide request to each copy and receiving a guide manual returned when each copy responds to the acquisition guide request; the guide manual consists of a mapping relation from a key value interval to a copy identifier and chronological value data, wherein the copy identifier is obtained from a key partition identifier and a partition mapping relation of the guide manual, and the key partition identifier is obtained by modulo the partition number of the guide manual by the key identifier; if the number of the received guide manuals is larger than the preset number, determining that the access connection is successful, outputting a reading request to a target copy in a multi-copy cluster, and outputting an acquisition guide request to other copies in the multi-copy cluster;

6. The data consensus processing system of claim 5, wherein any replica in the multi-replica cluster:

and broadcasting the updated guidance manual to the other copies for the other copies to update own guidance manual.

7. A data consensus processing apparatus, characterized in that the data consensus processing apparatus comprises:

the output module is further used for outputting a guide acquisition request to each copy;

8. A data consensus processing device, characterized in that the device comprises: a memory, a processor, and a data consensus processing program stored on the memory and executable on the processor, the data consensus processing program being configured to implement the steps of the data consensus processing method as claimed in any one of the claims 1 to 4.

9. A computer-readable storage medium, having stored thereon a data consensus processing program, which when executed by a processor, performs the steps of the data consensus processing method according to one of the claims 1 to 4.