CN114448781A - Data processing system - Google Patents
Data processing system Download PDFInfo
- Publication number
- CN114448781A CN114448781A CN202111583497.0A CN202111583497A CN114448781A CN 114448781 A CN114448781 A CN 114448781A CN 202111583497 A CN202111583497 A CN 202111583497A CN 114448781 A CN114448781 A CN 114448781A
- Authority
- CN
- China
- Prior art keywords
- service
- service node
- node
- data
- voting information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 47
- 238000000034 method Methods 0.000 claims description 36
- 230000008569 process Effects 0.000 claims description 30
- 230000004044 response Effects 0.000 claims description 25
- 238000010586 diagram Methods 0.000 description 20
- 238000013461 design Methods 0.000 description 19
- 230000009977 dual effect Effects 0.000 description 17
- 230000006870 function Effects 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A data processing system comprises a first interface, a second interface, a service routing layer, a master service node and a slave service node, wherein the first interface is used for receiving a first service request aiming at the master service node, the second interface is used for receiving a second service request aiming at the slave service node, and the service routing layer is used for routing the first service request to the master service node, routing a read service in the second service request to the slave service node and routing a write service in the second service request to the master service node. The system can execute the service at the same time through the master service node and the slave service node, the throughput of the system is effectively improved, the data of the latest version in the system can be ensured to be directly written by the master service node, the number of steps passed by the intermediate flow of the writing operation is further saved, and the purpose of reducing the time delay of the system is achieved.
Description
Technical Field
The invention relates to the technical field of data synchronization, in particular to a data processing system.
Background
With the development and popularization of cloud computing technology, more and more industries turn traditional self-built data centers to the cloud end due to the requirements of cost and business development. The cloud is adopted to replace the traditional self-built data center to store data, so that the operation and maintenance cost of an enterprise can be greatly reduced.
At present, in the field of cloud storage, enterprises usually select a "primary and standby" mode or a "dual active" mode to realize disaster recovery in different places. The main/standby mode is a mode in which only the main service node provides a read-write function for the external service data at the same time, and in this mode, the slave service node only needs to receive the data copied by the main service node, and does not need to implement the read-write function for providing the service data to the upper layer application. Obviously, although the operation complexity of the "master/standby" mode is low, the throughput of the system is obviously reduced because only one service node can execute services at the same time. The "dual active" mode refers to a mode in which the master service node and the slave service node can provide a read-write function of service data to the outside at the same time, and although the throughput of the system can be increased, in order to ensure the consistency of data storage, each write of the master service node and the slave service node needs to synchronize version number and version information with each other before a successfully written message is returned, which obviously increases additional delay overhead due to more steps of the synchronization operation. Therefore, no matter the mode is the 'main/standby' mode or the 'double active' mode, the effects of improving the system throughput and reducing the time delay cannot be considered.
In view of the foregoing, there is a need for a data processing system that can achieve the dual effects of improving system throughput and reducing latency.
Disclosure of Invention
The application provides a data processing system, which is used for realizing double effects of improving system throughput and reducing time delay.
The present application provides a data processing system comprising: the system comprises a first interface, a second interface, a service routing layer, a master service node and a slave service node, wherein the first interface is used for receiving a first service request aiming at the master service node, the second interface is used for receiving a second service request aiming at the slave service node, the service routing layer is used for routing the first service request to the master service node, routing a read service in the second service request to the slave service node and routing a write service request in the second service request to the master service node, the master service node is used for processing the write service request in the first service request and the second service request, and the slave service node is used for processing the read service request in the second service request.
In the above design, the operation process of the service routing layer is transparent for the client, so from the perspective of the client, the first interface and the second interface concurrently accept the service request, that is, the master service node and the slave service node can receive two concurrent service requests at the same time through the first interface and the second interface, and further concurrently execute the service, and therefore, compared with a mode in which the "master-slave" mode can only execute the service by the master service node at the same time, the system throughput of the scheme is 2 times that of the master-slave mode. Moreover, by uniformly placing the write service at the side of the master service node for execution, the latest version of data in the system can be ensured to be directly written by the master service node (namely the master service node writes the version information to the slave service node synchronously), and compared with a mode that the master service node and the slave service node in a double-active mode can complete data synchronization only by synchronizing the version number and the version information, the method can save the number of steps passed by an intermediate flow and achieve the purpose of saving time delay. Therefore, the design can achieve the double effects of improving the system throughput and reducing the time delay.
In one possible design, when the first service request includes a read service, the primary service node is specifically configured to: and acquiring the data to be read from the local storage of the main service node, and returning the data to be read to the client. According to the design, when the main service node executes the reading operation, the data can be returned to the client side only after the main service node reads the data from the local, namely, the data reading operation of the main service node side can be completed only by executing 2 steps, the number of the steps is small, and the time delay is saved.
In one possible design, when the first service request includes a write service, the primary service node is specifically configured to: writing the data to be written in the first service request into the local storage of the master service node, synchronizing the data to be written to the slave service node, after the slave service node writes the data to be written in the local storage of the slave service node, receiving a successful writing response sent by the slave service node, and returning a notification message of successful writing to the client. According to the design, the writing operation is executed through the main service node, only the main service node needs to synchronize data to be written to the slave service node, the version number and the version information do not need to be sequentially synchronized between the main service node and the slave service node, namely the writing operation of the main service node side can be completed only through 4 steps, the number of the required steps is small, and the time delay is saved.
In one possible design, when the second service request includes a read service, the slave serving node is specifically configured to: sending a latest version request message of the data to be read to the main service node, receiving the latest version of the data to be read returned by the main service node, and returning the latest version of the data to be read to the client. Wherein the latest version is obtained by the primary service node from the local storage of the primary service node. According to the design, when the slave service node executes the reading operation, the slave service node can return data to the client only after acquiring the latest data from the master service node, namely, the reading operation of the slave service node side can be completed only by 4 steps, the number of the required steps is small, and the time delay is saved.
In one possible design, when the second service request includes a write service, the slave service node is specifically configured to: and synchronizing the data to be written in the second service request to the main service node, receiving a successful writing response sent by the main service node after the main service node writes the data to be written in the local storage of the main service node, writing the data to be written in the local storage of the slave service node, and returning a notification message of successful writing to the client. According to the design, when the write-in operation is executed through the slave service node, the slave service node only needs to synchronize the data to be written into the master service node, and the version number and the version information between the master service node and the slave service node are not needed to be synchronized in sequence, namely, the write-in operation of the slave service node side can be completed only through 4 steps, the number of the required steps is small, and the time delay is saved.
In one possible design, the data processing system further includes a service-based election main layer, where the service-based election main layer includes a server cluster, the server cluster includes N service nodes, and N is a positive integer greater than or equal to 2. In implementation, N service nodes generate voting information and send out the voting information in sequence, the voting information of each service node comprises an identifier of the service node and a transaction identifier, after each service node receives the voting information sent by other service nodes, the transaction identifier in the voting information of the service node is compared with the transaction identifiers in the voting information of other service nodes, if the transaction identifiers are different, a vote is cast to the service node with a large transaction identifier, if the transaction identifiers are the same, the vote is cast to the service node with a large service identifier of the service node, if the voting number of a certain service node exceeds the preset percentage of the number of servers in a cluster, the service node is used as a main service node, and after the service node which sends the voting information subsequently modifies the voting information of the service node into the voting information of the main service node according to the cast main service node, and sending the voting information to other service nodes, judging whether the service node is a main service node or not after each service node determines to receive the voting information of all the service nodes, if so, updating the node state of the service node to be the main service node, and if not, updating the node state of the service node to be a slave service node or an observation node. According to the design, the main service node is selected in a voting mode, so that the system can be guaranteed to have only one main service node at the same time period, a new main service node can be timely voted and selected when the original main service node goes wrong, and the reliability and the availability of the system are improved.
In a possible design, the voting information of each service node further includes a voting round, and after each service node receives the voting information sent by other service nodes, before comparing the transaction identifier in the voting information of itself with the transaction identifier in the voting information of other service nodes, it may be determined that the voting round in the voting information of the service node is the same as the voting round in the voting information of other service nodes. The design can be used for executing the voting operation in combination with the voting information under the condition of ensuring that the voting information belongs to the same round of voting, and the accuracy of the voting result is ensured.
In one possible design, if the polling turn in the polling information of the service node is greater than the polling turn in the polling information of other service nodes, the polling information of other service nodes is ignored, and after receiving the polling information sent by the next service node, the transaction identifier in the polling information of the service node is compared with the transaction identifier in the polling information of the next service node. On the contrary, if the polling turns in the polling information of the service node are smaller than those of other service nodes, the polling turns in the polling information of the service node are updated, and the polling information received by the service node is emptied. According to the design, under the condition that the voting information does not belong to the same round of voting, the voting rounds in the laggard service nodes can be updated in time, the laggard voting information can be abandoned in time, and the accuracy of the voting rounds in each service node can be ensured while the accuracy of the voting results is ensured.
In one possible design, the service nodes which have sent out the voting information are classified into an active queue in advance, after each service node generates the voting information, before sending out the voting information, the connection state of the service node and the service nodes in the active queue can be judged, and when the service node is in the unconnected state, the connection with the service nodes in the active queue is established, so that the voting information can be sent to the started service nodes smoothly, and the voting can be carried out smoothly.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 schematically illustrates an application scenario of cloud storage provided in an embodiment of the present application;
FIG. 2 is a flow chart illustrating data processing in a master/slave mode provided in the industry;
FIG. 3 illustrates a flow diagram of data processing in a dual active mode provided by the industry;
FIG. 4 is a block diagram illustrating an architecture of a data processing system provided by an embodiment of the present application;
fig. 5 schematically illustrates a flow chart of data processing provided by an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 schematically illustrates an application scenario of cloud storage provided by an embodiment of the present application, as shown in fig. 1, a cloud storage scenario includes a client 100 and a data processing system 200, and the data processing system 200 includes a master service node 210 and a slave service node 220. The client 100 may be connected to the master service node 210 and the slave service node 220, respectively, and the connection manner may be a wired connection or a wireless connection, which is not limited specifically.
Based on the system architecture illustrated in fig. 1, fig. 2 exemplarily shows a flow diagram of data processing in an active/standby mode provided in the industry, where:
fig. 2 (a) shows a schematic diagram of a data writing process in the active/standby mode, and as shown in fig. 2 (a), in the active/standby mode, the whole writing process includes the following four steps: 1, a client 100 sends a write request to a main service node 210, wherein the write request includes service data to be written; 2, the master service node 210 writes the service data to be written into the local storage of the master service node 210, and synchronizes the service data to be written into the slave service node 220; 3, after writing the service data to be written into the local storage of the slave service node 220, the slave service node 220 returns a response message to the master service node 210; and 4, the master service node 210 receives the response message of successful writing from the service node 220, and returns a result response message of successful writing to the client.
Fig. 2 (B) shows a schematic diagram of a data reading flow in the active/standby mode, and as shown in fig. 2 (B), in the active/standby mode, the whole reading process includes the following two steps: 1, a client 100 sends a read request to a main service node 210, wherein the read request includes an identifier of service data to be read; 2, the main service node 210, according to the identifier of the service data to be read, returns the service data to be read to the client after obtaining the service data to be read from the local storage of the main service node 210.
Based on the content illustrated in fig. 2, in the active/standby mode, only the primary service node can interact with the client in the cloud storage scenario at the same time, and the secondary service node can only receive the service data to be written that is synchronized by the primary service node, which obviously makes the data throughput in the data processing system smaller, and cannot fully utilize the computing resources in the data processing system.
Based on this, in order to increase data throughput in a data processing system, another data processing mode, i.e., a dual active mode, is proposed in the industry. Based on the system architecture illustrated in fig. 1, fig. 3 illustrates a flow diagram of data processing in a dual active mode provided in the industry, wherein:
fig. 3 (a) is a schematic diagram illustrating a data writing process in the dual active mode, as shown in fig. 3 (a), in the dual active mode, taking the master service node 210 as an example, when the client 100 writes data to the master service node 210, there may be other clients writing data to the slave service node 220 or reading data, in this case, the whole writing process to the master service node 210 includes the following six steps: 1, a client 100 sends a write request to a main service node 210, wherein the write request includes service data to be written; 2, the main service node 210 writes the service data to be written into the local storage of the main service node 210, and sends a request message requesting a version number (which can be realized by carrying a target address corresponding to the service data to be written) corresponding to the service data to be written to the slave service node 220; 3, the slave service node 220 sends the version number corresponding to the locally stored service data to be written to the master service node 210; after receiving the version number returned by the slave service node 220, the master service node 210 sends the service data to be written and the latest version number to the slave service node 220, where the latest version number is the version number +1 returned by the slave service node 220; 5, the slave service node 220 writes the service data to be written into the local storage of the slave service node 220, updates the version number of the service data to be written, which is stored locally, to the latest version number, and returns a response message to the master service node 210; 6, the master service node 210 receives the response message of successful writing from the service node 220, and returns a result response message of successful writing to the client 100.
Fig. 3 (B) shows a schematic diagram of a data reading flow in the dual active mode, and as shown in fig. 3 (B), when the client 100 reads data from the master service node 210, for example, in the dual active mode, there may be other clients writing data to the slave service node 220, and therefore, the master service node 210 cannot simply return local data and needs to communicate with the slave service node 220 to obtain the latest version of data. In this case, the process of reading data from the master service node 210 includes the following five steps: 1, a client 100 sends a read request to a main service node 210, wherein the read request includes an identifier of service data to be read; 2, the master service node 210 sends a request message for requesting a version number corresponding to the service data to be read to the slave service node 220 according to the identifier of the service data to be read; 3, after receiving the request message of the master service node from the service node 220, returning a response to the master service node 210, and after writing the latest to-be-read service data and the latest version number (i.e. the version number +1 of the local storage) written by other clients into the local storage, sending the latest to-be-read service data and the latest version number to the master service node 210; after receiving the response, the main service node 210 switches to the suspended state, and updates the local service data to be read and the version number by using the latest service data to be read and the latest version number until the latest service data to be read and the latest version number sent by the slave service node 220 are received; 5, the main service node 210 returns the latest service data to be read to the client.
Fig. 3 (C) shows another schematic diagram of a data read/write flow in the dual active mode, as shown in fig. 3 (C), in the dual active mode, taking the main service node 210 as an example, when the client 100 writes data into the main service node 210, there may be another client reading data from the main service node 220, in this case, the whole read/write process to the main service node 210 includes the following six steps: 1, a client 100 sends a write request to a main service node 210, wherein the write request includes service data to be written, and other clients send read requests to the main service node 210, and the read requests include service data identifiers to be read; 2, the master service node 210 sends a request message for requesting a version number corresponding to the service data to be written to the slave service node 220; 3, slave serving node 220 sends a response to master serving node 210; after receiving the response sent by the slave service node 220, the master service node 210 determines that the current writing is the latest writing service, writes the service data to be written into the local storage, and synchronizes the service data to be written and the latest version number (original version number +1) to the slave service node 220; 5, the slave service node 220 writes the service data to be written and the latest version number into the local storage, and returns a response message to the master service node 210; 6, the master service node 210 receives the response message of successful writing from the slave service node 220, returns the result response message of successful writing to the client 100, reads the service data to be read in the local storage, and sends the service data to other clients.
Based on the content illustrated in fig. 3, in the dual active mode, although the master service node and the slave service node in the data processing system can interact with the client at the same time, which is helpful for improving the throughput of the data processing system, the steps of the read-write operation are more, and a great delay loss is additionally caused.
In view of this, embodiments of the present application provide a data processing system, which is used to reduce communication delay while ensuring high system throughput.
Fig. 4 is a schematic diagram illustrating an architecture of a data processing system provided by an embodiment of the present application, and as shown in fig. 4, in this example, the data processing system further includes a first interface 231, a second interface 232, and a traffic routing layer 240 in addition to the master service node 210 and the slave service node 220, where:
one side of the first interface 231 is connected to the client 110, the other side of the first interface 231 is connected to the traffic routing layer 240, and the first interface 231 is configured to receive a first traffic request for the primary service node 210;
one side of the second interface 232 is connected to the client 110, the other side of the second interface 232 is connected to the service routing layer 240, and the second interface 232 is configured to receive a first service request for the slave service node 220;
and the service routing layer 240 is configured to invoke the master service node 210 to process the first service request, invoke the slave service node 220 to process the write service in the second service request, route the read service in the second service request from the slave service node 220 to the master service node 210, and invoke the master service node 210 to process the read service in the second service request.
In the foregoing embodiment, the operation process of the service routing layer is transparent for the client, so from the perspective of the client, the first interface and the second interface concurrently accept the service request, that is, the master service node and the slave service node can receive two concurrent service requests at the same time through the first interface and the second interface, and further concurrently execute the service, and therefore, compared with a mode in which the "master-slave" mode can only execute the service by the master service node at the same time, the system throughput of the scheme is 2 times that of the master-slave mode. Moreover, by uniformly placing the write service at the side of the master service node for execution, the latest version of data in the system can be ensured to be directly written by the master service node (namely the master service node writes the version information to the slave service node synchronously), and compared with a mode that the master service node and the slave service node in a double-active mode can complete data synchronization only by synchronizing the version number and the version information, the method can save the number of steps passed by an intermediate flow and achieve the purpose of saving time delay. Therefore, the design can achieve the double effects of improving the throughput of the system and reducing the time delay.
The following describes in detail the read-write flow in the embodiment of the present application from the perspective that the first service request includes a read service or a write service, and the second service request includes a read service or a write service, respectively. Fig. 5 is a schematic flow chart illustrating data processing provided by an embodiment of the present application, where:
fig. 5 (a) shows a schematic diagram of a data writing process provided by an embodiment of the present application, as shown in fig. 5 (a):
the whole writing process of the client 100 writing data to the main service node 210 includes the following four steps: 1.1, a client 100 sends a first service request to a main service node 210, wherein the first service request comprises a write service type and service data to be written; 1.2, the master service node 210 writes the service data to be written into the local storage of the master service node 210, and synchronizes the service data to be written into the slave service node 220; 1.3, after writing the service data to be written into the local storage, the slave service node 220 sends a response message of successful writing to the master service node 210; 1.4, after receiving the response message of successful writing returned from the service node 220, the main service node 210 returns a result response message of successful writing to the client 100.
The whole writing process of the client 100 to write data from the service node 220 includes the following four steps: 2.1, the client 100 sends a second service request to the slave service node 220, wherein the second service request comprises a write service type and service data to be written; 2.2, under the routing function of the service routing layer 240, the slave service node 220 synchronizes the service data to be written to the master service node 210; 2.3, the master service node 210 writes the service data to be written into the local storage, and sends a response message of successful writing to the slave service node 220; 2.4, after receiving the response message of successful writing returned by the master service node 210, the slave service node 220 writes the service data to be written into the local storage, and returns a result response message of successful writing to the client 100.
As can be seen from the above, with the data processing system in the embodiment of the present application, no matter data is written to the master service node or the slave service node, the entire data writing process includes 4 steps, and compared with the 6 steps in the writing process in the dual active mode illustrated in fig. 3 (a) or fig. 3 (C), the complexity of the writing process can be reduced by saving the number of steps, and thus the communication delay required by the writing process can be reduced. In addition, although the number of 4 steps in the data writing flow is the same as that of 4 steps in the active/standby mode illustrated in fig. 2 (a), the data writing flow supports processing of requests of two clients in the same period, and thus the throughput is 2 times that of the active/standby mode. It is thus understood that the data writing flow can save communication delay while maintaining high throughput.
Further, fig. 5 (B) shows a schematic diagram of a data reading flow provided by an embodiment of the present application, as shown in fig. 5 (B):
considering that all data write operations in the data processing system are performed by the primary service node 210, that is, the data in the local storage of the primary service node 210 is necessarily the latest data, the data is read from the primary service node 210 by only directly reading the local storage and returning. Specifically, the whole reading process of the client 100 reading data from the main service node 210 includes the following two steps: 1.1, a client 100 sends a first service request to a main service node 210, wherein the first service request comprises address information of service data to be read; 1.2, the main service node 210 queries the service data to be read from the local storage according to the address information of the service data to be read, and returns the service data to the client 100.
Conversely, since all data write operations in the data processing system are performed by the master service node 210, the data in the local storage of the master service node 210 is the most recent data, while the data in the local storage of the slave service node 220 may not be the most recent data, because the slave service node 210 may be performing a write operation in the master service node 210 while performing a read data operation, or may have been successfully written but not yet synchronized with the most recent data to the slave service node 220, upon which the data is read from the master service node 210, it is also necessary to obtain the most recent version of the data from the master service node 210 side. Specifically, the whole reading process of the client 100 from the service node 220 includes the following four steps: 2.1, the client 100 sends a second service request to the slave service node 220, wherein the second service request includes address information of service data to be read; 2.2, the slave service node 220 sends the address information of the service data to be read to the master service node 210, so as to request the latest version of the service data to be read from the master service node 210; 2.3, the master service node 210, according to the address information of the service data to be read, returns the service data to be read to the slave service node 220 through a response after inquiring the service data to be read from the local storage; 2.4, the service node 220 returns the service data to be read to the client 100.
As can be seen from the above, with the data processing system in the embodiment of the present application, the data reading from the master service node includes 2 steps, and the data reading from the slave service node includes 4 steps, which can reduce the complexity of the reading process by saving the number of steps, and further reduce the communication delay required by the reading process, compared with 5 steps in the reading process in the dual active mode illustrated in fig. 3 (B). In addition, although the number of 2 steps in the read flow is the same for the active/standby mode illustrated in fig. 2 (B), the data read flow supports processing requests of two clients in the same time period, and thus the throughput is 2 times that of the active/standby mode. It can be seen that the data reading flow can also save communication delay while maintaining high throughput.
To more clearly illustrate the performance differences between the data processing system in the embodiment of the present application and the active/standby mode commonly used in the industry, table 1 exemplarily shows a schematic table of performance comparison:
TABLE 1
As shown in table 1, the data processing scheme in the embodiment of the present application can have higher system throughput than the active/standby mode, and has smaller time delay than the dual active mode, so that the data processing scheme in the embodiment of the present application can have the dual effects of improving the system throughput and reducing the time delay.
It should be noted that, the implementation of the above data processing operation needs to ensure that the entire data processing system has only one main service node at the same time, and for this reason, the embodiment of the present application further provides a scheme for deciding the main service node at each time in an election manner, which is described in detail below.
In an optional implementation manner, as shown with continued reference to fig. 4, the data processing system 200 may further include a service-based election layer 250, where the service-based election layer 250 includes a server cluster, and the server cluster includes N service nodes, where N is a positive integer greater than or equal to 2. The service election-based layer 250 may select a main service node from the N service nodes in a periodic manner, and at the beginning of each period, the N service nodes in the server cluster generate and send out voting information in sequence, where the voting information of each service node includes an identifier of the service node and a transaction identifier. The identifier of the service node may exemplarily refer to a number of the service node, for example, the identifiers of N service nodes are 1, 2, … …, and N, respectively, the transaction identifier may refer to a number of a transaction that can be processed by the service node, and the number may be preset in the service node by a developer. Furthermore, after receiving the voting information sent by other service nodes, each service node compares the transaction identifier in the voting information with the transaction identifiers in the voting information of other service nodes, and if the transaction identifiers are different, throws the vote to the service node with the larger transaction identifier, and if the transaction identifiers are the same, throws the vote to the service node with the larger service node identifier. Then, each service node determines whether the number of votes of a certain service node in the voted information exceeds a preset percentage of the number of servers in the cluster (where the preset percentage may be set by a person skilled in the art according to experience, and may be exemplarily set to be a proportion greater than or equal to 50%), if so, the service node is taken as a main service node, and if not, the voting information of other service nodes is continuously received and the above operation is performed until the main service node is determined. Therefore, the service nodes which subsequently send the voting information modify the voting information of the service nodes into the voting information of the main service node according to the cast main service node, and send the voting information to other service nodes until all the service nodes complete voting, each service node judges whether the service node is the main service node, if so, the node state of the service node is updated to the main service node, and if not, the node state of the service node is updated to the slave service node or the observation node.
For example, in the above embodiment, each service node may also have its own state, and the state of the service node may be one of the following four states: LOOKIN, LEADING, FOLLOWING, OBSERVING. The LOOKIN state means that a main service node is not decided currently, the leader state means that the service node is a posted main service node, the FOLLOWING state means that the service node is a slave service node, the OBSERVING state means that the service node is an observation node, and the observation node is a node specially used for voting. In a specific implementation, after the service node generates the voting information, the state of the next service node may be determined, and if the state is LOOKIN, which means that the main service node is not decided currently, the service node may send out the voting information, and perform the subsequent steps according to the above scheme. Further, in the process of executing voting, if the service node determines that the service node is the main service node that is cast, the state of the service node is updated to leader, the states of other service nodes that have voted for are updated to OBSERVING, and the states of other service nodes that have not voted for are updated to FOLLOWING.
For example, after each service node sends out the voting information, the service node may be classified into an active queue, before any other service node sends out the voting information, the connection state between itself and each service node in the active queue may also be determined, if a connection is established, the voting information of itself is directly sent to each service node in the active queue, if a connection is not established, a connection is established immediately, and after the connection is successful, the voting information of itself is sent. Therefore, the voting information sent by the service node can be successfully received by the started service node, and the voting is further ensured to be smoothly carried out.
Further exemplarily, the voting information of each service node may further include a voting round, which is incremented before each vote, for example, the voting round is 1 at the time of the first vote, and is incremented to 2 before the second vote, … …. In this way, after each service node receives the voting information sent by other service nodes, it may also compare whether the voting round in the voting information of the service node is the same as the voting round in the voting information of other service nodes. If the voting information is the same as the voting information, the service node and other service nodes belong to the same round of voting, so that the service node can compare the transaction identifier in the voting information of the service node with the transaction identifiers in the voting information of other service nodes and execute the operation according to the comparison result, so that the voting operation is executed by combining the voting information under the condition that the voting information belongs to the same round of voting, and the accuracy of the voting result is ensured. Otherwise, if the two service nodes are different, it is indicated that the service node and the other service nodes do not belong to the same round of voting, and the service node needs to perform a re-determination of whether the polling round in the polling information of the next service node is greater than the polling round in the polling information of the other service nodes. If yes, the service node belongs to the latest voting round, and other service nodes belong to the old voting round, so that the service node can directly ignore the voting information of other service nodes, and after receiving the voting information sent by the next service node, the transaction identifier in the voting information of the service node is compared with the transaction identifier in the voting information of the next service node. If not, the service node belongs to the old round of voting, and other service nodes belong to the new round of voting. Therefore, under the condition that the voting information does not belong to the same round of voting, the voting rounds in the laggard service nodes can be updated in time, the laggard voting information can be abandoned in time, and the accuracy of the voting rounds in each service node can be ensured while the accuracy of the voting results is ensured.
To facilitate understanding of the voting scheme described above, a specific example is described below by way of example.
In this example, assume that there are 5 service nodes in the server cluster in total: service node 1, service node 2, service node 3, service node 4 and service node 5, these 5 service node's sign is 1, 2,3, 4, 5 in proper order, and the transaction sign is 3,5, 1, 2 in proper order, and the preset percentage is 50%, then:
step one, 5 service nodes generate their own voting information, and the voting information includes the service node identifier and the transaction identifier, that is: the voting information generated by the service node 1 is (1,3), the voting information generated by the service node 2 is (2,3), the voting information generated by the service node 3 is (3,5), the voting information generated by the service node 4 is (4,1), and the voting information generated by the service node 5 is (5, 2);
secondly, the service node 1 sends out own voting information (1,3) firstly, at this time, because no other service node sends out the voting information, the voting information (1,3) sent out by the service node 1 has no response, the service node 1 throws the vote to itself, and the state is updated to LOOKING;
thirdly, the service node 2 sends out own voting information (2,3), at this time, after the service node 1 receives the voting information (2,3) of the service node 2, the service node votes again, because the transaction identifier 3 in the voting information (2,3) of the service node 2 is the same as the transaction identifier 3 in the voting information (1,3) of the service node 1, and the service node identifier 2 in the voting information (2,3) of the service node 2 is larger than the service node identifier 1 in the voting information (1,3) of the service node 1, the service node 1 votes to the service node 2; similarly, the service node 2 also puts the ticket into the service node 2, and at this time, the service node 2 obtains 2 tickets;
fourthly, although the service node 2 obtains 2 tickets, 5 service nodes exist in the whole server cluster, 50% of preset percentage corresponds to 2.5 service nodes, and the number of the service nodes 2 is less than 2.5, so that a main service node cannot be decided at present, and the states of the service node 1 and the service node 2 are still LOOKING;
fifthly, the service node 3 sends out own voting information (3,5), at this time, after the service node 1 and the service node 2 receive the voting information (3,5) of the service node 3, the service node 1 and the service node 2 vote again, and as the transaction identifier 5 in the voting information (2,5) of the service node 3 is larger than the transaction identifier 3 in the voting information (1,3) of the service node 1 and the voting information (2,3) of the service node 2, the service node 1 and the service node 2 both cast a vote to the service node 3; similarly, the service node 3 also puts the ticket into the service node 3, and at this time, the service node 3 obtains 3 tickets;
sixthly, 5 service nodes exist in the whole server cluster, 50% of preset percentage corresponds to 2.5 service nodes, and the number of votes 3 obtained by the service node 3 is greater than 2.5, so that a main service node can be decided at present, the main service node is the service node 3, the service node 3 updates the state of the main service node to LEADING, and the voted service nodes 1 and 2 update the state of the main service node to OBSERVING;
seventhly, the service node 4 and the service node 5 send out own voting information (4,1) and (5,2) in sequence, but since the main service node 3 is decided at present, the service node 4 and the service node 5 can only be used as followers, namely, tickets are cast to the decided main service node 3, and the state of the service node 4 and the service node 5 is updated to the vote.
In the foregoing embodiment of the present application, a data processing system includes a first interface, a second interface, a service routing layer, a master service node, and a slave service node, where the first interface is configured to receive a first service request for the master service node, the second interface is configured to receive a second service request for the slave service node, the service routing layer is configured to route the first service request to the master service node, route a read service in the second service request to the slave service node, and route a write service in the second service request to the master service node, the master service node is configured to process the write service in the first service request and the second service request, and the slave service node is configured to process the read service in the second service request. Thus, since the operation process of the service routing layer is transparent for the client, from the perspective of the client, the first interface and the second interface concurrently accept the service request, that is, the master service node and the slave service node can receive two concurrent service requests at the same time through the first interface and the second interface, and further concurrently execute the service, and therefore, compared with a mode in which the "master-slave" mode can only execute the service by the master service node at the same time, the system throughput of the scheme is 2 times that of the master-slave mode. Moreover, by uniformly placing the write service at the side of the master service node for execution, the latest version of data in the system can be ensured to be directly written by the master service node (namely the master service node writes the version information to the slave service node synchronously), and compared with a mode that the master service node and the slave service node in a double-active mode can complete data synchronization only by synchronizing the version number and the version information, the method can save the number of steps passed by an intermediate flow and achieve the purpose of saving time delay. Therefore, the design can achieve the double effects of improving the system throughput and reducing the time delay.
It should be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A data processing system is characterized by comprising a first interface, a second interface, a service routing layer, a master service node and a slave service node;
the first interface is used for receiving a first service request aiming at the main service node;
the second interface is used for receiving a second service request aiming at the slave service node;
the service routing layer is configured to route the first service request to the master service node, route a read service in the second service request to the slave service node, and route a write service in the second service request to the master service node;
the main service node is configured to process a write service request in the first service request and the second service request;
and the slave service node is used for processing the read service request in the second service request.
2. The system of claim 1, wherein the first service request includes a read service;
the primary service node is specifically configured to:
and acquiring data to be read from the local storage of the main service node, and returning the data to be read to the client.
3. The system of claim 1, wherein the first service request includes a write service;
the primary service node is specifically configured to:
writing the data to be written in the first service request into a local storage of the master service node, and synchronizing the data to be written to the slave service node;
after the slave service node writes the data to be written into the local storage of the slave service node, receiving a successful writing response sent by the slave service node;
and returning a notification message of successful writing to the client.
4. The system of claim 1, wherein the second service request includes a read service;
the slave serving node is specifically configured to:
sending a latest version request message of data to be read to the main service node;
receiving the latest version of the data to be read, which is returned by the main service node, wherein the latest version is obtained by the main service node from the local storage of the main service node;
and returning the latest version of the data to be read to the client.
5. The system of claim 1, wherein the second service request includes a write service;
the slave serving node is specifically configured to:
synchronizing the data to be written in the second service request to the main service node;
after the main service node writes the data to be written into the local storage of the main service node, receiving a successful writing response sent by the main service node;
and writing the data to be written into the local storage of the slave service node, and returning a notification message of successful writing to the client.
6. The system of claim 1, further comprising a service-based election master layer, the service-based election master layer comprising a server cluster, the server cluster comprising N service nodes, N being a positive integer greater than or equal to 2;
the N service nodes generate voting information and send out the voting information in sequence, and the voting information of each service node comprises the service node identification and the transaction identification;
after receiving voting information sent by other service nodes, each service node compares a transaction identifier in the voting information with transaction identifiers in the voting information of the other service nodes, if the transaction identifiers are different, a ticket is cast to the service node with a large transaction identifier, if the transaction identifiers are the same, the ticket is cast to the service node with the large service identifier of the service node, and if the voting number of a certain service node exceeds the preset percentage of the number of servers in the cluster, the service node is used as a main service node;
the service node which subsequently sends the voting information modifies the voting information of the service node into the voting information of the main service node according to the cast main service node, and then sends the voting information to other service nodes;
and after determining that the voting information of all the service nodes is received, each service node judges whether the service node is the main service node, if so, the node state of the service node is updated to the main service node, and if not, the node state of the service node is updated to the slave service node or the observation node.
7. The system of claim 6, wherein the voting information of each service node further comprises a voting turn;
after each service node receives the voting information sent by other service nodes, before comparing the transaction identifier in the voting information with the transaction identifier in the voting information of other service nodes, the service nodes are further configured to:
and determining that the polling turns in the polling information of the service node are the same as the polling turns in the polling information of the other service nodes.
8. The system of claim 7, wherein each service node is further configured to:
and if the voting turns in the voting information of the service nodes are determined to be larger than the voting turns in the voting information of other service nodes, ignoring the voting information of other service nodes, and comparing the transaction identifier in the voting information with the transaction identifier in the voting information of the next service node after receiving the voting information sent by the next service node.
9. The system of claim 7 or 8, wherein each service node is further configured to:
and if the voting turns in the voting information of the service node are smaller than the voting turns in the voting information of other service nodes, updating the voting turns in the voting information of the service node, and emptying the voting information received by the service node.
10. The system according to claim 7 or 8, characterized in that the service node which has sent out the voting information is classified into an active queue in advance;
after each service node generates voting information and before the voting information is sent out, the method is further configured to:
and judging the connection state of the service node and the service node in the active queue, and establishing the connection with the service node in the active queue when the service node is in the unconnected state.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111583497.0A CN114448781B (en) | 2021-12-22 | 2021-12-22 | Data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111583497.0A CN114448781B (en) | 2021-12-22 | 2021-12-22 | Data processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114448781A true CN114448781A (en) | 2022-05-06 |
| CN114448781B CN114448781B (en) | 2024-06-07 |
Family
ID=81364647
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111583497.0A Active CN114448781B (en) | 2021-12-22 | 2021-12-22 | Data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114448781B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014177085A1 (en) * | 2013-09-17 | 2014-11-06 | 中兴通讯股份有限公司 | Distributed multicopy data storage method and device |
| CN108234630A (en) * | 2017-12-29 | 2018-06-29 | 北京奇虎科技有限公司 | Method for reading data and device based on distributed consensus protocol realization |
| CN108234641A (en) * | 2017-12-29 | 2018-06-29 | 北京奇虎科技有限公司 | Data read-write method and device based on distributed consensus protocol realization |
| CN109587185A (en) * | 2017-09-28 | 2019-04-05 | 华为技术有限公司 | Object processing method in cloud storage system and cloud storage system |
| CN110597673A (en) * | 2019-09-25 | 2019-12-20 | 腾讯科技（深圳）有限公司 | Disaster recovery method, device and equipment of storage system and computer readable storage medium |
| CN111708787A (en) * | 2020-05-07 | 2020-09-25 | 中国人民财产保险股份有限公司 | Multi-center service data management system |
| CN112099728A (en) * | 2019-06-18 | 2020-12-18 | 华为技术有限公司 | Method and device for executing write operation and read operation |
| WO2021051492A1 (en) * | 2019-09-18 | 2021-03-25 | 招商银行股份有限公司 | Database service node switching method, apparatus and device, and computer storage medium |
-
2021
- 2021-12-22 CN CN202111583497.0A patent/CN114448781B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014177085A1 (en) * | 2013-09-17 | 2014-11-06 | 中兴通讯股份有限公司 | Distributed multicopy data storage method and device |
| CN109587185A (en) * | 2017-09-28 | 2019-04-05 | 华为技术有限公司 | Object processing method in cloud storage system and cloud storage system |
| CN108234630A (en) * | 2017-12-29 | 2018-06-29 | 北京奇虎科技有限公司 | Method for reading data and device based on distributed consensus protocol realization |
| CN108234641A (en) * | 2017-12-29 | 2018-06-29 | 北京奇虎科技有限公司 | Data read-write method and device based on distributed consensus protocol realization |
| CN112099728A (en) * | 2019-06-18 | 2020-12-18 | 华为技术有限公司 | Method and device for executing write operation and read operation |
| WO2021051492A1 (en) * | 2019-09-18 | 2021-03-25 | 招商银行股份有限公司 | Database service node switching method, apparatus and device, and computer storage medium |
| CN110597673A (en) * | 2019-09-25 | 2019-12-20 | 腾讯科技（深圳）有限公司 | Disaster recovery method, device and equipment of storage system and computer readable storage medium |
| CN111708787A (en) * | 2020-05-07 | 2020-09-25 | 中国人民财产保险股份有限公司 | Multi-center service data management system |
Non-Patent Citations (3)
| Title |
|---|
| R SHRESTHA: ""High Availability and Performance of Database in the Cloud-Traditional Master-slave Replication versus Modern Cluster-based Solutions"", 《OSLOMET》, 31 December 2017 (2017-12-31) * |
| TG PAPAIOANNOU: ""Scalia: An adaptive scheme for efficient multi-cloud storage"", 《IEEE》, 31 December 2012 (2012-12-31) * |
| 鲍禹含: ""多云存储关键技术研究进展"", 《计算机工程》, 31 October 2020 (2020-10-31) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114448781B (en) | 2024-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210406279A1 (en) | System and method for maintaining a master replica for reads and writes in a data store | |
| CN108280080B (en) | Data synchronization method and device and electronic equipment | |
| US9116862B1 (en) | System and method for data replication using a single master failover protocol | |
| US9489434B1 (en) | System and method for replication log branching avoidance using post-failover rejoin | |
| CN105512266A (en) | Method and device for achieving operational consistency of distributed database | |
| CN111338766A (en) | Transaction processing method and device, computer equipment and storage medium | |
| WO2018045758A1 (en) | Data storage method and data centre | |
| CN111143382B (en) | Data processing method, system and computer readable storage medium | |
| CN107919977B (en) | Online capacity expansion and online capacity reduction method and device based on Paxos protocol | |
| CN112162846B (en) | Transaction processing method, device and computer readable storage medium | |
| CN112148798A (en) | Data processing method and device applied to distributed system | |
| CN113010549A (en) | Data processing method based on remote multi-active system, related equipment and storage medium | |
| WO2016082594A1 (en) | Data update processing method and apparatus | |
| CN115098229A (en) | Transaction processing method, device, node equipment and storage medium | |
| CN115794499B (en) | Method and system for dual-activity replication data among distributed block storage clusters | |
| CN107025257B (en) | Transaction processing method and device | |
| CN113297159A (en) | Data storage method and device | |
| CN112052104A (en) | Message queue management method based on multi-computer-room realization and electronic equipment | |
| CN108924240B (en) | Distributed processing method, device and storage medium based on consistency protocol | |
| CN114448781B (en) | Data processing system | |
| CN115658245A (en) | Transaction submitting system, method and device based on distributed database system | |
| CN109376141A (en) | A kind of data migration method and device | |
| CN114217986A (en) | Data processing method, device, equipment, storage medium and product | |
| CN113064768A (en) | Method and device for switching fragment nodes in block chain system | |
| CN113438281B (en) | Storage method, device, equipment and readable medium of distributed message queue |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |