CN111258822A - Data processing method, server and computer readable storage medium - Google Patents
Data processing method, server and computer readable storage medium Download PDFInfo
- Publication number
- CN111258822A CN111258822A CN202010043982.8A CN202010043982A CN111258822A CN 111258822 A CN111258822 A CN 111258822A CN 202010043982 A CN202010043982 A CN 202010043982A CN 111258822 A CN111258822 A CN 111258822A
- Authority
- CN
- China
- Prior art keywords
- server
- cluster
- request
- client
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 claims abstract description 44
- 230000004044 response Effects 0.000 claims abstract description 25
- 238000004590 computer program Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 19
- 230000001360 synchronised effect Effects 0.000 abstract description 11
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000003993 interaction Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000011664 signaling Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000009469 supplementation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/1675—Temporal synchronisation or re-synchronisation of redundant processing components
- G06F11/1683—Temporal synchronisation or re-synchronisation of redundant processing components at instruction level
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method, a server and a computer readable storage medium, and relates to the field of cluster data processing. The method is applied to a first server with a cache, wherein the first server is a main server of a cluster, the cluster also comprises other N slave servers, and the cluster is connected with a client through an intermediate component; the method comprises the following steps: receiving a client request forwarded by the intermediate component; storing the request to a first cache of a first server; sending the request stored in the first cache to the other N slave servers; and when at least N/2 storage responses corresponding to the requests are received, sending data synchronization success information to the client so that the client confirms that the data synchronization of the current round is successful. The client communicates with the cluster through the intermediate component, so that the selection logic of the cluster by the client is reduced; when the data are synchronized, the slave server interacts with the cache of the main server instead of the disk of the main server, so that the data processing time is effectively reduced, and the data accuracy is ensured.
Description
Technical Field
The present invention relates to the field of data processing of clusters, and in particular, to a data processing method, a server, and a computer-readable storage medium.
Background
In the current master-slave high availability scheme, two modes of processing data, namely asynchronous and synchronous, exist: if the asynchronous processing mode is adopted, the data loss of the cluster can be directly caused by the downtime of the main server; however, assuming that the cluster is switched to a synchronous processing mode, except for causing the performance problem of the cluster, the steps of master-slave switching detection are complicated, the consistency of the cluster data cannot be ensured, dirty data is generated, and the cluster data can be recovered only by performing data repair subsequently; in addition, the master-slave switching has a large delay, in order to reduce the delay, a third-party service needs to be relied on, and if a node of the third-party service fails or a network partition occurs, a master-slave switching request of a cluster fails, so that the cluster is unavailable, and the high availability of the cluster cannot be guaranteed.
In order to ensure high availability of the cluster, a data peer-to-peer high availability scheme is generally used, and the data peer-to-peer high availability scheme is respectively connected with a client, a database and a slave server through a master server; the scheme can be applied to scenes with low requirements on data accuracy, such as barrage service, and users can see the barrages sent by the users; for the bullet screen sent by other people, the bullet screen can be stored in the disk of a different server, and even if the bullet screen of other users stored in the disk is lost, the user does not sense the bullet screen.
Disclosure of Invention
In view of the above, the present invention provides a data processing method, a server and a computer readable storage medium.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
in a first aspect, the invention provides a data processing method, which is applied to a first server with a cache, wherein the first server is a main server of a cluster, the cluster further comprises other N slave servers, N is more than or equal to 2, and the cluster is connected with a client through an intermediate component. The method comprises the following steps: receiving the request of the client forwarded by the intermediate component; storing the request to a first cache of the first server; sending the request stored in the first cache to the other N slave servers; and when at least N/2 storage responses corresponding to the requests and sent by the other N secondary servers are received, sending data synchronization success information to the client, so that the client confirms that the data synchronization of the current round is successful.
In an alternative embodiment, the first server has a voting timer for voting a master server of the cluster, and before the receiving of the request of the client forwarded by the intermediate component, the method further comprises: judging whether the waiting time of the voting timer is greater than or equal to a preset time or not; the waiting time is a difference value between the current time of the first server and the starting time of the voting timer. If yes, sending request voting information to the other N slave servers, so that the other N slave servers do not send voting requests within the election timeout time; the election timeout time is the interval time between the time when any one of the other N slave servers sends the voting request and the current time. Judging whether at least N/2 responses corresponding to the voting request information sent by the other N secondary servers are received within the voting timeout time; and if so, confirming that the first server is the main server of the cluster.
In an optional embodiment, the first server stores a first log having a current sequence number, and before the confirming that the first server is the master server of the cluster, the method further comprises: judging whether the current serial number is larger than or equal to all log serial numbers of other N slave servers; if yes, the step of confirming that the first server is the main server of the cluster is executed.
In an alternative embodiment, the first server having a lock-free queue, the storing the request to a first cache of the first server, comprising: sending the request to the lock-free queue; sending a cache request to the lock-free queue to store the request to the first cache.
In an optional embodiment, the first server includes a disk, and the storing the request to the first cache of the first server further includes: acquiring a snapshot file according to the request; the snapshot file comprises a current time point, and all the requests of the first cache are contained; and storing the snapshot file to the disk.
In an optional embodiment, the sending the request stored in the first cache to the other N secondary servers includes: judging whether the log identification is matched with log sequence information in a second server or not; the second server is any one or more servers in the other N slave servers; and if so, sending the request to the second server.
In a second aspect, the present invention provides a data processing method, applied to a second server, where the second server is a slave server of a cluster, the cluster further includes a master server with a cache, and the cluster is connected to a client through an intermediate component, and the method includes: receiving a request sent by the main server; the request is sent by the client and forwarded by the intermediate component, and is stored in a first cache of the main server. And when the log identification corresponding to the request is matched with the log serial number of the second server, storing the request. And sending a response corresponding to the request to the main server so that the main server sends data synchronization success information to the client, and the client confirms that the data synchronization of the current round is successful.
In a third aspect, the present invention provides a server, where the server is a master server of a cluster, the cluster further includes N other slave servers, N is greater than or equal to 2, the cluster is connected to a client through an intermediate component, the server includes a processor and a memory, the memory stores machine executable instructions capable of being executed by the processor, and the processor can execute the machine executable instructions to implement the method according to any one of the foregoing embodiments.
In a fourth aspect, the present invention provides a server, where the server is a slave server of a cluster, the cluster further includes a master server with a cache, and the cluster is connected to a client through an intermediate component. The server comprises a processor and a memory, the memory storing machine executable instructions executable by the processor to implement the method of the foregoing embodiments.
In a fifth aspect, the invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of the preceding embodiments.
The invention provides a data processing method, a server and a computer readable storage medium, and relates to the field of cluster data processing. The data processing method is applied to a first server with a cache, the first server is a main server of a cluster, the cluster further comprises other N slave servers, N is more than or equal to 2, the cluster is connected with a client through an intermediate component, and the method comprises the following steps: receiving the request of the client forwarded by the intermediate component; storing the request to a first cache of the first server; sending the request stored in the first cache to the other N slave servers; and when at least N/2 storage responses corresponding to the requests and sent by the other N secondary servers are received, sending data synchronization success information to the client, so that the client confirms that the data synchronization of the current round is successful. The client communicates with the cluster through the intermediate component, so that the selection logic of the client to the cluster is reduced; when the data are synchronized, the slave server interacts with the cache of the main server instead of the disk of the main server, so that the data processing time can be effectively reduced, and the accuracy of the data is ensured.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a schematic diagram illustrating interaction between a client and a cluster according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 3 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 4 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 5 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 6 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 7 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 8 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
fig. 9 is a signaling interaction diagram of a data processing method according to an embodiment of the present invention.
Icon: 11-first server, 12-slave server, 12 a-second server, 12 b-third server, 13-intermediary component, 14-client.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
It is noted that relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
In order to ensure high availability of the cluster, a data peer-to-peer high availability scheme is generally used, and the data peer-to-peer high availability scheme is respectively connected with a client, a database and a slave server through a master server; the scheme can be applied to scenes with low requirements on data accuracy, such as barrage service, and users can see the barrages sent by the users; for the bullet screen sent by other people, the bullet screen can be stored in the disk of a different server, and even if the bullet screen of other users stored in the disk is lost, the user does not sense the bullet screen. How to quickly process a request sent by a client and synchronize the request into a cluster is a problem to be solved urgently at present.
To solve the above problems and the disadvantages in the background art, an embodiment of the present invention provides a data processing method, which is applied to a server. Referring to fig. 1, fig. 1 is a schematic view illustrating interaction between a client and a cluster according to an embodiment of the present invention. The client 14 is connected to a cluster through the intermediate component 13, and the cluster includes a first server 11 with a cache and other N slave servers 12, where N is a natural number greater than or equal to two.
For the above-mentioned intermediate component 13, it may be a software process, for example, the intermediate component 13 is deployed on each server in the cluster, and the client can communicate with any server in the cluster through the intermediate component 13.
For the above client, it may be, but is not limited to, a mobile phone, a personal computer, an intelligent bracelet, an intelligent household appliance, a service server, etc.
It should be understood that any of the servers described above may include, but are not limited to, a memory, a processor, and a communication interface. The memory, processor and communication interface are electrically connected to each other, directly or indirectly, to enable transfer or interaction of data. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The memory may be used to store software programs and modules, such as program instructions/modules corresponding to the data processing method provided in the embodiments of the present invention, and the processor executes various functional applications and data processing by executing the software programs and modules stored in the memory. The communication interface may be used for communicating signaling or data with other node devices. The server may have a plurality of communication interfaces in the present invention.
The Memory may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
The processor may be an integrated circuit chip having signal processing capabilities. The Processor may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), etc.; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc.
Referring to fig. 2, fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present invention, based on the first server 11 shown in fig. 1. The data processing method comprises the following steps:
and S36, receiving the request of the client forwarded by the intermediate component.
For example, when a client is started, the client can directly communicate with the intermediate component, the intermediate component randomly selects a server in a cluster for communication, if the server selected for the first time is not the main server (returns an error), other servers of the cluster can be traversed until the main server is found, and then the server information is recorded into a cache of the client; it should be appreciated that in the above case, the client is unaware and does not need the selection logic of the master server for the cluster, but only communicates directly with the intermediate component. If the primary server in the cluster has crashed, the request of the intermediate component will time out, after which the process of randomly picking the server will be retried again.
And S37, storing the request to the first cache of the first server.
It should be understood that the request is stored in the cache of the first server, and when data synchronization is performed subsequently, the request may not interact with a disk of the server, but directly acquire data from the first cache, thereby effectively improving the data processing efficiency of the cluster.
And S38, sending the request stored in the first cache to other N slave servers.
It should be appreciated that the single point of storage of the primary server must not serve the purpose of high availability of the cluster, and in order to prevent the primary server from going down causing a loss of data or reduced accuracy, it is necessary to synchronize this request to the other servers of the cluster.
And S39, when at least N/2 storage responses corresponding to the request sent by other N slave servers are received, sending data synchronization success information to the client, so that the client confirms that the data synchronization of the current round is successful.
As can be appreciated, the client communicates with the cluster through the intermediate component, which reduces the selection logic of the client for the cluster; when the data are synchronized, the slave server interacts with the cache of the main server instead of the disk of the main server, so that the data processing time can be effectively reduced, and the accuracy of the data is ensured.
In an optional embodiment, when a service cluster is online, in order to obtain a main server of the cluster, a current technical solution depends on a third-party service, and in order to reduce the dependency on the third-party service, on the basis of fig. 2, taking the first server having a voting timer as an example, please refer to fig. 3, and fig. 3 is a schematic flow chart of another data processing method provided in the embodiment of the present invention. It should be understood that the voting timer is used for voting the main server of the cluster, the voting timer may be a timer thread started by the first server, or may be a timing software provided on the first server, and before the aforementioned S36, the data processing method may further include:
and S31, judging whether the waiting time of the voting timer is larger than or equal to the preset time.
The waiting time is the difference between the current time of the first server and the start time of the voting timer. For example, the preset time may be a random number of one millisecond, such as 1 millisecond, 3 milliseconds, etc.
If the waiting time of the voting timer is greater than or equal to the preset time, executing S32; if the waiting time of the voting timer is less than the preset time, the process proceeds to step S32 when the waiting time is greater than or equal to the preset time.
And S32, sending the request voting information to other N slave servers, so that the other N slave servers do not send voting requests within the election timeout time.
The election timeout is the interval between the time when any one of the other N slave servers sends the voting request and the current time. For example, the election timeout time may be greater than the broadcast time of the first server broadcasting the request voting information to the other N slave servers, in which case the first server may send the request voting information to the other N slave servers before the other N slave servers timeout, so as to avoid election conflicts when clustering the election master servers.
S33, judging whether receiving other N responses corresponding to the request voting information sent by the server at least N/2 within the voting overtime.
It is understood that, when receiving the request voting information sent by the first server from the server, a response corresponding to the request voting information may be generated, so that the first server confirms that the transmission of the request voting information is successful.
If at least N/2 responses corresponding to the request voting information sent by other N slave servers are not received within the voting timeout time, returning to execute S31; if at least N/2 responses corresponding to the requested voting information are received from the other N servers within the voting timeout period, S35 is executed.
And S35, confirming that the first server is the main server of the cluster.
It will be appreciated that by setting the voting timer, the main server of the cluster can be elected without the use of third party services. For example, the cluster may wait by starting a timer thread (voting timer), and the number of milliseconds to wait is a random number (preset time); when the voting waiting time of the first server reaches the preset time, the first server sends a voting request message to other servers of the cluster; the waiting time of the voting timer of each server is inconsistent, so that the servers can be dispersed, and in most cases, the waiting time of the voting timer of only one server (such as the first server) reaches the preset time, and then election is won to obtain the right of the main server; then before other servers overtime, a heartbeat packet (request voting information) is sent to other servers, so that the condition that a plurality of servers send the request voting information at the same time to cause voting conflict of main servers of the cluster is prevented, and the main servers are selected quickly; it should be understood that the airtime between servers < < election timeout time < < mean time between failures, which is the average time between two failures for a server, is an order of magnitude less than the mean time between failures.
It should be understood that in a cluster, a master server needs to have up-to-date data (e.g., request a corresponding up-to-date log record) to manage the cluster, such as a task process, and therefore, the master server must store all submitted log entries in the cluster.
In a conventional high availability scheme, a server may be selected as the primary server even if the server does not initially contain all committed log entries; in this case, it is necessary to include some additional mechanism to identify the missing log entry and to transmit the missing log to the server of the missing entry, which requires considerable additional mechanism and complexity; after the election of the main server, the log holes need to be supplemented additionally, and the sequentiality of the logs needs to be followed during supplementation, so that the performance of the cluster is low. To solve the above problem, on the basis of fig. 3, taking an example that the first server stores the first log with the current serial number as an example, please refer to fig. 4, and fig. 4 is a flowchart illustrating another data processing method according to an embodiment of the present invention. Prior to S35, the data processing method may further include:
and S34, judging whether the current sequence number is larger than or equal to all the log sequence numbers of other N slave servers.
If yes, go to S35; if not, the process returns to step S31.
It should be understood that, in the embodiment of the present invention, by giving each log a serial number, the server having the log with the latest serial number can become the master server, so that it is not necessary to transmit other log entries from the server to the elected master server, and the accuracy of data in the master server is forcibly ensured.
It should be noted that if the master server (first server) fails, i.e. the slave server does not receive the heartbeat or log of the first server, the voting timer of the slave server begins to wait for the timeout, and the slave server becomes a master server candidate of the cluster, initiating a request vote to other slave servers in the cluster; it should be understood that only the slave server that owns the most recent data (the most recent log record) may qualify as the new master server: if a slave server receives a vote of at least 50% of the other slave servers in the cluster, it may become the new master server in the cluster and serve the outsides. For example, the reselection time for switching from the slave server to the new master server is generally 150-; since the new master server has the latest data (latest log record), no special operation is needed to restore cluster data consistency when the master server authority is obtained; the new master server only needs to perform normal operation, then the log can automatically reply the consistency check of a Remote Procedure call protocol (RPC) of the additional log, when the log fails, cluster data automatically tend to be consistent, the interaction times among a plurality of servers in the cluster are reduced, the generation of dirty data in the cluster and the operation of recovering the cluster data are avoided, and therefore the service performance of the cluster is improved.
In the prior art, a lock queue mode is generally adopted for obtaining a request, normal logic operation of the lock queue is to perform locking processing on queue operation, performance overhead of locking is high, when a first server needs to send very many requests at high concurrency, time consumption of persistent operation of the request on the first server is long, how to avoid timeout failure of a client request caused by waiting for synchronization timeout and how to ensure high performance is avoided, on the basis of fig. 2, taking the first server having a lock-free queue as an example, please refer to fig. 5, and fig. 5 is a schematic flow diagram of another data processing method provided by the embodiment of the present invention. The above S37 may include:
s371, send the request to the lock-free queue.
It should be appreciated that a lock-free queue is a message queue without locks, which does not risk "deadlock", reduces the locking process, and may improve the efficiency of data processing. For example, a lock-free queue may be suitable for use in parallel with only two threads, one for pushing data (receiving requests) and the other for popping data (buffering requests).
And S372, sending the cache request to the lock-free queue so as to store the request to the first cache.
It should be appreciated that new requests in the lock-free queue may be stored in the first cache, the requests in the first cache may be replicated in bulk, and sent to other servers in the cluster.
For example, the master server establishes a Transmission Control Protocol (TCP) connection pool with other servers in the cluster, and data is concurrently sent to other servers through the TCP connection pool instead of being sent by a single TCP connection; when the main server (first server) copies the request, the main server can also carry a log identifier of the requested data; the slave server checks that the log of the last requested data matches before sending a store response to the master server. When the main server (first server) receives the storage response of the half slave server, the request of the client can be marked in an asynchronous processing mode, and the logs of the marked request are stored in batches; the first server can also reply a data synchronization success message of the client so that the client confirms that the cluster has realized the requested data synchronization and the current request is successfully stored.
If the slave server fails, after the configuration is changed (the failed slave server is offline, a new normal server is added), the new normal server sends a command to be synchronized to the master server, and after the master server synchronously applies the command to be synchronized, data are not synchronized to the failed slave server any more, and then the data are transmitted to the new normal server. However, the new normal server does not have data before the cluster, the synchronization of the cluster data takes too long, if the log stored before the cluster is too much, the disk of the new normal server may not be enough to store so much log, in an alternative embodiment, on the basis of fig. 5, taking the first server including a disk as an example, please refer to fig. 6, and fig. 6 is a flowchart of another data processing method provided by the embodiment of the present invention. The above S37 may further include:
and S373, acquiring the snapshot file according to the request.
The snapshot file includes all requests that the first cache has at the current point in time. For example, when the number of requests reaches a certain number, the requests are saved as snapshot files, and the request entries contained in the snapshot files are determined by the log records corresponding to the requests, so that the storage resources of the server can be effectively saved.
And S374, storing the snapshot file to a disk.
For example, the requests of the clients can be merged and then written into the disk in batch, and the data structure is flattened, so that only one-time writing is needed to avoid the full operation of the disk. And after the data is written into the disk, caching the storage position index of the request of the client. If the number of the index entries of the storage positions reaches a certain number, the main server creates a snapshot file, the state of the whole cluster is written into a stable persistent storage (a first service disk) in a snapshot mode, and logs before the time point of creating the snapshot file are all discarded, so that the performance is prevented from being influenced by the overlarge size of a single file, and the storage resources of the server are saved; for a new normal server, when the copied log is smaller than the last log contained in the snapshot file, the snapshot file is copied; for ease of transfer, the snapshot file may be divided into transmission blocks, which are installed directly into the state machine local to the new normal server after the new normal server is full.
In an optional implementation manner, in order to ensure accuracy of cluster data, on the basis of fig. 2, please refer to fig. 7 by taking an example that a request includes a log identifier, and fig. 7 is a schematic flowchart of another data processing method provided in an embodiment of the present invention. The above S38 may include:
and S381, judging whether the log identification is matched with the log sequence information in the second server.
The second server is any one or more servers in the other N slave servers.
If not, continue to execute S381; if so, then S382 is performed.
And S382, sending the request to the second server.
For example, when the main server concurrently sends the request data, the request data may arrive ahead of time, and at this time, the main server (the first server) considers that the request data sending fails; when the request data fails to be sent, the TCP connection pool takes the request data from the first cache again, and retry transmission is carried out. If the log in front of the slave server (second server) is missing, namely the log identification corresponding to the request data is not matched, the slave server can also perform local backward calculation to obtain matched log sequence information and inform the master server, and the master server performs transmission of the request data again, so that Match is skipped in batches, and the master server does not need to perform examination retry by one instruction.
It can be understood that the data synchronization of a single laggard slave server does not affect the performance of the whole cluster, the log writing is an idempotent RPC, if the data synchronization fails, the master server can retry the log, and the high availability of the cluster is not affected.
It is anticipated that while the cluster is running, a network partition may also occur, which separates the master and slave servers: when the slave server cannot receive the heartbeat of the master server, election is initiated to generate a new master server, at this time, a cluster generates the situation of double master servers, the original master server (a first server) is independently arranged in a network area, and a client submits data to the first server and cannot be copied to most slave servers, so that the data is submitted unsuccessfully; the client side can successfully submit the data to the new main service; after the network is recovered, the old main server (the first server) finds that a new main server with updated popularity exists in the cluster, and automatically downgrades the new main server to a slave server. It should be understood that the invention can ensure the high performance and high availability of the server cluster and the data accuracy of the cluster in a high concurrency scene without depending on a third-party service through various mechanisms such as data redundancy fault tolerance, a random countdown algorithm, lock-free queues, fine-grained lock log caching, batch replication, concurrent transmission of synchronous data, asynchronous callback response to a client and the like.
Next, on the basis of the slave server 12 shown in fig. 1, in order to achieve accuracy of cluster data and high availability of clusters, an embodiment of the present invention further provides a data processing method, which is applied to a second server, where the second server is any one of a plurality of slave servers of a cluster, please refer to fig. 8, and fig. 8 is a flowchart of another data processing method provided in the embodiment of the present invention. The data processing method comprises the following steps:
and S41, receiving the request sent by the main server.
The request is sent by the client and forwarded by the intermediate component, and the request is stored in a first cache of the main server.
And S42, storing the request when the log identification corresponding to the request is matched with the log serial number of the second server.
And S43, sending a response corresponding to the request to the main server so that the main server sends data synchronization success information to the client, and the client confirms that the data synchronization is successful in the current round.
As can be appreciated, the client communicates with the cluster through the intermediate component, which reduces the selection logic of the client for the cluster; when the data are synchronized, the slave server interacts with the cache of the main server instead of the disk of the main server, so that the data processing time can be effectively reduced, and the accuracy of the data is ensured.
To facilitate understanding of any data processing method, please refer to fig. 9, where fig. 9 is a signaling interaction diagram of a data processing method according to an embodiment of the present invention, where a client 14 communicates with a cluster through an intermediate component 13, where the cluster includes a first server 11, a second server 12a, and a third server 12b, the first server 11 is a master server of the cluster, and the second server 12a and the third server 12b are slave servers of the cluster.
When the cluster does not elect the main server, taking the voting waiting time of the first server 11 smaller than the voting waiting time of the communication between the second server 12a and the third server 12b as an example, each server of the cluster starts a respective voting timer, and when the waiting time of the first server 11 reaches a preset time, the election process of the data processing method includes:
s501, the first server sends the voting request information to the second server.
S502, the first server sends the voting request information to the third server.
It should be understood that S501 and S502 described above may be performed simultaneously, such as by transmitting the request voting information in a broadcast manner.
When the second server 12a receives the request voting information and confirms that the first server 11 has the latest log record, S503 is performed.
And S503, the second server sends a voting response corresponding to the voting information request to the first server.
When the third server 12b receives the request voting information and confirms that the first server 11 has the latest log record, S504 is performed.
And S504, the third server sends a voting response corresponding to the voting information request to the first server.
When the first server 11 receives the voting response sent by the second server 12a or the third server 12b, it is determined that the first server 11 is the main server of the cluster.
After the cluster elects the main server as the first server 11, the client 14 communicates with the cluster through the intermediate component 13, and the data or request synchronization process of the data processing method may include:
and S505, the client sends a request to the intermediate component.
S506, the intermediate component forwards the request of the client to the first server.
In the embodiment of the present invention, when the client 14 is started, the client directly communicates with the intermediate component 13, the intermediate component 13 randomly selects one server of the cluster to communicate, if the server selected for the first time is not the primary server, the intermediate component 13 traverses all servers of the cluster until the primary server (the first server 11) is found, and then records the information of the first server 11 in the cache of the client.
When the first server 11 receives the request forwarded by the intermediate component 13, store the request into a first cache of the first server 11 itself, and send the request stored in the first cache to other slave servers of the cluster, the synchronization process of the data processing method may further include:
s507, the first server sends the request stored in the first cache to the second server.
And S508, the first server sends the request stored in the first cache to the third server.
S509, the second server sends a storage response to the first server.
S510, the third server sends a storage response to the first server.
When the first server 11 receives the storage response corresponding to the request sent by the second server 12a or the third server 12b, it can confirm that the server data synchronization corresponding to the storage response is successful, so as to complete the request synchronization of the current round.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, the functional modules in the embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.
In summary, the present invention provides a data processing method, a server and a computer-readable storage medium, which relate to the field of cluster data processing. The method is applied to a first server with a cache, wherein the first server is a main server of a cluster, the cluster also comprises other N auxiliary servers, and the cluster is connected with a client through an intermediate component, and the method comprises the following steps: receiving a request of a client forwarded by an intermediate component; storing the request to a first cache of a first server; sending the request stored in the first cache to the other N slave servers; and when receiving at least N/2 storage responses corresponding to the requests sent by other N slave servers, sending data synchronization success information to the client, so that the client confirms that the data synchronization of the current round is successful. The client communicates with the cluster through the intermediate component, so that the selection logic of the client to the cluster is reduced; when the data are synchronized, the slave server interacts with the cache of the main server instead of the disk of the main server, so that the data processing time can be effectively reduced, and the accuracy of the data is ensured.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method is applied to a first server with a cache, wherein the first server is a main server of a cluster, the cluster further comprises other N slave servers, N is more than or equal to 2, the cluster is connected with a client through an intermediate component, and the method comprises the following steps:
receiving the request of the client forwarded by the intermediate component;
storing the request to a first cache of the first server;
sending the request stored in the first cache to the other N slave servers;
and when at least N/2 storage responses corresponding to the requests and sent by the other N secondary servers are received, sending data synchronization success information to the client, so that the client confirms that the data synchronization of the current round is successful.
2. The method of claim 1, wherein the first server has a voting timer for voting a master server of the cluster, and wherein prior to the receiving the request of the client forwarded by the intermediate component, the method further comprises:
judging whether the waiting time of the voting timer is greater than or equal to a preset time or not; the waiting time is the difference value between the current time of the first server and the starting time of the voting timer;
if yes, sending request voting information to the other N slave servers, so that the other N slave servers do not send voting requests within the election timeout time; the election timeout time is the interval time between the time when any one of the other N slave servers sends the voting request and the current time;
judging whether at least N/2 responses corresponding to the voting request information sent by the other N secondary servers are received within the voting timeout time;
and if so, confirming that the first server is the main server of the cluster.
3. The method of claim 2, wherein the first server stores a first log having a current sequence number, and wherein prior to the confirming that the first server is a primary server of the cluster, the method further comprises:
judging whether the current serial number is larger than or equal to all log serial numbers of other N slave servers;
if yes, the step of confirming that the first server is the main server of the cluster is executed.
4. The method of claim 1, wherein the first server has a lock-free queue, and wherein storing the request to a first cache of the first server comprises:
sending the request to the lock-free queue;
sending a cache request to the lock-free queue to store the request to the first cache.
5. The method of claim 4, wherein the first server comprises a disk, wherein storing the request to a first cache of the first server further comprises:
acquiring a snapshot file according to the request; the snapshot file comprises a current time point, and all the requests of the first cache are contained;
and storing the snapshot file to the disk.
6. The method of claim 1, wherein the request includes a log identification, and wherein sending the request stored in the first cache to the other N slave servers comprises:
judging whether the log identification is matched with log sequence information in a second server or not; the second server is any one or more servers in the other N slave servers;
and if so, sending the request to the second server.
7. A data processing method applied to a second server, wherein the second server is a slave server of a cluster, the cluster further includes a master server having a cache, and the cluster is connected to a client through an intermediate component, the method comprising:
receiving a request sent by the main server; the request is sent by the client and forwarded by the intermediate component, and is stored in a first cache of the main server;
when the log identification corresponding to the request is matched with the log serial number of the second server, storing the request;
and sending a response corresponding to the request to the main server so that the main server sends data synchronization success information to the client, and the client confirms that the data synchronization of the current round is successful.
8. A server is characterized in that the server is a main server of a cluster, the cluster also comprises other N slave servers, N is more than or equal to 2, and the cluster is connected with a client through an intermediate component;
the server comprises a processor and a memory, the memory storing machine executable instructions executable by the processor to implement the method of any one of claims 1-6.
9. A server is characterized in that the server is a slave server of a cluster, the cluster further comprises a master server with a cache, and the cluster is connected with a client through an intermediate component;
the server comprises a processor and a memory, the memory storing machine executable instructions executable by the processor to implement the method of claim 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1-6 or the method of claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010043982.8A CN111258822B (en) | 2020-01-15 | 2020-01-15 | Data processing method, server, and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010043982.8A CN111258822B (en) | 2020-01-15 | 2020-01-15 | Data processing method, server, and computer-readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111258822A true CN111258822A (en) | 2020-06-09 |
| CN111258822B CN111258822B (en) | 2024-02-27 |
Family
ID=70952130
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010043982.8A Active CN111258822B (en) | 2020-01-15 | 2020-01-15 | Data processing method, server, and computer-readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111258822B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111953772A (en) * | 2020-08-11 | 2020-11-17 | 北京达佳互联信息技术有限公司 | Request processing method, device, server and storage medium |
| CN112235363A (en) * | 2020-09-28 | 2021-01-15 | 华云数据控股集团有限公司 | Data processing method, device, electronic equipment, storage medium and system |
| CN112835936A (en) * | 2021-02-05 | 2021-05-25 | 北京电解智科技有限公司 | Data processing method and device and service cluster |
| CN113032232A (en) * | 2021-03-09 | 2021-06-25 | 广州虎牙科技有限公司 | Related method for detecting performance of equipment, related device and equipment |
| CN113297236A (en) * | 2020-11-10 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Method, device and system for electing main node in distributed consistency system |
| CN113626011A (en) * | 2021-07-21 | 2021-11-09 | 北京万维之道信息技术有限公司 | Data request processing method, device and equipment based on PHP architecture |
| CN113783921A (en) * | 2021-01-27 | 2021-12-10 | 北京京东振世信息技术有限公司 | Method and device for creating cache component |
| CN114143127A (en) * | 2021-11-29 | 2022-03-04 | 网易（杭州）网络有限公司 | Data broadcasting method and device and electronic equipment |
| CN114461593A (en) * | 2022-04-13 | 2022-05-10 | 云和恩墨(北京)信息技术有限公司 | Log writing method and device, electronic equipment and storage medium |
| CN114979180A (en) * | 2022-05-24 | 2022-08-30 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
| CN114969072A (en) * | 2022-06-06 | 2022-08-30 | 北京友友天宇系统技术有限公司 | Data transmission method, device and equipment based on state machine and data persistence |
| CN114979164A (en) * | 2022-04-14 | 2022-08-30 | 网易（杭州）网络有限公司 | Virtual room allocation method and device and electronic equipment |
| CN116048424A (en) * | 2023-03-07 | 2023-05-02 | 浪潮电子信息产业股份有限公司 | IO data processing method, device, equipment and medium |
| CN114979180B (en) * | 2022-05-24 | 2024-05-17 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077635A1 (en) * | 2006-09-22 | 2008-03-27 | Digital Bazaar, Inc. | Highly Available Clustered Storage Network |
| CN105554143A (en) * | 2015-12-25 | 2016-05-04 | 浪潮(北京)电子信息产业有限公司 | High-availability cache server and data processing method and system thereof |
| CN106878473A (en) * | 2017-04-20 | 2017-06-20 | 腾讯科技（深圳）有限公司 | A kind of message treatment method, server cluster and system |
| CN108322533A (en) * | 2018-01-31 | 2018-07-24 | 广州鼎甲计算机科技有限公司 | Configuration and synchronization method between distributed type assemblies node based on operation log |
| CN109547512A (en) * | 2017-09-22 | 2019-03-29 | 中国移动通信集团浙江有限公司 | A kind of method and device of the distributed Session management based on NoSQL |
-
2020
- 2020-01-15 CN CN202010043982.8A patent/CN111258822B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077635A1 (en) * | 2006-09-22 | 2008-03-27 | Digital Bazaar, Inc. | Highly Available Clustered Storage Network |
| CN105554143A (en) * | 2015-12-25 | 2016-05-04 | 浪潮(北京)电子信息产业有限公司 | High-availability cache server and data processing method and system thereof |
| CN106878473A (en) * | 2017-04-20 | 2017-06-20 | 腾讯科技（深圳）有限公司 | A kind of message treatment method, server cluster and system |
| CN109547512A (en) * | 2017-09-22 | 2019-03-29 | 中国移动通信集团浙江有限公司 | A kind of method and device of the distributed Session management based on NoSQL |
| CN108322533A (en) * | 2018-01-31 | 2018-07-24 | 广州鼎甲计算机科技有限公司 | Configuration and synchronization method between distributed type assemblies node based on operation log |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111953772B (en) * | 2020-08-11 | 2022-11-22 | 北京达佳互联信息技术有限公司 | Request processing method, device, server and storage medium |
| CN111953772A (en) * | 2020-08-11 | 2020-11-17 | 北京达佳互联信息技术有限公司 | Request processing method, device, server and storage medium |
| CN112235363B (en) * | 2020-09-28 | 2022-02-01 | 华云数据控股集团有限公司 | Data processing method, device, electronic equipment, storage medium and system |
| CN112235363A (en) * | 2020-09-28 | 2021-01-15 | 华云数据控股集团有限公司 | Data processing method, device, electronic equipment, storage medium and system |
| CN113297236A (en) * | 2020-11-10 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Method, device and system for electing main node in distributed consistency system |
| CN113783921A (en) * | 2021-01-27 | 2021-12-10 | 北京京东振世信息技术有限公司 | Method and device for creating cache component |
| CN112835936A (en) * | 2021-02-05 | 2021-05-25 | 北京电解智科技有限公司 | Data processing method and device and service cluster |
| CN113032232A (en) * | 2021-03-09 | 2021-06-25 | 广州虎牙科技有限公司 | Related method for detecting performance of equipment, related device and equipment |
| CN113032232B (en) * | 2021-03-09 | 2023-06-20 | 广州虎牙科技有限公司 | Correlation method, correlation device and equipment for performance detection of equipment |
| CN113626011B (en) * | 2021-07-21 | 2024-02-13 | 北京万维之道信息技术有限公司 | PHP architecture-based data request processing method, device and equipment |
| CN113626011A (en) * | 2021-07-21 | 2021-11-09 | 北京万维之道信息技术有限公司 | Data request processing method, device and equipment based on PHP architecture |
| CN114143127A (en) * | 2021-11-29 | 2022-03-04 | 网易（杭州）网络有限公司 | Data broadcasting method and device and electronic equipment |
| CN114143127B (en) * | 2021-11-29 | 2023-10-20 | 网易（杭州）网络有限公司 | Data broadcasting method and device and electronic equipment |
| CN114461593B (en) * | 2022-04-13 | 2022-07-29 | 云和恩墨(北京)信息技术有限公司 | Log writing method and device, electronic device and storage medium |
| CN114461593A (en) * | 2022-04-13 | 2022-05-10 | 云和恩墨(北京)信息技术有限公司 | Log writing method and device, electronic equipment and storage medium |
| CN114979164A (en) * | 2022-04-14 | 2022-08-30 | 网易（杭州）网络有限公司 | Virtual room allocation method and device and electronic equipment |
| CN114979164B (en) * | 2022-04-14 | 2023-11-17 | 网易（杭州）网络有限公司 | Virtual room distribution method and device and electronic equipment |
| CN114979180A (en) * | 2022-05-24 | 2022-08-30 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
| CN114979180B (en) * | 2022-05-24 | 2024-05-17 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
| CN114969072B (en) * | 2022-06-06 | 2023-06-09 | 北京友友天宇系统技术有限公司 | Data transmission method, device and equipment based on state machine and data persistence |
| CN114969072A (en) * | 2022-06-06 | 2022-08-30 | 北京友友天宇系统技术有限公司 | Data transmission method, device and equipment based on state machine and data persistence |
| CN116048424A (en) * | 2023-03-07 | 2023-05-02 | 浪潮电子信息产业股份有限公司 | IO data processing method, device, equipment and medium |
| CN116048424B (en) * | 2023-03-07 | 2023-06-06 | 浪潮电子信息产业股份有限公司 | IO data processing method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111258822B (en) | 2024-02-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111258822B (en) | Data processing method, server, and computer-readable storage medium | |
| US11853263B2 (en) | Geographically-distributed file system using coordinated namespace replication over a wide area network | |
| US11360854B2 (en) | Storage cluster configuration change method, storage cluster, and computer system | |
| US10621200B2 (en) | Method and apparatus for maintaining replica sets | |
| US8140623B2 (en) | Non-blocking commit protocol systems and methods | |
| US8166007B2 (en) | Failure tolerant transaction processing system | |
| US9361311B2 (en) | Distributed file system using consensus nodes | |
| EP1222540B1 (en) | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas | |
| US9495381B2 (en) | Geographically-distributed file system using coordinated namespace replication over a wide area network | |
| US9846704B2 (en) | Distributed file system using consensus nodes | |
| CN107832138B (en) | Method for realizing flattened high-availability namenode model | |
| US20080077635A1 (en) | Highly Available Clustered Storage Network | |
| EP2434729A2 (en) | Method for providing access to data items from a distributed storage system | |
| CN111368002A (en) | Data processing method, system, computer equipment and storage medium | |
| CN108989391B (en) | Consistency processing method and system | |
| CN110635941A (en) | Database node cluster fault migration method and device | |
| CN112527759B (en) | Log execution method and device, computer equipment and storage medium | |
| CN111752488B (en) | Management method and device of storage cluster, management node and storage medium | |
| CN113965578A (en) | Method, device, equipment and storage medium for electing master node in cluster | |
| CN114244859B (en) | Data processing method and device and electronic equipment | |
| CN110661841B (en) | Data consistency method for distributed service discovery cluster in micro-service architecture | |
| CN109726211B (en) | Distributed time sequence database | |
| CN113326251B (en) | Data management method, system, device and storage medium | |
| CN106951443B (en) | Method, equipment and system for synchronizing copies based on distributed system | |
| CN111291063B (en) | Master and backup copy election method, system, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |