CN111258822B - Data processing method, server, and computer-readable storage medium - Google Patents
Data processing method, server, and computer-readable storage medium Download PDFInfo
- Publication number
- CN111258822B CN111258822B CN202010043982.8A CN202010043982A CN111258822B CN 111258822 B CN111258822 B CN 111258822B CN 202010043982 A CN202010043982 A CN 202010043982A CN 111258822 B CN111258822 B CN 111258822B
- Authority
- CN
- China
- Prior art keywords
- server
- request
- cluster
- client
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 claims abstract description 48
- 230000004044 response Effects 0.000 claims abstract description 26
- 238000004590 computer program Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 21
- 230000001360 synchronised effect Effects 0.000 abstract description 12
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 8
- 230000003993 interaction Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000011664 signaling Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000002441 reversible effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000002045 lasting effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/1675—Temporal synchronisation or re-synchronisation of redundant processing components
- G06F11/1683—Temporal synchronisation or re-synchronisation of redundant processing components at instruction level
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method, a server and a computer readable storage medium, and relates to the field of cluster data processing. The method is applied to a first server with a cache, wherein the first server is a master server of a cluster, the cluster also comprises other N slave servers, and the cluster is connected with a client through an intermediate component; the method comprises the following steps: receiving a client request forwarded by an intermediate component; storing the request to a first cache of a first server; transmitting the request stored in the first cache to other N slave servers; and when receiving at least N/2 storage responses corresponding to the request, sending data synchronization success information to the client so that the client confirms that the data synchronization of the round is successful. The client communicates with the cluster through the intermediate component, so that the selection logic of the client to the cluster is reduced; when data is synchronized, the slave server interacts with the cache of the master server, but not with the disk of the master server, so that the data processing time is effectively reduced, and the data accuracy is ensured.
Description
Technical Field
The present invention relates to the field of data processing of clusters, and in particular, to a data processing method, a server, and a computer readable storage medium.
Background
In the current master-slave high availability scheme, there are two ways of processing data, asynchronous and synchronous: if the method is an asynchronous processing mode, the main server is down to directly cause the data loss of the cluster; the method has the advantages that the method is characterized in that the method adopts a synchronous processing mode for cluster switching, the performance problem of the clusters is caused, the steps of master-slave switching detection are complicated, the consistency of cluster data cannot be ensured, dirty data can be generated, and the cluster data can be recovered by subsequent data restoration; in addition, there is a large delay in master-slave switching, in order to reduce the delay, a third party service needs to be relied on, if a node of the third party service fails or a network is partitioned, a master-slave switching request of a cluster fails, and then the cluster is unavailable, so that the high availability of the cluster cannot be guaranteed.
In order to ensure high availability of the cluster, a high availability scheme of data peer-to-peer is generally used, which is respectively connected with a client, a database and a slave server through a master server, when the master server receives a request of the client, the master server stores the request into a disk of the master server and sends the request in the disk to the slave server so as to realize data synchronization and high availability of the cluster; the scheme can be applied to scenes with low requirements on data accuracy, such as barrage service, and a user can see barrages sent by the user; for the bullet screen sent by others, the bullet screen may be stored in the disk of a different server, and even if the bullet screen of other users stored in the disk is lost, the user does not feel the bullet screen.
Disclosure of Invention
In view of this, an object of the present invention is to provide a data processing method, a server, and a computer-readable storage medium.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
in a first aspect, the present invention provides a data processing method, applied to a first server with a cache, where the first server is a master server of a cluster, the cluster further includes N other slave servers, N is greater than or equal to 2, and the cluster is connected to a client through an intermediate component. The method comprises the following steps: receiving a request of the client forwarded by the intermediate component; storing the request to a first cache of the first server; sending the request stored in the first cache to the other N slave servers; and when receiving at least N/2 storage responses corresponding to the requests sent by the other N slave servers, sending data synchronization success information to the client so that the client confirms that the data synchronization of the round is successful.
In an alternative embodiment, the first server has a voting timer for electing a master server of the cluster, and before the receiving the request of the client forwarded by the intermediate component, the method further comprises: judging whether the waiting time of the voting timer is larger than or equal to preset time; the waiting time is the difference between the current time of the first server and the starting time of the voting timer. If yes, sending request voting information to the other N slave servers, so that the other N slave servers do not send voting requests within the election timeout time; the election timeout time is an interval time between the time of sending the voting request from any one of the other N slave servers and the current time. Judging whether at least N/2 responses corresponding to the request voting information sent by the other N slave servers are received in the election timeout time; if yes, confirming that the first server is a master server of the cluster.
In an alternative embodiment, the first server stores a first log having a current serial number, and before the identifying the first server as the primary server of the cluster, the method further includes: judging whether the current serial number is larger than or equal to all log serial numbers of the other N slave servers; if yes, executing the step of confirming that the first server is the master server of the cluster.
In an alternative embodiment, the first server has a lock-free queue, and the storing the request to the first cache of the first server includes: sending the request to the lock-free queue; and sending a cache request to the lock-free queue to store the request to the first cache.
In an alternative embodiment, the first server includes a disk, and the storing the request to the first cache of the first server further includes: according to the request, obtaining a snapshot file; the snapshot file comprises a current time point, and the first cache has all the requests; and storing the snapshot file to the disk.
In an alternative embodiment, the request includes a log identifier, and the sending the request stored in the first cache to the other N slave servers includes: judging whether the log identification is matched with log sequence information in a second server or not; the second server is any one or more servers in the other N slave servers; and if so, sending the request to the second server.
In a second aspect, the present invention provides a data processing method, applied to a second server, where the second server is a slave server of a cluster, and the cluster further includes a master server with a cache, where the cluster is connected to a client through an intermediate component, and the method includes: receiving a request sent by the main server; the request is a request sent by the main server at the client terminal which receives the forwarding of the intermediate component, and the request is stored in a first cache of the main server. And storing the request when the log identifier corresponding to the request is matched with the log serial number of the second server. And sending a response corresponding to the request to the main server so that the main server sends data synchronization success information to the client, and the client confirms that the data synchronization of the round is successful.
In a third aspect, the present invention provides a server, the server being a master server of a cluster, the cluster further comprising N slave servers, N being ≡2, the cluster being connected to a client via an intermediate component, the server comprising a processor and a memory, the memory storing machine executable instructions executable by the processor, the processor being executable to implement the method of any one of the preceding embodiments.
In a fourth aspect, the present invention provides a server, where the server is a slave server of a cluster, and the cluster further includes a master server with a cache, where the cluster is connected to a client through an intermediate component. The server includes a processor and a memory storing machine executable instructions executable by the processor to implement the method of the previous embodiments.
In a fifth aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of the preceding embodiments.
The invention provides a data processing method, a server and a computer readable storage medium, and relates to the field of cluster data processing. The data processing method is applied to a first server with a cache, the first server is a master server of a cluster, the cluster further comprises other N slave servers, N is more than or equal to 2, the cluster is connected with a client through an intermediate component, and the method comprises the following steps: receiving a request of the client forwarded by the intermediate component; storing the request to a first cache of the first server; sending the request stored in the first cache to the other N slave servers; and when receiving at least N/2 storage responses corresponding to the requests sent by the other N slave servers, sending data synchronization success information to the client so that the client confirms that the data synchronization of the round is successful. The client communicates with the cluster through the intermediate component, so that the selection logic of the client for the cluster is reduced; when the data is synchronized, the slave server interacts with the cache of the master server instead of the disk of the master server, so that the data processing time can be effectively reduced, and the accuracy of the data is ensured.
In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of interaction between a client and a cluster according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating another data processing method according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating another data processing method according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating another data processing method according to an embodiment of the present invention;
FIG. 6 is a flowchart illustrating another data processing method according to an embodiment of the present invention;
FIG. 7 is a flowchart of another data processing method according to an embodiment of the present invention;
FIG. 8 is a flowchart illustrating another data processing method according to an embodiment of the present invention;
fig. 9 is a signaling interaction diagram of a data processing method according to an embodiment of the present invention.
Icon: 11-first server, 12-slave server, 12 a-second server, 12 b-third server, 13-intermediate component, 14-client.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present invention.
It is noted that relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In order to ensure high availability of the cluster, a high availability scheme of data peer-to-peer is generally used, which is respectively connected with a client, a database and a slave server through a master server, when the master server receives a request of the client, the master server stores the request into a disk of the master server and sends the request in the disk to the slave server so as to realize data synchronization and high availability of the cluster; the scheme can be applied to scenes with low requirements on data accuracy, such as barrage service, and a user can see barrages sent by the user; for the bullet screen sent by others, the bullet screen may be stored in the disk of a different server, and even if the bullet screen of other users stored in the disk is lost, the user does not feel the bullet screen. How to rapidly process the request sent by the client and synchronize the request into the cluster is a problem to be solved at present.
In order to solve the above-mentioned problems and the shortcomings set forth in the background art, an embodiment of the present invention provides a data processing method, which is applied to a server. The server is deployed in a cluster, please refer to fig. 1, and fig. 1 is a schematic diagram illustrating interaction between a client and a cluster according to an embodiment of the present invention. The client 14 is connected to the cluster through an intermediate component 13, which cluster comprises a first server 11 with a cache and other N slave servers 12, N being a natural number greater than or equal to two.
For the above-mentioned intermediate component 13, it may be a software process, for example, the intermediate component 13 is deployed on each server in the cluster, and the client may communicate with any server in the cluster through the intermediate component 13.
For the above-mentioned clients, it may be, but not limited to, mobile phones, personal computers, smart bracelets, smart home appliances, service servers, etc.
It should be appreciated that any of the servers described above may include, but are not limited to, memory, a processor, and a communication interface. The memory, the processor and the communication interface are electrically connected with each other directly or indirectly to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The memory may be used for storing software programs and modules, such as program instructions/modules corresponding to the data processing methods provided in the embodiments of the present invention, and the processor executes the software programs and modules stored in the memory, thereby executing various functional applications and data processing. The communication interface may be used for communication of signaling or data with other node devices. The server may have a plurality of communication interfaces in the present invention.
The Memory may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
The processor may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
Referring to fig. 2, fig. 2 is a flow chart of a data processing method according to an embodiment of the present invention. The data processing method comprises the following steps:
S36, receiving the request of the client forwarded by the intermediate component.
For example, when the client starts, the client directly communicates with the intermediate component, the intermediate component randomly selects a server in a cluster to communicate, if the first selected server is not the main server (returns an error), the client traverses other servers in the cluster until the main server is found, and then records the server information into the cache of the client; it should be appreciated that in the above case, the client is unaware, does not need to perform the selection logic of the primary server on the cluster, and only needs to communicate directly with the intermediate component. If the primary server in the cluster has crashed, the request of the intermediate component will time out, after which the random pick server process will be retried again.
S37, storing the request to a first cache of the first server.
It should be understood that, when the request is stored in the cache of the first server and data synchronization is performed subsequently, the data can be directly obtained from the first cache without interaction with the disk of the server, so that the data processing efficiency of the cluster is effectively improved.
S38, sending the request stored in the first cache to other N slave servers.
It should be appreciated that single point storage of the primary server must not achieve the goal of high availability of the cluster, and that this request needs to be synchronized to other servers of the cluster in order to prevent the primary server from downtime resulting in data loss or reduced accuracy.
And S39, when receiving at least N/2 storage responses corresponding to the requests sent by the other N slave servers, sending data synchronization success information to the client so that the client confirms that the data synchronization of the round is successful.
It can be understood that the client communicates with the cluster through the intermediate component, so that the selection logic of the client for the cluster is reduced; when the data is synchronized, the slave server interacts with the cache of the master server instead of the disk of the master server, so that the data processing time can be effectively reduced, and the accuracy of the data is ensured.
In an alternative embodiment, when the service cluster is online, in order to obtain the main server of the cluster, the current technical solution may rely on a third party service, and in order to reduce the dependency on the third party service, on the basis of fig. 2, taking the first server with the voting timer as an example, fig. 3 is a schematic flow chart of another data processing method provided in the embodiment of the present invention. It should be understood that the voting timer is used to elect a master server of the cluster, and the voting timer may be a timer thread started by the first server, or may be a timer software provided on the first server, and the data processing method may further include, before S36 described above:
S31, judging whether the waiting time of the voting timer is larger than or equal to the preset time.
The waiting time is the difference between the current time of the first server and the start time of the voting timer. For example, the preset time may be a random number of the order of one millisecond, such as 1 millisecond, 3 milliseconds, etc.
If the waiting time of the voting timer is greater than or equal to the preset time, executing S32; if the waiting time of the voting timer is less than the preset time, waiting until the waiting time is greater than or equal to the preset time, and executing S32.
S32, sending the voting request information to the other N slave servers so that the other N slave servers do not send voting requests within the voting timeout time.
The election timeout is an interval between the time of sending the voting request from any one of the other N slave servers and the current time. For example, the election timeout may be greater than the broadcast time of the first server to the other N slave servers, in which case the first server may send the request voting information to the other N slave servers before the other N slave servers timeout to avoid election conflicts when the cluster elects the master server.
S33, judging whether other N responses which are sent by the server and correspond to the request voting information are received or not within the election timeout time.
It will be appreciated that when the request voting information sent by the first server is received from the server, a response corresponding to the request voting information may be generated, so that the first server confirms that the request voting information is sent successfully.
If at least N/2 responses corresponding to the request voting information sent by the other N slave servers are not received within the election timeout time, returning to execute S31; if at least N/2 responses corresponding to the request voting information transmitted from the server are received within the election timeout, S35 is executed.
S35, confirming that the first server is a master server of the cluster.
It can be appreciated that by setting the voting timer, the master server of the cluster can be elected on the basis of not using the third party service. For example, the cluster may wait by starting a timer thread (voting timer), the number of milliseconds waiting being a random number (preset time); when the voting waiting time of the first server reaches the preset time, the first server sends out a voting request message to other servers of the cluster; because the waiting time of the voting timer of each server is inconsistent, the servers can be scattered, and in most cases, the waiting time of the voting timer of only one server (such as the first server) can reach the preset time, and then the voting is won, so that the rights of the main server are obtained; then, sending a heartbeat packet (request voting information) to other servers before other servers timeout, so that a plurality of servers are prevented from sending request voting information simultaneously, and the master server election conflict of the cluster is caused, so that the master server can be selected quickly; it should be appreciated that the broadcast time between servers < < election timeout < average failure interval time, which is the average time between two failures for a server, is an order of magnitude less than the average failure interval time.
It should be appreciated that in a cluster, the primary server needs to have the latest data (such as the latest log record corresponding to the request) to manage the cluster, such as task processes, so the primary server must store all the submitted log entries in the cluster.
In a traditional high availability scenario, a server may be selected as the primary server even if it does not initially contain all of the journal entries that have been committed; at this point, it is necessary to include some additional mechanism to identify the missing log entry and transmit the missing log to the server of the missing entry, which requires considerable additional mechanism and complexity; because the primary server needs to supplement log holes additionally after election, the supplementing process needs to follow the sequence of the log, which can result in poor performance of the cluster. In order to solve the above-mentioned problem, taking the example that the first server stores the first log with the current serial number on the basis of fig. 3, please refer to fig. 4, fig. 4 is a flow chart of another data processing method according to an embodiment of the present invention. Prior to S35, the data processing method may further include:
s34, judging whether the current serial number is larger than or equal to all log serial numbers of other N slave servers.
If yes, executing the step S35; if not, the process returns to S31.
It should be understood that, in the embodiment of the invention, by providing each log with a serial number, the server with the latest serial number log can become the master server, so that the slave server is not required to transmit other log entries to the elected master server, and the accuracy of data in the master server is compulsorily ensured.
It should be noted that if the master server (the first server) fails, i.e. the slave server does not receive the heartbeat or log of the first server, the slave server's voting timer begins to wait for a timeout, the slave server becomes the master server candidate of the cluster, and requests voting to other slave servers in the cluster; it should be appreciated that only the slave servers that own the latest data (latest log records) may qualify as new master servers: if at least 50% of the votes from the other slave servers in the cluster are received by the slave servers, then it becomes the new master server in the cluster and provides services to the outside. For example, the reselection time for a slave server to switch to a new master server is typically 150-300 milliseconds, much less than the time for the slave server to release a version; since the new main server has the latest data (the latest log record), no special operation is needed to restore the consistency of the cluster data when the main server authority is obtained; the new main server only needs to perform normal operation, then the log can automatically check the consistency of the remote procedure call protocol (Remote Procedure Call Protocol, RPC) for replying the additional log, when the remote procedure call protocol fails, the cluster data automatically tends to be consistent, the number of interactions among a plurality of servers in the cluster is reduced, the generation of dirty data in the cluster is avoided, and the operation of recovering the cluster data is avoided, so that the service performance of the cluster is improved.
In the prior art, a lock queue is generally adopted for acquiring a request, normal logic operation of the lock queue is to perform locking processing on the queue operation, the locking performance cost is high, when a first server needs high concurrency and very many requests, the duration of the lasting operation of the first server is long, how to avoid the failure of the client request timeout caused by waiting for synchronization timeout, and how to ensure high performance, on the basis of fig. 2, taking the example that the first server has a lock-free queue, please refer to fig. 5, and fig. 5 is a flow diagram of another data processing method provided by the embodiment of the invention. The S37 may include:
s371, the request is sent to a lock-free queue.
It should be appreciated that a lock-free queue is a message queue that is not locked, has no risk of "deadlock", reduces the locking process, and may improve the efficiency of data processing. For example, a lock-free queue may be suitable for use with only two threads in parallel, one being a thread that pushes data (receives requests) and the other being a thread that pops data (caches requests).
S372, sending the cache request to a lock-free queue to store the request to the first cache.
It should be appreciated that new requests in the lock-free queue may be stored in the first cache, the requests in the first cache may be batch replicated, and sent to other servers in the cluster.
For example, the primary server establishes a pool of transmission control protocol (Transmission Control Protocol, TCP) connections with other servers of the cluster, and data is sent concurrently to the other servers over the pool of links, rather than a single link; when the main server (first server) copies the request, the main server can also carry the log identification of the last request data; the slave server sends a store response to the master server if the log of the last requested data matches, by checking. When the master server (the first server) receives the half storage response of the slave server, the request of the client can be marked in an asynchronous processing mode, and logs of marked requests are stored in batches; the first server may also reply to the client with a data synchronization success message, so that the client confirms that the cluster has achieved the requested data synchronization, and the request is successfully stored.
If the slave server fails, after configuration is changed (the failed slave server is disconnected, a new normal server is added), the new normal server sends a command to be synchronized to the master server, and after the master server synchronously applies the command to be synchronized, the data is not synchronized to the failed slave server, and is transferred to the new normal server. However, if the new normal server does not have data before the cluster, synchronization of the cluster data takes too long, and if the logs stored before the cluster are too many, the disk of the new normal server may not be enough to store so many logs, and in an alternative embodiment, taking the first server including the disk as an example on the basis of fig. 5, fig. 6 is a schematic flow chart of another data processing method provided in the embodiment of the present invention. The step S37 may further include:
S373, obtaining the snapshot file according to the request.
The snapshot file includes all requests that the first cache has at the current point in time. For example, when a certain number of requests are reached, a plurality of requests are saved as a snapshot file, and the request entries contained in the snapshot file are determined by requesting the corresponding log records, so that the storage resources of the server can be effectively saved.
S374, storing the snapshot file to a disk.
For example, the requests of the clients may be first merged and then written to the disk in batches, and the data structure is flattened, and only one write is needed to avoid the full operation of the disk. After writing to disk, the storage location index of the request of the client is cached. If the number of storage position index entries reaches a certain number, the main server creates snapshot files, the state of the whole cluster is written into stable persistent storage (a disk of a first service) in a snapshot mode, and all logs before the time point of creating the snapshot files are discarded, so that the influence of overlarge single files on performance is avoided, and the storage resources of the server are saved; for a new normal server, when the copied log is smaller than the last log contained in the snapshot file, changing the copied snapshot file; to facilitate transfer, the snapshot file may be partitioned into blocks for transfer, and after the new normal server is full, directly installed to a state machine local to the new normal server.
In an alternative implementation manner, in order to ensure accuracy of the cluster data, taking an example that the request includes a log identifier on the basis of fig. 2, please refer to fig. 7, fig. 7 is a flowchart of another data processing method according to an embodiment of the present invention. The step S38 may include:
s381, judging whether the log identification is matched with log sequence information in the second server.
The second server is any one or more of the other N slave servers.
If not, continuing to execute S381; if so, S382 is performed.
S382, the request is sent to the second server.
For example, when the main server concurrently transmits the request data, the request data may arrive in advance, and the main server (the first server) considers that the request data transmission fails; and when the transmission of the request data fails, the TCP connection pool is taken out of the first buffer memory again, and retry transmission is carried out. If the log in front of the slave server (the second server) is missing, namely the log identification corresponding to the request data is not matched, the slave server can also perform local reverse calculation to obtain matched log sequence information and inform the master server, and the master server performs transmission of the request data again, so that Match is skipped in batches, and an instruction of the master server is not needed for checking and retrying.
It can be understood that the data synchronization of a single backward slave server does not affect the performance of the whole cluster, the log writing is an idempotent RPC, and if the data synchronization fails, the master server retries, so that the high availability of the cluster is not affected.
It is envisioned that when the cluster is running, it may also happen that a network partition separates the master server and the slave servers: when the slave server cannot receive the heartbeat of the master server, a new master server is generated by initiating election, and the cluster generates the situation of double master servers, the original master server (first server) is independently in a network area, and the client side submits data to the first server and cannot copy the data to a plurality of slave servers, so that the data is always submitted unsuccessfully; the client submits data to the new main service, and the data can be submitted successfully; after the network is restored, the old main server (first server) finds out the new main server with updated popularity in the cluster, and then automatically demotes to the slave server. It should be understood that the present invention can ensure high performance and high availability of the server cluster and data accuracy of the cluster in a high concurrency scenario by various mechanisms such as data redundancy fault tolerance, random countdown algorithm, lock-free queue, fine-grained lock log cache, batch replication, concurrent transmission of synchronous data, asynchronous callback response to the client, etc., without relying on a third party service.
In order to achieve accuracy of cluster data and high availability of clusters, the embodiment of the present invention further provides a data processing method applied to a second server, which is any one of a plurality of slave servers of a cluster, based on the slave servers 12 shown in fig. 1, please refer to fig. 8, and fig. 8 is a flow chart of another data processing method provided in the embodiment of the present invention. The data processing method comprises the following steps:
s41, receiving a request sent by the main server.
The request is a request sent by the primary server at a client terminal receiving the intermediate component forwarding, and the request is stored in a first cache of the primary server.
S42, storing the request when the log identification corresponding to the request is matched with the log serial number of the second server.
S43, sending a response corresponding to the request to the main server so that the main server sends data synchronization success information to the client, and the client confirms that the data synchronization of the round is successful.
It can be understood that the client communicates with the cluster through the intermediate component, so that the selection logic of the client for the cluster is reduced; when the data is synchronized, the slave server interacts with the cache of the master server instead of the disk of the master server, so that the data processing time can be effectively reduced, and the accuracy of the data is ensured.
In order to facilitate understanding of any one of the above data processing methods, please refer to fig. 9, fig. 9 is a signaling interaction diagram of a data processing method provided in an embodiment of the present invention, in which a client 14 communicates with a cluster through an intermediate component 13, the cluster includes a first server 11, a second server 12a, and a third server 12b, the first server 11 is a master server of the cluster, and the second server 12a and the third server 12b are slave servers of the cluster.
Taking the example that the voting waiting time of the first server 11 is smaller than the voting waiting time of the second server 12a and the third server 12b when the cluster does not elect the main server, each server of the cluster starts a respective voting timer, and when the waiting time of the first server 11 reaches a preset time, the election process of the data processing method comprises the following steps:
s501, the first server sends request voting information to the second server.
S502, the first server sends request voting information to the third server.
It should be appreciated that S501 and S502 described above may be performed simultaneously, such as by broadcasting to transmit request voting information.
When the second server 12a receives the request voting information and confirms that the first server 11 has the latest log record, S503 is executed.
S503, the second server sends a voting response corresponding to the voting information request to the first server.
When the third server 12b receives the request voting information and confirms that the first server 11 has the latest log record, S504 is performed.
S504, the third server sends a voting response corresponding to the voting information request to the first server.
When the first server 11 receives the voting response transmitted from the second server 12a or the third server 12b, it is determined that the first server 11 is the master server of the cluster.
After the cluster elects the primary server to be the first server 11, the client 14 communicates with the cluster through the intermediate component 13, and the data or request synchronization process of the data processing method may include:
s505, the client sends a request to the intermediate component.
S506, the intermediate component forwards the request of the client to the first server.
In the embodiment of the present invention, when the client 14 is started, it directly communicates with the intermediate component 13, where the intermediate component 13 randomly selects one server of the cluster to communicate, and if the first selected server is not the primary server, the intermediate component 13 traverses all servers of the cluster until the primary server (the first server 11) is found, and then records the information of the first server 11 in the cache of the client.
When the first server 11 receives the request forwarded by the intermediate component 13, the request is stored in a first cache of the first server 11, and the request stored in the first cache is sent to other slave servers of the cluster, and the synchronization process of the data processing method may further include:
s507, the first server sends the request stored in the first cache to the second server.
S508, the first server sends the request stored in the first cache to the third server.
S509, the second server sends a storage response to the first server.
S510, the third server sends a storage response to the first server.
When the first server 11 receives the storage response corresponding to the request sent by the second server 12a or the third server 12b, the success of the server data synchronization corresponding to the storage response can be confirmed, so as to complete the request synchronization of the round.
In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, of the flowcharts and block diagrams in the figures that illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present invention may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.
In summary, the present invention provides a data processing method, a server and a computer readable storage medium, and relates to the field of cluster data processing. The method is applied to a first server with a cache, wherein the first server is a master server of a cluster, the cluster also comprises other N slave servers, and the cluster is connected with a client through an intermediate component, and the method comprises the following steps: receiving a request of a client forwarded by an intermediate component; storing the request to a first cache of a first server; transmitting the request stored in the first cache to other N slave servers; and when receiving at least N/2 storage responses corresponding to the requests sent by the other N slave servers, sending data synchronization success information to the client so that the client confirms that the data synchronization of the round is successful. The client communicates with the cluster through the intermediate component, so that the selection logic of the client for the cluster is reduced; when the data is synchronized, the slave server interacts with the cache of the master server instead of the disk of the master server, so that the data processing time can be effectively reduced, and the accuracy of the data is ensured.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A data processing method, applied to a first server having a cache, the first server being a master server of a cluster, the cluster further comprising N slave servers, N being ≡2, the cluster being connected to a client via an intermediary component, the first server having a voting timer for electing the master server of the cluster, the method comprising:
judging whether the waiting time of the voting timer is larger than or equal to preset time; the waiting time is the difference value between the current time of the first server and the starting time of the voting timer;
if yes, sending request voting information to the other N slave servers, so that the other N slave servers do not send voting requests within the election timeout time; the election timeout time is the interval time between the time of sending the voting request by any one of the other N slave servers and the current time;
Judging whether at least N/2 responses corresponding to the request voting information sent by the other N slave servers are received within the election timeout time, and if yes, confirming that the first server is the master server of the cluster;
receiving a request of the client forwarded by the intermediate component;
storing the request to a first cache of the first server;
sending the request stored in the first cache to the other N slave servers;
and when receiving at least N/2 storage responses corresponding to the requests sent by the other N slave servers, sending data synchronization success information to the client so that the client confirms that the data synchronization of the round is successful.
2. The method of claim 1, wherein the first server stores a first log having a current sequence number, the method further comprising, prior to the validating that the first server is a primary server of the cluster:
judging whether the current serial number is larger than or equal to all log serial numbers of the other N slave servers;
if yes, executing the step of confirming that the first server is the master server of the cluster.
3. The method of claim 1, wherein the first server has a lock-free queue, the storing the request to a first cache of the first server, comprising:
sending the request to the lock-free queue;
and sending a cache request to the lock-free queue to store the request to the first cache.
4. The method of claim 3, wherein the first server comprises a disk, the storing the request to a first cache of the first server, further comprising:
according to the request, obtaining a snapshot file; the snapshot file comprises a current time point, and the first cache has all the requests;
and storing the snapshot file to the disk.
5. The method of claim 1, wherein the request includes a log identification, and wherein the sending the request stored in the first cache to the other N slave servers comprises:
judging whether the log identification is matched with log sequence information in a second server or not; the second server is any one or more servers in the other N slave servers;
And if so, sending the request to the second server.
6. A data processing method, applied to a second server, the second server being a slave server of a cluster, the cluster further comprising a master server with a cache, the cluster being connected to a client via an intermediary component, the master server having a voting timer for electing the master server of the cluster, the method comprising:
receiving a request sent by the main server; the request is a request sent by the client terminal forwarded by the intermediate component and received by the main server, and the request is stored in a first cache of the main server;
storing the request when the log identifier corresponding to the request is matched with the log serial number of the second server;
transmitting a response corresponding to the request to the main server so that the main server transmits data synchronization success information to the client, and the client confirms that the data synchronization of the round is successful;
the master server is used for electing to be a master server in the cluster by the following modes:
judging whether the waiting time of the voting timer is larger than or equal to preset time; the waiting time is the difference value between the current time of the main server and the starting time of the voting timer; if yes, sending request voting information to other N slave servers in the cluster, so that the other N slave servers do not send voting requests within the election timeout time; the election timeout time is the interval time between the time of sending the voting request by any one of the other N slave servers and the current time; and judging whether at least N/2 responses corresponding to the request voting information sent by the other N slave servers are received within the election timeout time, and if yes, confirming that the master server is the master server of the cluster.
7. The server is characterized in that the server is a master server of a cluster, the cluster also comprises other N slave servers, N is more than or equal to 2, and the cluster is connected with a client through an intermediate component;
the server comprises a processor and a memory storing machine executable instructions executable by the processor to implement the method of any one of claims 1-5.
8. A server, wherein the server is a slave server of a cluster, the cluster further comprises a master server with a cache, and the cluster is connected with a client through an intermediate component;
the server includes a processor and a memory storing machine executable instructions executable by the processor to implement the method of claim 6.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any one of claims 1-5 or the method of claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010043982.8A CN111258822B (en) | 2020-01-15 | 2020-01-15 | Data processing method, server, and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010043982.8A CN111258822B (en) | 2020-01-15 | 2020-01-15 | Data processing method, server, and computer-readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111258822A CN111258822A (en) | 2020-06-09 |
| CN111258822B true CN111258822B (en) | 2024-02-27 |
Family
ID=70952130
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010043982.8A Active CN111258822B (en) | 2020-01-15 | 2020-01-15 | Data processing method, server, and computer-readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111258822B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111953772B (en) * | 2020-08-11 | 2022-11-22 | 北京达佳互联信息技术有限公司 | Request processing method, device, server and storage medium |
| CN112235363B (en) * | 2020-09-28 | 2022-02-01 | 华云数据控股集团有限公司 | Data processing method, device, electronic equipment, storage medium and system |
| CN113297236A (en) * | 2020-11-10 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Method, device and system for electing main node in distributed consistency system |
| CN113783921A (en) * | 2021-01-27 | 2021-12-10 | 北京京东振世信息技术有限公司 | Method and device for creating cache component |
| CN112835936A (en) * | 2021-02-05 | 2021-05-25 | 北京电解智科技有限公司 | Data processing method and device and service cluster |
| CN113032232B (en) * | 2021-03-09 | 2023-06-20 | 广州虎牙科技有限公司 | Correlation method, correlation device and equipment for performance detection of equipment |
| CN113221195A (en) * | 2021-04-23 | 2021-08-06 | 北京达佳互联信息技术有限公司 | Method, device and storage medium for storing business data |
| CN113626011B (en) * | 2021-07-21 | 2024-02-13 | 北京万维之道信息技术有限公司 | PHP architecture-based data request processing method, device and equipment |
| CN114143127B (en) * | 2021-11-29 | 2023-10-20 | 网易（杭州）网络有限公司 | Data broadcasting method and device and electronic equipment |
| CN114461593B (en) * | 2022-04-13 | 2022-07-29 | 云和恩墨(北京)信息技术有限公司 | Log writing method and device, electronic device and storage medium |
| CN114979164B (en) * | 2022-04-14 | 2023-11-17 | 网易（杭州）网络有限公司 | Virtual room distribution method and device and electronic equipment |
| CN114979180B (en) * | 2022-05-24 | 2024-05-17 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
| CN114969072B (en) * | 2022-06-06 | 2023-06-09 | 北京友友天宇系统技术有限公司 | Data transmission method, device and equipment based on state machine and data persistence |
| CN116048424B (en) * | 2023-03-07 | 2023-06-06 | 浪潮电子信息产业股份有限公司 | IO data processing method, device, equipment and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105554143A (en) * | 2015-12-25 | 2016-05-04 | 浪潮(北京)电子信息产业有限公司 | High-availability cache server and data processing method and system thereof |
| CN106878473A (en) * | 2017-04-20 | 2017-06-20 | 腾讯科技（深圳）有限公司 | A kind of message treatment method, server cluster and system |
| CN108322533A (en) * | 2018-01-31 | 2018-07-24 | 广州鼎甲计算机科技有限公司 | Configuration and synchronization method between distributed type assemblies node based on operation log |
| CN109547512A (en) * | 2017-09-22 | 2019-03-29 | 中国移动通信集团浙江有限公司 | A kind of method and device of the distributed Session management based on NoSQL |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077635A1 (en) * | 2006-09-22 | 2008-03-27 | Digital Bazaar, Inc. | Highly Available Clustered Storage Network |
-
2020
- 2020-01-15 CN CN202010043982.8A patent/CN111258822B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105554143A (en) * | 2015-12-25 | 2016-05-04 | 浪潮(北京)电子信息产业有限公司 | High-availability cache server and data processing method and system thereof |
| CN106878473A (en) * | 2017-04-20 | 2017-06-20 | 腾讯科技（深圳）有限公司 | A kind of message treatment method, server cluster and system |
| CN109547512A (en) * | 2017-09-22 | 2019-03-29 | 中国移动通信集团浙江有限公司 | A kind of method and device of the distributed Session management based on NoSQL |
| CN108322533A (en) * | 2018-01-31 | 2018-07-24 | 广州鼎甲计算机科技有限公司 | Configuration and synchronization method between distributed type assemblies node based on operation log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111258822A (en) | 2020-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111258822B (en) | Data processing method, server, and computer-readable storage medium | |
| US11853263B2 (en) | Geographically-distributed file system using coordinated namespace replication over a wide area network | |
| US5941999A (en) | Method and system for achieving high availability in networked computer systems | |
| US8140623B2 (en) | Non-blocking commit protocol systems and methods | |
| JP6556851B2 (en) | Database system, server device, program, and information processing method | |
| US9495381B2 (en) | Geographically-distributed file system using coordinated namespace replication over a wide area network | |
| EP2434729A2 (en) | Method for providing access to data items from a distributed storage system | |
| TW202037138A (en) | Consensus system downtime recovery | |
| CN111368002A (en) | Data processing method, system, computer equipment and storage medium | |
| US20080077635A1 (en) | Highly Available Clustered Storage Network | |
| CN107832138B (en) | Method for realizing flattened high-availability namenode model | |
| CN105830033B (en) | System and method for supporting persistent storage versioning and integrity in a distributed data grid | |
| CN112955874A (en) | System and method for self-healing in decentralized model building using machine learning of blockchains | |
| US8533525B2 (en) | Data management apparatus, monitoring apparatus, replica apparatus, cluster system, control method and computer-readable medium | |
| TW202037122A (en) | Consensus system downtime recovery | |
| JP5292351B2 (en) | Message queue management system, lock server, message queue management method, and message queue management program | |
| EP3702936A1 (en) | Message recovery system for computing nodes with replicated databases | |
| CN110635941A (en) | Database node cluster fault migration method and device | |
| CN113965578A (en) | Method, device, equipment and storage medium for electing master node in cluster | |
| CN114244859B (en) | Data processing method and device and electronic equipment | |
| CN113326251B (en) | Data management method, system, device and storage medium | |
| CN106951443B (en) | Method, equipment and system for synchronizing copies based on distributed system | |
| CN111291063B (en) | Master and backup copy election method, system, computer equipment and storage medium | |
| CN116232893A (en) | Consensus method and device of distributed system, electronic equipment and storage medium | |
| CN116132530A (en) | Method for realizing MQTT Broker server by applying Raft algorithm based on Netty framework |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |