CN111813550A

CN111813550A - Data processing method, device, server and storage medium

Info

Publication number: CN111813550A
Application number: CN202010651605.2A
Authority: CN
Inventors: 谢亚晋
Original assignee: Shenzhen Tencent Domain Computer Network Co Ltd
Current assignee: Shenzhen Tencent Domain Computer Network Co Ltd
Priority date: 2020-07-08
Filing date: 2020-07-08
Publication date: 2020-10-23

Abstract

The application discloses a data processing method, a device, a server and a storage medium, the scheme of the application is applied to a business service platform, the business service platform runs a master process and at least one slave process, and the method comprises the following steps: acquiring a data processing request sent by a client; in the case that the data processing request comprises a read request, determining a first data identification of data requested to be read by the read request; if the first data identifier belongs to a data identifier in an unsynchronized record of the cache, forwarding the reading request to the main process for processing, wherein the unsynchronized record records a data identifier corresponding to data which is not synchronized from the main process to the slave process; if the first data identification does not belong to the data identification in the unsynchronized record, the read request is forwarded to the slave process for processing. The scheme of the application can improve the data request processing efficiency and simultaneously reduce the data reading error or abnormal condition caused by the inconsistency of the master process data and the slave process data.

Description

Data processing method, device, server and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method, an apparatus, a server, and a storage medium.

Background

In a service scenario involving a large amount of data and having a high requirement on data read-write performance, a service platform generally caches data and processes a data read-write request through a process, so as to implement relatively efficient data read-write processing.

At present, there are two common ways for a service platform to process a data read-write request through a process: one is a single process mode, and the other is a master-slave process mode. The single-process mode refers to that a read request and a write request are processed simultaneously through a single process. The single-process mode has the advantages that data synchronization is not needed, the situation of data inconsistency does not exist, and the client request delay is long due to the large read-write pressure of the single process. In the master-slave process mode, the business service platform runs a master process and at least one slave process. The main process is responsible for processing the write request of the client and synchronizing the data to the slave process, and on the basis, the slave process is responsible for processing the read request of the client, so that the data read-write pressure of the main process can be reduced, the data read-write efficiency can be improved, and the request processing delay can be reduced. However, in the master-slave process mode, data may not be consistent between the master process and the slave process due to the fact that the master process synchronizes data to the slave process in time, and thus data fed back to the client from the slave process is incorrect or data reading is abnormal. It can be seen that how to reduce the data reading error or exception caused by data inconsistency while reducing the request processing latency is a technical problem that needs to be solved urgently by those skilled in the art.

Disclosure of Invention

In view of the above, the present application provides a data processing method, an apparatus, a server, and a storage medium, so as to improve data request processing efficiency, reduce data request processing delay, and reduce data reading errors or exceptions caused by inconsistency between master process data and slave process data.

In order to achieve the purpose, the application provides the following technical scheme:

in one aspect, the present application provides a data processing method applied to a service platform, where the service platform runs a master process and at least one slave process, and the method includes:

acquiring a data processing request sent by a client;

determining a first data identification of data requested to be read by the read request in the case that the data processing request comprises the read request;

if the first data identifier belongs to a data identifier in an unsynchronized record of the cache, forwarding the read request to the master process for processing, wherein the unsynchronized record records a data identifier corresponding to data which is not synchronized from the master process to the slave process;

and if the first data identification does not belong to the data identification in the unsynchronized record, forwarding the read request to the slave process for processing.

In one possible implementation, the method further includes:

in the case that the data processing request comprises a write request, determining a second data identification of data requested to be written by the write request;

and forwarding the write request to the main process for processing, and adding the second data identifier to the unsynchronized record.

In another possible implementation manner, the method further includes:

obtaining a synchronization completion indication fed back by the master process, wherein the synchronization completion indication carries a third data identifier and is used for representing that the master process synchronizes data represented by the third data identifier to the slave process;

deleting the third data identification in the unsynchronized record.

In yet another possible implementation manner, the business service platform runs a plurality of slave processes;

the method further comprises the following steps:

under the condition that the master process is confirmed to have a fault, querying a synchronization sequence number stored in each of the plurality of slave processes, wherein the synchronization sequence number represents the total amount of data synchronized from the master process to the slave processes;

determining a target slave process which obtains the maximum total amount of synchronous data from the master process based on the synchronous sequence numbers stored in the plurality of slave processes respectively;

and determining the target slave process as a master process.

In another aspect, the present application further provides a data processing apparatus applied to a service platform, where the service platform runs a master process and at least one slave process, and the apparatus includes:

the request obtaining unit is used for obtaining a data processing request sent by a client;

a first identification determination unit, configured to determine, in a case where the data processing request includes a read request, a first data identification of data requested to be read by the read request;

a first request processing unit, configured to forward the read request to the master process for processing if the first data identifier belongs to a data identifier in an unsynchronized record of the cache, where a data identifier corresponding to data that has not been synchronized from the master process to the slave process is recorded in the unsynchronized record;

and the second request processing unit is used for forwarding the read request to the slave process for processing if the first data identifier does not belong to the data identifier in the unsynchronized record.

In yet another aspect, the present application further provides a server comprising a memory and a processor;

wherein the memory is used for storing programs;

the processor is configured to execute the program, which when executed is specifically configured to implement the data processing method of any of claims 1 to 6.

In yet another aspect, the present application further provides a storage medium for storing a program, which when executed, is used to implement the data processing method as described in any one of the above.

As can be seen from the above, the service platform of the present application caches an unsynchronized record, where a data identifier corresponding to data that has not been synchronized from the master process to the slave process is stored in the unsynchronized record. Based on this, after receiving a read request of a client, the application only forwards the read request to a slave process for processing if the unsynchronized record does not include a data identifier requested by the read request, that is, if it is determined that the data requested by the read request is synchronized to the slave process by the master process; if the unsynchronized records comprise the data identification requested by the read request, the read request is still forwarded to the main process for processing, so that on the premise of reducing the number of the read requests processed by the main process to the maximum extent, data reading errors or exceptions caused by data inconsistency between the main process and the slave process are reduced, and the situations of the data reading errors or exceptions are reduced while the request processing efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on the provided drawings without creative efforts.

FIG. 1 is a schematic diagram illustrating the architecture of a scenario to which the present application is applicable;

FIG. 2 is a schematic flow chart diagram illustrating one embodiment of a data processing method of the present application;

FIG. 3 is a schematic diagram of a framework for implementing the data processing method of the present application;

FIG. 4 shows a flowchart interaction diagram of yet another embodiment of the data processing method of the present application;

FIG. 5 is a flow chart illustrating a routing process selecting a master process from a plurality of slave processes according to the present application;

FIG. 6 illustrates an example diagram of a routing process selecting a host process;

FIG. 7 is a schematic diagram showing a component structure of the data processing apparatus of the present application;

fig. 8 shows a schematic diagram of a component architecture of the server of the present application.

Detailed Description

The data processing method is suitable for service platforms in various service scenes. For example, when the service scenario is a game service scenario, the service platform is a game service platform, and accordingly, the game client may request the game service platform to read or write data related to the game. For another example, in a scenario where the service scenario is a financial transaction service, the service platform is a transaction service platform, and in such a scenario, the client may request to read or write data from or to the service platform when performing a transaction such as payment or transfer. Of course, the service scenario applicable to the present application may have other possibilities, which is not limited to this.

In the application, the service platform may be an independent server, or may be a cloud platform, a server cluster, or a distributed system, which is formed by a plurality of servers.

In the embodiment of the application, a master process for processing a read or write request and at least one slave process for assisting the master process in processing the read request are run on the business service platform.

In the embodiment of the application, after receiving a data processing request of a client, a service platform can reasonably distribute the data processing request to a master process or a slave process according to the type of the data processing request and the synchronization condition of data between the master process and the slave process, so that data reading errors or reading exceptions are reduced while the processing efficiency of the data processing request is ensured.

In a possible implementation manner, the business service platform further runs a routing process, which is a process independent from the master process and the slave process, is responsible for managing the master process and the slave process, and allocates a read request or a write request to the master process and the slave process.

When the service platform comprises a plurality of servers, the routing process, the master process and at least one slave process can run on the same server or different servers.

Optionally, in order to improve disaster tolerance (for example, to avoid that a single server fails to simultaneously affect multiple processes in the master process and the slave process), or in order to ensure the operation performance of each process, the routing process, the master process, and each slave process may be respectively operated on different servers.

For convenience of understanding, a service platform is taken as a cloud platform as an example, and a scenario to which the scheme of the present application is applied is described below, as shown in fig. 1.

As can be seen from fig. 1, the scenario includes: the cloud platform 10, which may include a plurality of cloud servers 101.

The cloud platform is also called a cloud computing platform, and is a network platform constructed based on cloud technology. The Cloud technology (Cloud technology) is a hosting technology for unifying series resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data.

Cloud technology (Cloud technology) is a generic term of network technology, information technology, integration technology, management platform technology, application technology and the like based on Cloud computing business model application, can form a resource pool, is used as required, and is flexible and convenient. Background services of technical network systems require a large amount of computing, storage resources, such as image storage and encoding, etc. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.

Among them, cloud computing (cloud computing) is a computing mode that distributes computing tasks over a resource pool formed by a large number of computers, so that various application systems can acquire computing power, storage space, and information services as needed. The network that provides the resources is referred to as the "cloud". Resources in the "cloud" appear to the user as being infinitely expandable and available at any time, available on demand, expandable at any time, and paid for on-demand.

As a basic capability provider of cloud computing, a cloud computing resource pool (called as an ifas (Infrastructure as a Service) platform for short is established, and multiple types of virtual resources are deployed in the resource pool and are selectively used by external clients.

As shown in fig. 1, a cloud platform 10 runs a master process and at least one slave process. In addition, a routing process other than the master process and the slave process is also operated on the cloud platform. The master process, the routing process and the at least one slave process can be respectively run on different cloud servers of the cloud platform.

At least one client 20 is further included in the scenario in fig. 1, and the client may send a data processing request, which may be a write request or a read request, to the cloud platform.

Correspondingly, the cloud platform responds to the data processing request through the routing process, and determines a process for processing the data processing request from the master process and the slave process.

It can be understood that fig. 1 is illustrated by taking the service platform as the cloud platform, and in practical applications, the service platform is a server cluster or the like, and details are not described herein again.

In conjunction with the above, the image processing method of the present application is described below with reference to a flowchart.

As shown in fig. 2, which shows a flowchart of an embodiment of a data processing method, the method of this embodiment may be applied to a business service platform, for example, may be specifically executed by a routing process of the business service platform. As can be seen from the foregoing, the business service platform runs with a master process and at least one slave process.

The method of the embodiment may include:

s201, obtaining a data processing request sent by a client.

The data processing request may include one or both of a write request and a read request. Wherein the write request is for requesting to write data; the read request is for requesting to read data. It will be appreciated that in most business scenarios, the data processing request is typically one of a write request and a read request.

S202, under the condition that the data processing request comprises a read request, determining a first data identifier of the data requested to be read by the read request.

It will be appreciated that read requests and write requests will have different identification means, and thus, by parsing the data processing request, it can be analyzed whether the data processing request includes a read request.

Wherein the data identification of the data is used for uniquely identifying the data. In different service scenarios, the specific form or composition structure of the data identifier of the unique identifier data may also be different, and may be specifically set as required.

For example, taking a service scenario as a game scenario, one data can be uniquely identified by the data type and number of the data. For example, taking a group game scenario as an example, an organization composed of a plurality of players in a group game may be referred to as a group, and therefore, for game data generated by any group, a unique identifier may be identified by a group data type and a group internal identifier.

In the embodiment of the present application, for convenience of distinction, a data identifier corresponding to data requested by a read request is referred to as a first data identifier.

It can be understood that, in order to enable the service platform to confirm the data requested to be read by the read request, the read request may include or carry related information of a data identifier used for characterizing the data, and therefore, by parsing the read request, the first data identifier used for characterizing the data to be read may be parsed.

S203, if the first data identifier belongs to the data identifier in the cached unsynchronized record, forwarding the read request to the main process for processing.

The unsynchronized record records data identification corresponding to data which are not synchronized from the master process to the slave process.

The particular form of the record data identifier in the unsynchronized record and the unsynchronized record may not be limited. For example, the data identifier may be directly stored in the unsynchronized record; for another example, the unsynchronized record may include records corresponding to different data identifiers, where the record corresponding to each data identifier includes not only the data identifier, but also a time when the client requests to write data corresponding to the data identifier, and may also record other data related to the data identifier as needed.

The service platform may cache the unsynchronized process in a fixed cache space, for example, the service platform may cache the unsynchronized record in a memory space corresponding to the routing process under the condition that the service platform manages the master process and the slave process through the routing process.

There are various ways for the service platform to determine the unsynchronized records, and for convenience of understanding, the description is made in several possible cases.

In one possible scenario, the business service platform may query the slave master process for data identifications of data that have not been synchronized to the slave process on a polling cycle or sporadic basis. Correspondingly, the business service platform can record the data identifier corresponding to the data which is not synchronized from the master process to the slave process into the unsynchronized record; or updating the unsynchronized record based on the queried data identification.

However, in this case, the service platform needs to continuously query the master process to synchronize data with the slave process, which may increase the occupation of bandwidth resources between the service platform and the master process and may also occupy data processing resources of the master process.

In order to reduce the bandwidth occupation and avoid occupying the data processing resources of the host process, in yet another possible case, the service platform may determine, in the case of receiving a write request, a second data identifier of data requested to be written by the write request, and add the second data identifier to the unsynchronized record. Of course, the business service platform will also forward the write request to the host process for processing while or after adding the second data identifier to the unsynchronized record.

It can be understood that, when the service platform receives the write request, since the data requested to be written by the write request has not been written by the master process, the second data identifies that the corresponding data has not been synchronized from the master process to the slave process naturally. Therefore, after the business service platform receives the write request, the data requested to be written by the write request can be determined as data which is not synchronized from the main process to the auxiliary process, and the second data identifier in the write request is stored in the unsynchronized record, so that the data identifier corresponding to the data which is not synchronized is not required to be inquired from the main process.

The above two ways of generating unsynchronized records by the service platform are taken as examples, and the way of determining the data identifier corresponding to the data which is not synchronized yet is also applicable to the embodiment, which is not limited to this.

It can be understood that, in order to ensure that the data that has not been synchronized by the master process to the slave process is consistent with the data represented by the data identifier recorded in the unsynchronized record cached by the service platform, the service platform may further continuously update the unsynchronized record to delete the data identifier in the unsynchronized record to complete synchronization.

For example, in one possible implementation manner, the service platform may periodically send a synchronized information request to the master process, where the synchronized information request is used to request the master process to feed back a data identifier corresponding to data that has been synchronized to the slave process in the current period. After the master process successfully synchronizes data to the slave process, the master process stores a data identifier corresponding to the synchronized data. Correspondingly, after the host process receives the synchronized information request, the host process sends the information of the data identifier corresponding to the synchronized data in the latest period to the service platform. On the basis, the business service platform deletes the corresponding data identifier in the unsynchronized record based on the information of the data identifier fed back by the main process.

In this implementation manner, the service platform actually updates the unsynchronized records periodically, and therefore, there may be a case where the unsynchronized records are not updated timely, so that some data is synchronized to the slave process by the master process, but the unsynchronized records of the service platform still indicate that the data is not synchronized to the slave process, and further, the read request for the data is forwarded to the master process for processing, so that the number of read requests processed by the master process is relatively large.

In order to reduce the number of read requests processed by the master process to the maximum extent and reduce the pressure of the master process to the maximum extent so as to further improve the processing efficiency of data processing requests and reduce the time delay of data processing requests of the client, the method and the device can also actively report the data identifier corresponding to the synchronized data to the service platform after the master process synchronizes the data to the slave process.

Specifically, the service platform may obtain a synchronization completion indication fed back by the host process. And the synchronization completion indication is used for representing that the master process synchronizes the data represented by the third data identifier to the slave process.

Wherein the synchronization completion indication is generated after the master process confirms that the data identified by the third data has been synchronized to the slave process. It is to be understood that, for the sake of distinction, the data identifier corresponding to the data that the master process has synchronized to the slave process is referred to as a third data identifier. Accordingly, the service platform may delete the third data identifier in the unsynchronized record based on the synchronization completion indication.

In the second way of updating the unsynchronized records, the master process notifies the service platform of the data synchronized to the slave process each time, so that the data determined based on the unsynchronized records and not synchronized yet is consistent with the data synchronized to the slave process in the master process, and therefore, the read requests corresponding to all the synchronized data can be favorably forwarded to the slave process for processing, the number of the read requests required to be processed by the master process is reduced to the greatest extent, and the data processing pressure of the master process is reduced.

It can be understood that, if the data identifier of a certain data exists in the unsynchronized record, it indicates that the data is not synchronized from the master process to the slave process, that is, the data is only cached in the memory space corresponding to the master process. Therefore, under the condition that the first data identifier carried in the read request belongs to the data identifier in the unsynchronized record, in order to avoid data read-write errors or abnormalities caused by inconsistency of data represented by the first data identifier in the master process and the slave process, the read request is allocated to the master process for processing.

S204, if the first data identification does not belong to the data identification in the unsynchronized record, the read request is forwarded to the slave process for processing.

It will be appreciated that if the first data identity of the data requested to be read by the read request does not belong to an unsynchronised record, it may be determined that the data has been synchronised from the master process to the slave process and therefore the read request may be sent to the slave process for processing in order to avoid increasing the stress on the master process. Moreover, since the data in the master process and the slave process are synchronized, data reading errors or exceptions caused by inconsistency of the data in the master process and the slave process can be avoided.

It will be appreciated that in the case where the business service platform runs with a plurality of slave processes, the business service platform may select one slave process from the plurality of slave processes and forward the read request to the selected slave process. There are various ways for the service server platform to select the slave process for processing the read request, for example, the slave process with a smaller load may be selected to process the read request based on the load condition of the slave process, which is not limited herein.

Therefore, the service platform of the application caches the unsynchronized record, and the unsynchronized record stores the data identifier corresponding to the data which is not synchronized from the master process to the slave process. Based on this, after receiving a read request of a client, the application only forwards the read request to a slave process for processing if the unsynchronized record does not include a data identifier requested by the read request, that is, if it is determined that the data requested by the read request is synchronized to the slave process by the master process; if the unsynchronized records comprise the data identification requested by the read request, the read request is still forwarded to the main process for processing, so that on the premise of reducing the number of the read requests processed by the main process to the maximum extent, data reading errors or exceptions caused by data inconsistency between the main process and the slave process are reduced, and the situations of the data reading errors or exceptions are reduced while the request processing efficiency is improved.

For convenience of understanding, the following description takes an example that the service platform runs with a routing process, and schedules data processing requests to the master process and the slave process through the routing process.

As shown in fig. 3, it shows a schematic frame diagram of one implementation principle of the data processing method of the present application.

In fig. 3 the business service platform 300 runs a routing process 301, a master process 302 and at least one slave process 303.

The unsynchronized records are cached in the routing process as shown in fig. 3.

After the client 310 sends a read request or a write request to the service platform, the routing process of the service platform obtains the read request or the write request sent by the client.

Wherein, for the obtained write request, the routing process 301 may directly send the write request to the host process.

For the obtained read request, the routing process will detect whether the cached unsynchronized record is hit, i.e. whether the data identifier carried by the read request belongs to the data identifier in the unsynchronized record. Correspondingly, the routing process sends the read request which misses the unsynchronized record to the slave process; if the read request hits the unsynchronized record, the read request is sent to the host process for processing.

Based on the schematic framework diagram of fig. 3, the data processing method of the present application is described in detail with reference to fig. 4, wherein the flow of interaction among the slave routing process, the master process, and the slave process is described in detail below.

The data processing method as shown in fig. 4 may include:

s401, the client sends a read request to the service platform, so that the routing process obtains the read request.

It is understood that, in this embodiment, the data processing request sent by the client to the service platform may be one of a read request and a write request. If the data processing request sent by the client to the service platform includes a read request and a write request, the routing process may separate the two requests and perform corresponding processing for the read request and the write request, respectively.

S402, the routing process obtains the first data identification carried by the reading request.

Wherein the first data identification is used to characterize the data requested to be read by the read request.

And S403, if the routing process confirms that the first data identifier belongs to the data identifier in the cached unsynchronized record, the routing process forwards the read request to the main process for processing.

It is understood that, after receiving the read request, the host process may query the cached service data for corresponding data based on the read request, and then return the queried data to the client. There is no limitation to the specific implementation of the host process returning the corresponding data to the client based on the read request, and this part of the content is not shown in fig. 4 either.

S404, if the routing process confirms that the first data identification does not belong to the data identification in the cached unsynchronized record, the routing process selects a slave process for processing the read request from at least one slave process, and forwards the read request to the selected slave process.

Similar to the processing of the read request by the master process, since the slave process also caches each data synchronized by the master process, the slave process can respond to the read request, query the data corresponding to the read request from the cached data, and return the queried data to the client.

S405, the client sends a write request to the service platform, so that the routing process obtains the write request sent by the client.

S406, the routing process confirms the second data identification of the data requested to be written by the write request and adds the second data identification to the cached unsynchronized record.

It will be appreciated that the data corresponding to the write request has not been written by the master process before the write request is forwarded to the master process, and therefore, the data does not naturally belong to data that has been synchronized to the slave process, and therefore, the routing process may determine that the data is data that has not been synchronized to the slave process by the master process, and thus store the second data identification of the data in the unsynchronized record.

Alternatively, the second data identifier is added to the unsynchronized record, and a record corresponding to the second data identifier may be generated. For example, the second data identifier is stored in the record corresponding to the second data identifier. And other data related to the second data identifier can be stored in the record corresponding to the second data identifier.

It is to be understood that, in any of the above embodiments of the present application, in consideration that the space for caching the unsynchronized record in the business service platform is relatively limited, for example, the memory space allocated to the routing process by the business service platform is limited, so that if the number of data processing requests processed by the business service platform is relatively large, the number of data identifiers that may need to be stored in the unsynchronized record is also relatively large.

Under the condition, under the condition that the service platform confirms that the cache space is insufficient, the data identification which is not used for a long time in the non-synchronous record can be deleted. The data identifier which is not used for a long time in the non-synchronous record refers to the data identifier of data which is not requested to be read in a relatively long time.

Specifically, after the routing process obtains the read request for the second data identifier (i.e., the read request requesting to read the data corresponding to the second data identifier), if the second data identifier belongs to the unsynchronized record, the routing process further determines the time when the read request is obtained as the access time of the second data identifier, and stores the access time in the record corresponding to the second data identifier.

When the access time is stored in the record of the second data identifier, the routing process updates the access time in the record of the second data identifier by using the access time determined last time, so that the last access time of the second data identifier is stored in the record of the second data identifier.

Correspondingly, if the routing process confirms that the cache space is insufficient, the record of at least one data identifier with longer access time from the current time in the unsynchronized records can be deleted. For example, the record of each data identifier whose access time is longer than the set time from the current time is deleted, or the record of one or more data identifiers whose access time is the longest from the current time is deleted.

S407, the routing process forwards the write request to the main process for processing.

It should be noted that, the order of the two steps of adding the second data identifier to the cached unsynchronized record and forwarding the write request to the host process is not limited, for example, the two steps may also be executed simultaneously.

S408, the main process executes data writing operation based on the writing request and caches the writing operation result.

And the main process responds to the write request to execute the write operation, so that the data corresponding to the write operation is cached.

S409, the master process sends at least one piece of written data which is not synchronized to each slave process.

The specific manner of sending the data to be synchronized from the master process to the slave process may not be limited. For example, after the write operation corresponding to the write request is executed, the master process may send the written data to each slave process to synchronize the data to the slave process. For another example, to avoid occupying too many processing resources of the master process due to frequent data synchronization, the master process may also periodically (e.g., according to a set time interval or period) send a data synchronization indication to the master process, where the data synchronization indication may carry all data that the master process has written into the cache space of the master process but is not synchronized to the slave process. Of course, the master process may also have other ways of synchronizing data with the slave process, which are not described herein again.

S410, after the master process confirms that the at least one data is successfully synchronized to the slave process, the master process confirms the third data identifier of the at least one data which is successfully synchronized, and feeds back a synchronization completion indication corresponding to the at least one data to the routing process.

There may be various ways for the master process to determine whether the data is successfully synchronized with the slave process. For example, after the slave process completes data synchronization based on the data sent by the master process, a success indication may be fed back to the master process, where the success indication represents that the slave process completes synchronization of the corresponding data. For example, the success indication carries the data identification of the data with which the synchronization is successful. Of course, the master process may actively query the slave process for the synchronization status to detect whether the data synchronization is successful.

S411, the routing process deletes the at least one third data identifier in the unsynchronized record based on the at least one third data identifier carried by the synchronization completion indication.

After obtaining the synchronization completion indication fed back by the main process, the routing process may determine that the synchronization completion indication carries data synchronization completed by the data represented by the third data identifier, and on this basis, may delete the third data identifier from the unsynchronized record. Correspondingly, after subsequently receiving the read request carrying the third data identifier, the read request can be forwarded to the slave process for processing.

It can be understood that, if a main process in the service platform fails, the processing of the service platform on the read-write request of the client may be affected, and therefore, after it is determined that the main process fails, a corresponding recovery mechanism is further required. However, if the main process is restarted directly, the complexity is high and the time is long. In order to solve the problem, after the business service platform determines that the main process fails, one slave process is reselected from a plurality of slave processes to serve as the main process.

As shown in fig. 5, which shows a schematic flow diagram of selecting a master process from a plurality of slave processes by a routing process according to the present application, this embodiment may include:

s501, when the heartbeat packet sent by the master process is not detected after the set time length is exceeded, the routing process inquires the synchronization serial numbers stored in the slave processes.

The main process may send the heartbeat packet to the routing process according to a set first heartbeat cycle. Correspondingly, the slave process may send the heartbeat packet to the routing process according to the set second heartbeat cycle. The first heartbeat cycle and the second heartbeat cycle may be the same or different.

For example, the master process or the slave process may send a heartbeat packet to the service platform according to a heartbeat cycle corresponding to each master process or slave process when a heartbeat sending time is reached.

It can be understood that, because the master process and the slave process both send heartbeat packets to the routing process periodically, the routing process can analyze whether the master process is abnormal or not based on the condition of the heartbeat packets sent by the master process; correspondingly, the routing process can also analyze whether the slave process has an exception or not based on the condition of the heartbeat packet sent by the slave process.

And if the routing process does not receive the heartbeat packet of the main process after the set time length is exceeded, the routing process confirms that the main process has a fault.

It can be understood that the present embodiment analyzes whether the main process has a failure based on the condition that the main process sends the heartbeat packet to the routing process. However, it is understood that in practical applications, the routing process may also find the failure of the main process in other manners, for example, the routing process may send a detection packet to the main process periodically, and if no feedback of the main process for the detection packet is received after a set time length is exceeded, or if a plurality of consecutive detection reports are all received, it is determined that the main process has the failure.

Wherein, the synchronization sequence number represents the total amount of data synchronized from the master process to the slave process. The total number of the data synchronized by the master process to the slave processes may also be considered as the total number of rounds of the data synchronized by the master process to the multiple slave processes. And each round of data synchronization corresponds to a round of data synchronization indication initiated by the master process to a plurality of slave processes aiming at one data to be synchronized.

In one possible implementation manner, before the master process synchronizes data to the slave process, the synchronization sequence number is updated each time, so that the total amount of data synchronized by the master process to the slave process, which is characterized by the synchronization sequence number, is increased. For example, the synchronization sequence number may be the amount of data synchronized by the master process to the slave process, and each time the synchronization sequence number is updated, the amount is incremented by one. Accordingly, the master process can send the data to be synchronized and the latest updated synchronization sequence number one to each slave process at a time, so that the slave processes store the synchronization sequence numbers.

In another possible implementation manner, the master process updates the synchronization sequence number each time the write request is received, so that the total amount of data synchronized from the master process to the slave process, which is characterized by the synchronization sequence number, is increased. Correspondingly, when the master process synchronizes data to the slave processes, the data to be synchronized and the currently determined synchronization sequence number one can be sent to each slave process.

For example, after the master process receives the write request for the first time, the synchronization sequence number may be set to "0001", and then after the master process completes the write operation of the write request, the master process may send the data corresponding to the write operation and the synchronization sequence number "0001" to each slave process together. After receiving the write request for the second time, the synchronization sequence number is changed to "0002", so that after the master process completes the write operation of the write request received for the second time, the data is synchronized to the slave processes, and the synchronization sequence number "0002" is also sent to the processes.

It can be understood that, since the synchronization sequence number may reflect the total amount of data synchronized by the master process to the slave processes, if a failure occurs in the master process and causes part of the data in the master process to be synchronized only to part of the slave processes, it may be determined which data synchronized in the slave processes is relatively complete according to the synchronization sequence number stored in the slave processes.

For example, if the above master process carries the synchronization sequence number "0002" while sending the synchronization data to each synchronization process for the second time, the synchronization sequence number stored in the slave process should be "0002" if the slave process has received the data synchronized for the second time from the master process. However, if the synchronization sequence number stored by the slave process is "0001", it means that the slave process does not complete the synchronization of the data for the second time, and obtains the data synchronized for the first time by the master process.

S502, based on the synchronization sequence numbers stored in the multiple slave processes, determining the target slave process which obtains the maximum total amount of synchronization data from the master process.

The target slave process refers to the slave process with the largest total number, which is characterized by the synchronization sequence numbers stored in the plurality of slave processes. It will be appreciated that in the case of at least two slave processes having the highest total number of characterising synchronization data, one slave process may be randomly selected as the target slave process; or one target slave process can be selected from the at least two slave processes according to the load condition.

For example, in the above example, the slave process having the stored synchronization sequence number "0002" obtains more total amount of synchronization data from the master process than the slave process having the stored synchronization sequence number "0001".

S503, the target slave process is determined as the master process.

As can be seen from the above description, the data in the target slave process has the highest consistency with the data in the master process, and therefore, determining a new master process by the target slave process is beneficial to reducing data reading exceptions caused by data inconsistency.

The operation that needs to be executed by determining the target slave process as the master process may include: storing the information that the target slave process is the master process; the method can also comprise the following steps: and broadcasting the information of the target slave process as the master process to the target slave process and other slave processes.

It will be appreciated that after the target slave process becomes the new master process, the target slave process may synchronize its stored data with other slave processes to ensure data consistency.

For convenience of understanding, the total amount of data synchronized from the master process to the slave processes by using the synchronization sequence number is taken as an example for explanation, for example, see fig. 6, which shows a schematic diagram of a routing process selecting a master process from a plurality of slave processes.

When the master process goes down in fig. 6, the synchronization sequence number stored in the master process is 112, which indicates that the master process has synchronized 112 data to the slave processes (or initiated 112 rounds of data synchronization), but because the master process goes down in the process of synchronizing the 112 th data, only part of the slave processes complete the synchronization of the 112 th data.

As shown in fig. 6, if the synchronization sequence number in the process 1 is 112, it indicates that the process 1 has obtained the 112 th data synchronized by the master process to the slave process 1, and the synchronization requirement in the processes 2 and 3 is still 111, so that the process 20 indicates that the consistency of the data between the slave process 1 and the failed master process is the highest due to the maximum synchronization sequence number stored in the slave process 1, and thus, the slave process can be determined as a new master process.

Accordingly, the new master process can synchronize the data of the new master process to other slave processes, and the data consistency between each master process and the new master process is guaranteed.

The application also provides a data processing device corresponding to the data processing method.

As shown in fig. 7, which illustrates a schematic structural diagram of an embodiment of a data processing apparatus according to the present application, the apparatus of the present embodiment is applied to a business service platform, the business service platform runs with a master process and at least one slave process, and the apparatus includes:

a request obtaining unit 701, configured to obtain a data processing request sent by a client;

a first identifier determining unit 702, configured to determine, in a case where the data processing request includes a read request, a first data identifier of data requested to be read by the read request;

a first request processing unit 703, configured to forward the read request to the master process for processing if the first data identifier belongs to a data identifier in an unsynchronized record of the cache, where a data identifier corresponding to data that has not been synchronized from the master process to the slave process is recorded in the unsynchronized record;

a second request processing unit 704, configured to forward the read request to the slave process for processing if the first data identifier does not belong to the data identifier in the unsynchronized record.

In one possible implementation, the apparatus may further include:

a second identification determination unit configured to determine, in a case where the data processing request includes a write request, a second data identification of data requested to be written by the write request;

and the third request processing unit is used for forwarding the write request to the main process for processing and adding the second data identifier to the unsynchronized record.

Optionally, the apparatus further comprises:

an indication obtaining unit, configured to obtain a synchronization completion indication fed back by the master process, where the synchronization completion indication carries a third data identifier, and the synchronization completion indication is used to represent that the master process has synchronized data represented by the third data identifier to the slave process;

and the identification deleting unit is used for deleting the third data identification in the unsynchronized record.

Optionally, the third request processing unit is specifically configured to add the record of the second data identifier to the unsynchronized record when the second data identifier is added to the unsynchronized record;

correspondingly, the device can further comprise:

the time recording unit is used for determining the time of obtaining the read request as the access time of the second data identifier after obtaining the read request aiming at the second data identifier, and storing the access time of the second data identifier into the record of the second data identifier;

and the record deleting unit is used for deleting at least one record of the data identifier, of which the access time is longer than the current time, in the unsynchronized record based on the adding time of each data identifier in the unsynchronized record under the condition that the cache space is insufficient.

In one possible implementation manner, the business service platform runs with a plurality of slave processes;

the apparatus of the present application may further include:

the sequence number query unit is used for querying the synchronization sequence numbers stored by the plurality of slave processes respectively under the condition that the master process is confirmed to have a fault, wherein the synchronization sequence numbers represent the total number of data synchronized from the master process to the slave processes;

the process determining unit is used for determining a target slave process which obtains the maximum total amount of synchronous data from the master process based on the synchronous sequence numbers stored in the plurality of slave processes;

and the master process replacing unit is used for determining the target slave process as the master process.

Optionally, when the sequence number query unit determines that the main process fails, the method specifically includes: and under the condition that the heartbeat packet sent by the main process is not detected after the set duration, confirming that the main process has a fault, wherein the main process sends the heartbeat packet to the service platform according to the set heartbeat period when the heartbeat sending time is reached.

In yet another aspect, the present application further provides a server, which may be an independent server, a server in a server cluster, or a node in a cloud platform, etc. Fig. 8 is a schematic diagram illustrating a component architecture of a server provided in the present application. In fig. 8, the server 800 may include: a processor 801 and a memory 802.

Optionally, the server may further include: a communication interface 803, an input unit 804, and a display 805 and a communication bus 806.

The processor 801, the memory 802, the communication interface 803, the input unit 804 and the display 805 all communicate with each other via a communication bus 806.

In the embodiment of the present application, the processor 801 may be a central processing unit, an application specific integrated circuit, or the like.

The processor may call a program stored in the memory 802, and in particular, the processor may perform the operations performed on the cloud server side in the above embodiments.

The memory 802 is used for storing one or more programs, which may include program codes including computer operation instructions, and in this embodiment, at least a program for implementing the data processing method in any one of the above embodiments is stored in the memory.

In one possible implementation, the memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, the above-mentioned programs, and application programs required for the functions of image playing, etc.; the storage data area may store data created according to the use of the server.

The communication interface 803 may be an interface of a communication module.

The present application may further include an input unit 804, which may include a touch sensing unit, a keyboard, and the like.

The display 805 includes a display panel, such as a touch display panel or the like.

Of course, the server structure shown in fig. 8 does not constitute a limitation to the server in the embodiment of the present application, and in practical applications, the server may include more or less components than those shown in fig. 8, or some components in combination.

In another aspect, the present application further provides a storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are loaded and executed by a processor, the data processing method in any one of the above embodiments is implemented.

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in any one of the above data processing method embodiments.

It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the device-like embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that it is obvious to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and these modifications and improvements should also be considered as the protection scope of the present invention.

Claims

1. A data processing method is applied to a business service platform, wherein the business service platform runs a master process and at least one slave process, and the method comprises the following steps:

acquiring a data processing request sent by a client;

2. The method of claim 1, further comprising:

3. The method of claim 2, further comprising:

deleting the third data identification in the unsynchronized record.

4. The method of claim 2, wherein adding the second data identification to the unsynchronized record comprises:

adding the record identified by the second data in the unsynchronized record;

the method further comprises the following steps:

after a read request aiming at the second data identifier is obtained, determining the moment of obtaining the read request as the access moment of the second data identifier, and storing the access moment of the second data identifier into a record of the second data identifier;

and under the condition that the cache space is insufficient, deleting at least one record of the data identifier, which has a longer time length from the access time to the current time, in the unsynchronized record based on the adding time of each data identifier in the unsynchronized record.

5. The method of claim 1, wherein the business service platform runs a plurality of slave processes;

the method further comprises the following steps:

and determining the target slave process as a master process.

6. The method of claim 5, wherein the confirming the failure of the host process comprises:

and under the condition that the heartbeat packet sent by the main process is not detected after the set duration is exceeded, confirming that the main process has a fault, wherein the main process sends the heartbeat packet to the service platform according to the set heartbeat period when the heartbeat sending time is reached.

7. A data processing apparatus, applied to a business service platform, where the business service platform runs a master process and at least one slave process, the apparatus comprising:

8. The apparatus of claim 7, further comprising:

9. A server, comprising a memory and a processor;

wherein the memory is used for storing programs;

10. A storage medium storing a program for implementing the data processing method according to any one of claims 1 to 6 when executed.