CN111897880B

CN111897880B - Database synchronization method and device, terminal equipment and storage medium

Info

Publication number: CN111897880B
Application number: CN202011058753.XA
Authority: CN
Inventors: 严石伟; 丁凯; 蒋楠
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Cloud Computing Beijing Co Ltd
Priority date: 2020-09-30
Filing date: 2020-09-30
Publication date: 2020-12-29
Anticipated expiration: 2040-09-30
Also published as: CN111897880A

Abstract

The embodiment of the application discloses a database synchronization method, a database synchronization device, terminal equipment and a storage medium, wherein the method comprises the following steps: the retrieval main node receives a data synchronization request sent by the retrieval slave node, wherein the data synchronization request comprises a retrieval service identifier and a synchronization data identifier; when the retrieval main node determines that the retrieval service identifier is the same as the service identifier of the retrieval service currently operated by the retrieval main node and the synchronous data identifier is contained in the synchronous cache queue, initiating partial synchronization of a main retrieval database of the retrieval main node and a slave retrieval database of the retrieval slave node; the synchronous cache queue stores a plurality of write requests, one write request corresponds to one synchronous data identifier, and any synchronous data identifier is used for identifying the total data volume written into the main retrieval database after the write request corresponding to any synchronous data identifier is executed. By adopting the embodiment of the application, the data backup speed of the master-slave retrieval database can be improved, and the occupation of machine resources can be reduced.

Description

Database synchronization method and device, terminal equipment and storage medium

Technical Field

The present application relates to the field of data processing, and in particular, to a database synchronization method, apparatus, terminal device, and storage medium.

Background

Currently, in order to ensure the retrieval efficiency, the retrieval service is usually deployed on a machine with richer computing resources, such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Video Processing Unit (VPU), and the like, however, when there is a highly concurrent retrieval request, a great pressure is brought to the machine where the retrieval service is located. The search system is directly rendered unusable as soon as a problem arises with the search service or the machine on which it is located fails. The real-time retrieval scenes such as passenger flow, reminding and the like are disastrous results, and even if the non-real-time retrieval scenes such as identity filing and the like are used, if the retrieval service cannot be quickly recovered, the filing failure can be directly caused. Therefore, the technical personnel in the field provide a scheme capable of distributing the retrieval service to the retrieval slave nodes so as to realize the read-write separation of retrieval and greatly reduce the pressure of the retrieval master node on processing the request. In order to implement the above scheme, it is one of the basic preconditions to ensure data synchronization between the retrieval slave node and the retrieval master node, and therefore, how to increase the data backup speed of the master-slave retrieval database and reduce the occupation of machine resources becomes one of the problems to be solved urgently at present.

Disclosure of Invention

The embodiment of the application provides a database synchronization method, a database synchronization device, terminal equipment and a storage medium, which can improve the data backup speed of a master-slave retrieval database and reduce the occupation of machine resources.

In a first aspect, an embodiment of the present application provides a database synchronization method, where the method includes:

the retrieval main node receives a data synchronization request sent by a retrieval slave node, wherein the data synchronization request comprises a retrieval service identifier and a synchronization data identifier;

when the retrieval main node determines that the retrieval service identifier is the same as the service identifier of the retrieval service currently operated by the retrieval main node and the synchronous data identifier is contained in a synchronous cache queue, initiating partial synchronization of a main retrieval database of the retrieval main node and a slave retrieval database of the retrieval slave node;

the synchronous cache queue stores a plurality of write requests, one write request corresponds to one synchronous data identifier, and any synchronous data identifier is used for identifying the total data volume written into the main retrieval database after the write request corresponding to any synchronous data identifier is executed.

With reference to the first aspect, in a possible implementation manner, the initiating partial synchronization between the master search database of the search master node and the slave search database of the search slave node includes:

and determining all write class requests stored after the write class request corresponding to the synchronous data identifier from the synchronous cache queue, and sending the all write class requests to the retrieval slave node to trigger the retrieval slave node to execute the all write class requests to update the data stored in the retrieval database and the synchronous data identifier in the retrieval slave node.

With reference to the first aspect, in one possible implementation, the method further includes:

when the retrieval main node determines that the retrieval service identification is different from the service identification of the retrieval service currently operated by the retrieval main node, initiating complete synchronization of the main retrieval database and the auxiliary retrieval database; or

When the retrieval master node determines that the synchronization data identifier is not included in the synchronization buffer queue, full synchronization of the master retrieval database and the slave retrieval database is initiated.

With reference to the first aspect, in one possible implementation manner, the initiating full synchronization of the master search database and the slave search database includes:

the retrieval main node sends a binary log to the retrieval slave node to trigger the retrieval slave node to acquire a write request recorded in the binary log by deserializing the binary log and update the data stored in the slave retrieval database and the synchronous data identifier by executing the write request.

the retrieval master node persists the master retrieval database snapshot at the current time to a feature file, and sends the feature file and each write-class request received after the current time to the retrieval slave node, so as to trigger the retrieval slave node to update the data stored in the slave retrieval database and the synchronous data identifier by loading the feature file to the slave retrieval database and executing each write-class request.

With reference to the first aspect, in a possible implementation manner, after the initiating of partial synchronization of the master search database of the search master node and the slave search database of the search slave node, or after the initiating of full synchronization of the master search database and the slave search database, the method further includes:

the retrieval master node sends the service identification to the retrieval slave node so that the retrieval slave node updates the retrieval service identification to the service identification.

the retrieval main node receives a client writing request sent by a client;

the retrieval main node executes the client write request to update the data stored in the main retrieval database and the synchronous cache queue;

the retrieval master node sends the client write request to the retrieval slave node to trigger the retrieval slave node to execute the client write request to update the data stored in the slave retrieval database and the synchronous data identifier in the retrieval slave node.

the retrieval main node receives a heartbeat detection request sent by the retrieval slave node, wherein the heartbeat detection request comprises a first synchronization data identifier corresponding to the slave retrieval database;

if the retrieval main node determines that the total data volume identified by the first synchronous data identification is less than the total data volume identified by a second synchronous data identification stored in the synchronous cache queue last and the first synchronous data identification is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database and the auxiliary retrieval database;

and if the retrieval main node determines that the total data volume identified by the first synchronous data identification is less than the total data volume identified by the second synchronous data identification and the first synchronous data identification is not contained in the synchronous cache queue, initiating complete synchronization of the main retrieval database and the auxiliary retrieval database.

In a second aspect, an embodiment of the present application provides a database synchronization apparatus, where the database synchronization apparatus is a search master node, and the apparatus includes:

a data synchronization request acquisition module, configured to receive a data synchronization request sent by a retrieval slave node, where the data synchronization request includes a retrieval service identifier and a synchronization data identifier;

a first synchronization initiating module, configured to initiate partial synchronization between a master search database of the search master node and a slave search database of the search slave node when it is determined that the search service identifier is the same as a service identifier of a search service currently operated by the search master node and the synchronization data identifier is included in a synchronization cache queue;

With reference to the second aspect, in a possible implementation manner, the first synchronization initiating module is configured to:

With reference to the second aspect, in a possible implementation manner, the apparatus includes a second synchronization initiating module, where the second synchronization initiating module includes:

a first complete synchronization determining unit, configured to initiate complete synchronization between the master search database and the slave search database when it is determined that the search service identifier is not identical to a service identifier of a search service currently operated by the search master node; or

A second full synchronization determining unit, configured to initiate full synchronization of the master search database and the slave search database when it is determined that the synchronization data identifier is not included in the synchronization buffer queue.

With reference to the second aspect, in a possible implementation manner, the second synchronization initiating module includes:

and the first full synchronous execution unit is used for sending the binary log to the retrieval slave node so as to trigger the retrieval slave node to acquire a writing request recorded in the binary log by deserializing the binary log and update the data stored in the retrieval database and the synchronous data identifier by executing the writing request.

and a second full-synchronization executing unit, configured to persist the snapshot of the master search database at the current time to a feature file, and send the feature file and each write-class request received after the current time to the search slave node, so as to trigger the search slave node to update the data stored in the slave search database and the synchronization data identifier by loading the feature file to the slave search database and executing each write-class request.

With reference to the second aspect, in a possible implementation manner, the apparatus further includes a service identifier updating module, where the service identifier updating module is configured to:

and sending the service identifier to the retrieval slave node so that the retrieval slave node updates the retrieval service identifier to the service identifier.

With reference to the second aspect, in a possible implementation manner, the apparatus further includes a write class request synchronization module, where the write class request synchronization module includes:

the write request receiving unit is used for receiving a client write request sent by a client;

a main search database updating unit, configured to execute the client write request to update data stored in the main search database and the synchronization buffer queue;

and a write request sending unit, configured to send the client write request to the retrieval slave node, so as to trigger the retrieval slave node to execute the client write request to update the data stored in the slave retrieval database and the synchronization data identifier in the retrieval slave node.

With reference to the second aspect, in a possible implementation manner, the apparatus further includes a heartbeat detection module, where the heartbeat detection module includes:

a heartbeat detection request receiving unit, configured to receive a heartbeat detection request sent by the retrieval slave node, where the heartbeat detection request includes a first synchronization data identifier corresponding to the slave retrieval database;

a first heartbeat detection processing unit, configured to initiate partial synchronization between the master search database and the slave search database if it is determined that a total data amount identified by the first synchronization data identifier is smaller than a total data amount identified by a second synchronization data identifier last stored in the synchronization buffer queue and the first synchronization data identifier is included in the synchronization buffer queue;

and a second heartbeat detection processing unit, configured to initiate full synchronization between the master search database and the slave search database if it is determined that the total data amount identified by the first synchronization data identifier is smaller than the total data amount identified by the second synchronization data identifier and the first synchronization data identifier is not included in the synchronization buffer queue.

In a third aspect, an embodiment of the present application provides a terminal device, where the terminal device includes a processor and a memory, and the processor and the memory are connected to each other. The memory is configured to store a computer program that supports the terminal device to execute the method provided by the first aspect and/or any one of the possible implementation manners of the first aspect, where the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method provided by the first aspect and/or any one of the possible implementation manners of the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, where the computer program includes program instructions, and the program instructions, when executed by a processor, cause the processor to execute the method provided by the first aspect and/or any one of the possible implementation manners of the first aspect.

In the embodiment of the application, the retrieval master node receives a data synchronization request sent by the retrieval slave node, wherein the data synchronization request comprises a retrieval service identifier and a synchronization data identifier. And when the retrieval main node determines that the retrieval service identifier is the same as the service identifier of the retrieval service currently operated by the retrieval main node and the synchronous data identifier is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database of the retrieval main node and the auxiliary retrieval database of the retrieval auxiliary node. The synchronous cache queue stores a plurality of write requests, one write request corresponds to one synchronous data identifier, and any synchronous data identifier is used for identifying the total data volume written into the main retrieval database after the write request corresponding to any synchronous data identifier is executed. By adopting the embodiment of the application, the data backup speed of the master-slave retrieval database can be improved, and the occupation of machine resources can be reduced.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is an architectural diagram of a retrieval system;

FIG. 2 is a schematic diagram of the architecture of a retrieval service master-slave synchronization system;

FIG. 3a is a flow chart of a database synchronization method according to an embodiment of the present disclosure;

FIG. 3b is a schematic diagram of a scenario of updating a synchronization data identifier according to an embodiment of the present application;

fig. 3c is a schematic flowchart of initiating partial synchronization according to an embodiment of the present application;

FIG. 3d is a schematic diagram of a partially synchronized scenario provided by an embodiment of the present application;

FIG. 4a is a schematic flow chart of a database synchronization method according to an embodiment of the present application;

fig. 4b is a schematic flowchart of initiating full synchronization provided by an embodiment of the present application;

FIG. 5a is a schematic flow chart of a database synchronization method according to an embodiment of the present application;

FIG. 5b is a flowchart illustrating synchronization of write class requests according to an embodiment of the present application;

FIG. 6a is a schematic flow chart of a database synchronization method according to an embodiment of the present application;

fig. 6b is a schematic application scenario diagram of a database synchronization method provided in an embodiment of the present application;

fig. 6c is a schematic flow chart of heartbeat detection provided in the embodiment of the present application;

FIG. 7 is a schematic structural diagram of a database synchronization apparatus according to an embodiment of the present application;

FIG. 8 is a schematic diagram of another structure of a database synchronization apparatus according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of a terminal device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The database synchronization method provided by the embodiment of the application can be widely applied to retrieval scenes, for example, the database synchronization method can be applied to various business super scenes such as intelligent retail stores, large stores, department stores, shopping centers and the like, and provides retrieval services such as stable and continuous identity, passenger flow, reminding and the like of customers for the stores and the shopping malls by establishing a more efficient and more available track retrieval system. Referring to fig. 1, fig. 1 is a schematic diagram of an architecture of a retrieval system. As shown in fig. 1, a typical retrieval system mainly includes three major parts, namely, a picture acquisition module, a picture processing module, and a feature retrieval service module. The image acquisition module is used for shooting a customer image in a shop floor based on a camera, the image processing module is used for preprocessing the shot image, detecting the image and extracting the features, and then a retrieval request is generated and sent to the retrieval service module, so that the retrieval service module searches for a face to be retrieved or adds the face features according to the retrieval request. It is understood that the retrieval service is a core part of the retrieval system, and the stable operation of the retrieval service is a precondition for ensuring the normal operation of the whole retrieval system. Generally speaking, the search service may be composed of a feature management module and a search management module, wherein the feature management module mainly performs processing such as adding, deleting and changing feature data in a search database, and the search management module is mainly used for executing query of feature data, checking of identities in a database, and the like. In order to ensure the retrieval efficiency, the retrieval service is generally deployed on a machine with richer computing resources, such as a GPU. However, when there are highly concurrent retrieval requests, it will put a lot of pressure on the machine where the retrieval service is located. The search system is directly rendered unusable as soon as a problem arises with the search service or the machine on which it is located fails. The real-time retrieval scenes such as passenger flow, reminding and the like are disastrous results, and even if the non-real-time retrieval scenes such as identity filing and the like are used, if the retrieval service cannot be quickly recovered, the filing failure can be directly caused. Therefore, the technical personnel in the field provide a scheme capable of distributing the retrieval service to the retrieval slave nodes so as to realize the read-write separation of retrieval and greatly reduce the pressure of the retrieval master node on processing the request. It is to be understood that the master search node and the slave search node in the embodiments of the present application may be various terminal devices. The terminal device includes, but is not limited to, a server, a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like, which is not limited herein.

Referring to fig. 2, fig. 2 is a schematic diagram of a retrieval service master-slave synchronization system. As shown in fig. 2, the retrieval service master-slave synchronization system may include two major modules, a retrieval agent and a retrieval service. The retrieval agent is mainly used for completing load balance of distributing the write requests sent by the client to the retrieval agent to the retrieval main node and the read requests. The retrieval service may be composed of one retrieval master node and a plurality of retrieval slave nodes. The retrieval master node is configured to receive and execute a write-type request that is distributed to the retrieval master node by a retrieval agent to implement feature management, and optionally, the retrieval master node may be further configured to receive and execute a read-type request that is distributed to the retrieval master node by the retrieval agent to implement retrieval management. The retrieval slave node is mainly used for receiving and executing a read request distributed to the retrieval slave node by a retrieval agent so as to realize retrieval management. The write request and the read request may be collectively referred to as a search request. In particular, the write-type request is used to add, and/or delete, and/or modify data (e.g., facial feature pictures) included in the search database. The read-type request is used to implement a data query (e.g., a user identity query). It should be understood that, since the read-type request does not change the state of the retrieval database (i.e., the master retrieval database), the database synchronization method provided in the embodiment of the present application is generally directed to a master-slave retrieval database synchronization scenario in which the retrieval master node receives a write-type request sent by a client, and the state of the master retrieval database in the retrieval master node is changed based on the write-type request. It should be understood that the master-slave search database includes a master search database and a slave search database. The master retrieval database is used for storing the characteristic data in the master retrieval node, and the slave retrieval database is used for storing the characteristic data in the slave retrieval node. Generally, in order to reduce the processing pressure of the highly concurrent retrieval request on the retrieval master node, the retrieval service may be generally distributed to the retrieval slave nodes, that is, the query service (i.e., the retrieval service of the read request) of the user is distributed to the retrieval slave nodes, and therefore, it is one of the basic prerequisites to ensure the data synchronization between the retrieval slave nodes and the retrieval master node. For convenience of description, the following embodiments of the present application will be described by taking an example in which the retrieval service includes one retrieval master node and one retrieval slave node.

According to the method in the embodiment of the application, the retrieval main node receives a data synchronization request sent by the retrieval slave node, and the data synchronization request comprises a retrieval service identifier and a synchronization data identifier. And when the retrieval main node determines that the retrieval service identifier is the same as the service identifier of the retrieval service currently operated by the retrieval main node and the synchronous data identifier is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database of the retrieval main node and the auxiliary retrieval database of the retrieval auxiliary node. The synchronous cache queue stores a plurality of write requests, one write request corresponds to one synchronous data identifier, and any synchronous data identifier is used for identifying the total data volume written into the main retrieval database after the write request corresponding to any synchronous data identifier is executed. By adopting the embodiment of the application, the data backup speed of the master-slave retrieval database can be improved, and the occupation of machine resources can be reduced.

The method and the related apparatus provided by the embodiments of the present application will be described in detail with reference to fig. 3a to 9, respectively. The method provided by the embodiment of the application can comprise data processing stages for receiving a data synchronization request, determining to initiate partial synchronization or full synchronization according to the data synchronization request and the like. The implementation of each data processing stage can be seen in the following implementation shown in fig. 3a to 6 c.

Referring to fig. 3a, fig. 3a is a flow chart of a database synchronization method according to an embodiment of the present disclosure. The method provided by the embodiment of the application can comprise the following steps S101 to S102:

s101, the retrieval main node receives a data synchronization request sent by the retrieval slave node.

In some possible embodiments, the retrieval master node may determine the status of the retrieval slave node from the retrieval database according to the data synchronization request by receiving the data synchronization request sent by the retrieval slave node. The data synchronization request comprises a retrieval service identifier and a synchronization data identifier. It will be understood that each search service will automatically generate a unique identifier (i.e., search service identifier) at service startup to distinguish different running processes. Generally speaking, after the retrieval slave node and the retrieval master node are completely or partially synchronized, the retrieval master node sends the retrieval service identifier corresponding to the currently running retrieval service to the retrieval slave node, so that the retrieval slave node caches the retrieval service identifier, thereby facilitating subsequent use. And the synchronous data identification is used for identifying the total data volume written into the main retrieval database after the write-type request corresponding to the synchronous data identification is executed. Alternatively, or in other words, the synchronization data identification may be used to identify the amount of data that has been successfully synchronized. Generally, the synchronization data identifier may include a synchronization Offset, wherein the retrieval master node and the retrieval slave node may each maintain a synchronization data identifier of successfully synchronized data. For convenience of description, the following embodiments of the present application will be described by taking the identification of synchronization data as a synchronization Offset as an example.

For example, please refer to fig. 3b, and fig. 3b is a schematic view of a scene of updating the synchronization data identifier according to an embodiment of the present application. As shown in fig. 3b, it is assumed that at a certain time, the total data amount in the master search database of the search master node is 1000 bytes (i.e., Offset = 1000), the total data amount in the slave search database 1 of the search slave node 1 is 1000 bytes (i.e., Offset = 1000), and the total data amount in the slave search database 2 of the search slave node 2 is 1000 bytes (i.e., Offset = 1000). Assuming that the retrieving master node receives a 100-byte write class request at another time after the time, the synchronization Offset is increased by 100 (i.e., Offset = 1100) after the retrieving master node completes executing the write class request. Further, after the retrieval slave node receives the write class request of synchronizing 100 bytes sent by the retrieval master node and succeeds in synchronization by executing the write class request, the synchronization Offset of the retrieval slave node can be increased by 100 (i.e., Offset = 1100) in the same way, otherwise, if synchronization fails, the synchronization Offset will not be updated (i.e., Offset = 1000). It is understood that by comparing the Offset in the retrieval master node with the Offset in the retrieval slave node, it can be determined whether the data in the master retrieval database in the retrieval master node and the data in the slave retrieval database in the retrieval slave node are consistent.

S102, when the retrieval main node determines that the retrieval service identification is the same as the service identification of the retrieval service currently operated by the retrieval main node and the synchronous data identification is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database of the retrieval main node and the auxiliary retrieval database of the retrieval auxiliary node.

In some possible embodiments, when the retrieval master node determines that the retrieval service identification in the data synchronization request is the same as the service identification of the retrieval service currently operated by the retrieval master node and the synchronization data identification is contained in the synchronization buffer queue, partial synchronization of the master retrieval database of the retrieval master node and the slave retrieval database of the retrieval slave node may be initiated. Optionally, when the retrieval master node determines that the retrieval service identifier included in the data synchronization request is not identical to or consistent with the service identifier of the retrieval service currently operated by the retrieval master node, complete synchronization of the master retrieval database of the retrieval master node and the slave retrieval database of the retrieval slave node may be initiated. It should be understood that when the retrieval service identifier included in the data synchronization request is different from the service identifier of the retrieval service currently running in the retrieval master node, it means that the retrieval master service process in the retrieval master node copied before the retrieval slave node is disconnected or restarted is not already the master process currently running in the retrieval master node, and therefore, it is necessary to initiate full synchronization. Optionally, when the retrieval master node determines that the retrieval service identifier included in the data synchronization request is the same as the service identifier of the retrieval service currently operated by the retrieval master node, the synchronized data identifier may be continuously determined. Specifically, when the synchronous data identifier included in the data synchronization request is not in or not included in the synchronous cache queue of the retrieval main node, full synchronization of the main retrieval database of the retrieval main node and the slave retrieval database of the retrieval slave node is initiated, and when the synchronous data identifier included in the data synchronization request is included in the synchronous cache queue of the retrieval main node, partial synchronization of the main retrieval database of the retrieval main node and the slave retrieval database of the retrieval slave node is initiated. It should be understood that a plurality of write class requests are stored in the synchronization buffer queue, wherein one write class request corresponds to one synchronization data identifier, and any synchronization data identifier is used for identifying the total data volume written into the main search database after the write class request corresponding to any synchronization data identifier is executed.

Specifically, when the partial synchronization is initiated, all write class requests stored after the write class request corresponding to the synchronized data identifier may be determined from the synchronization buffer queue, and all write class requests may be sent to the retrieval slave node to trigger the retrieval slave node to execute all write class requests to update the data stored in the retrieval database and the synchronized data identifier. That is, the retrieval master node may directly issue all write class requests in the synchronization buffer queue from the synchronized data identifier to the retrieval slave node, and the retrieval slave node may perform feature management operations such as adding, deleting, and/or modifying data in the retrieval slave database by executing the received write class requests, so as to update the status of the retrieval slave database and retrieve the synchronized data identifier in the slave node. It should be appreciated that the synchronization buffer queue is a fixed size First Input First Output (FIFO) memory queue maintained by the retrieval master node. The size of the synchronous buffer queue or the queue length may be determined according to an actual application scenario, which is not limited herein. For example, the synchronization buffer queue size may be configured to be 100M. Wherein, the new data can be enqueued from the tail of the queue, and when the number of enqueue elements is larger than the length of the queue, the old data enqueued first is dequeued from the head of the queue.

For ease of understanding, please refer to fig. 3c, and fig. 3c is a schematic flowchart of initiating partial synchronization according to an embodiment of the present application. As shown in fig. 3c, the retrieval slave node sends a data synchronization request to the retrieval master node, wherein the data synchronization request includes a retrieval service identifier ID stored in the retrieval slave node and operated by the retrieval master node, and a synchronization data identifier Offset in the retrieval slave node. The retrieval master node receives a data synchronization request sent by the retrieval slave node, and can firstly judge whether a retrieval service identifier ID included in the data synchronization request is the same as a service identifier ID 'of a currently running retrieval service, if the ID = ID', whether a synchronous data identifier Offset included in the data synchronization request is contained in a synchronous cache queue is continuously judged, and if the synchronous data identifier Offset is contained in the synchronous cache queue, partial synchronization is initiated, namely, the retrieval master node sends all write-class requests stored after the write-class request corresponding to the synchronous data identifier Offset in the synchronous cache queue to the retrieval slave node. Wherein, the retrieval slave node can update the data stored in the retrieval database and the synchronous data identification Offset in the retrieval slave node by receiving all the write class requests sent by the retrieval master node and executing all the write class requests. It will be appreciated that if ID ≠ ID', or Offset is not contained in the synchronization cache queue, then full synchronization is initiated. Wherein, after performing the full synchronization or the partial synchronization, the retrieval master node may transmit the service identification ID' of the currently running retrieval service to the retrieval slave node. The retrieval slave node receives the service identification ID ' sent by the retrieval master node, and can update the retrieval service identification ID to be the service identification ID ', namely, the service identification ID ' corresponding to the retrieval service operated in the retrieval master node is cached.

For example, please refer to fig. 3d, where fig. 3d is a schematic view of a partial synchronization scenario provided in the embodiment of the present application. Suppose that the retrieval service identifier and the synchronized data identifier included in the data synchronization request are ID004 and Offset =999 respectively, and the service identifier of the retrieval service currently operated by the retrieval master node is ID004, where 100 write requests and the synchronized data identifier corresponding to each write request are stored in the synchronization buffer queue. For example, as shown in fig. 3d, the write class request r1, and the synchronization data identifier Offset =999 corresponding to the write class request r1, the write class request r2, and the synchronization data identifier Offset =1000, … corresponding to the write class request r2, the write class request r99, and the synchronization data identifier Offset =1085 corresponding to the write class request r99, the write class request r100, and the synchronization data identifier Offset =1086 corresponding to the write class request r 100. Since the retrieval service ID included in the data synchronization request is the same as the service ID of the retrieval service currently operated by the retrieval master node (both ID 004), and the synchronization data ID Offset =999 included in the data synchronization request is included in the synchronization buffer queue, partial synchronization can be initiated. That is, the retrieving master node may send all write class requests (i.e., write class requests r 2-r 100) stored in the synchronization buffer queue after the write class request r1 corresponding to the synchronized data identification Offset =999 to the retrieving slave node to trigger the retrieving slave node to perform all the write class requests to update the data stored in the retrieving database and to retrieve the synchronized data identification in the slave node (i.e., to update the synchronized data identification in the retrieving slave node to Offset = 1086).

Referring to fig. 4a, fig. 4a is another schematic flow chart of a database synchronization method according to an embodiment of the present disclosure. The database synchronization method provided in the embodiment of the present application can be described by the following implementation manners provided in steps S201 to S203:

s201, the retrieval main node receives a data synchronization request sent by the retrieval slave node.

S202, when the retrieval main node determines that the retrieval service identification is the same as the service identification of the retrieval service currently operated by the retrieval main node and the synchronous data identification is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database of the retrieval main node and the auxiliary retrieval database of the retrieval auxiliary node.

The specific implementation manner of the steps S201 to S202 may refer to the description of the steps S101 to S102 in the embodiment corresponding to fig. 3a, and is not described herein again.

S203, when the retrieval main node determines that the retrieval service identification is different from the service identification of the retrieval service currently operated by the retrieval main node, or the synchronous data identification is not contained in the synchronous cache queue, initiating the complete synchronization of the main retrieval database and the auxiliary retrieval database.

In some possible embodiments, when the retrieval master node determines that the retrieval service identification is not the same as the service identification of the retrieval service currently operated by the retrieval master node, or the synchronization data identification is not included in the synchronization buffer queue, a full synchronization of the master and slave retrieval databases may be initiated.

When the full synchronization is initiated, the retrieval master node can determine a specific full synchronization mode by evaluating the resource occupation condition (for example, GPU occupation condition) of the node. Wherein, the full synchronization includes binary log (binlog) full synchronization and feature file full synchronization. It should be appreciated that the binlog full sync mode may be selected if the retrieval master node resources are scarce or otherwise not free, and the signature file full sync mode may be selected if the retrieval master node resources are free. It should be understood that the node resources include CPU resources, GPU resources, VPU resources, and the like, which are determined according to the actual application scenario, and are not limited herein.

Specifically, when it is determined that full synchronization is binlog full synchronization, the retrieval master node may send the binlog to the retrieval slave node to trigger the retrieval slave node to obtain a write class request recorded in the binlog by deserializing the binlog and update data stored in the retrieval database and the synchronization data identification by executing the write class request. It should be understood that, after retrieving the write class requests in the binlog obtained from the node deserialization, the same write class requests may be merged and all the write class requests obtained from the deserialization may be deleted to obtain valid write class requests, and then the valid write class requests may be executed to update the data stored in the retrieval database and the synchronization data identifier.

When the complete synchronization is determined to be the complete synchronization of the feature files, the retrieval main node can make the snapshot of the main retrieval database at the current time persistent to the feature files and send the feature files and all the write-class requests received after the current time to the retrieval slave node so as to trigger the retrieval slave node to update the data stored in the slave retrieval database and the synchronous data identification by loading the feature files to the slave retrieval database and executing all the write-class requests. In general, full synchronization is mainly applicable to a scenario where a retrieval slave node starts synchronization for the first time (i.e., the retrieval slave node is a newly added node) or partial synchronization fails.

In some optional embodiments, after initiating the complete synchronization of the master retrieval database of the retrieval master node and the slave retrieval database of the retrieval slave node, the retrieval master node may further send the service identifier of the retrieval service currently operated by the retrieval master node to the retrieval slave node, so that the retrieval slave node updates the retrieval service identifier in the retrieval slave node to the service identifier for subsequent use.

For ease of understanding, please refer to fig. 4b, and fig. 4b is a schematic flowchart of initiating full synchronization provided by an embodiment of the present application. As shown in fig. 4b, the retrieval slave node sends a data synchronization request to the retrieval master node, wherein the data synchronization request includes a retrieval service identifier ID stored in the retrieval slave node and operated by the retrieval master node, and a synchronization data identifier Offset in the retrieval slave node. The retrieval main node can firstly judge whether a retrieval service identification ID included in the data synchronization request is the same as a service identification ID 'of the currently running retrieval service or not by receiving the data synchronization request sent by the retrieval slave node, and if the ID is not equal to the ID', the complete synchronization is initiated. If ID = ID', it is continuously determined whether the synchronization data flag Offset included in the data synchronization request is included in the synchronization buffer queue, and if not, full synchronization is initiated, and if the synchronization data flag Offset is included in the synchronization buffer queue, partial synchronization is initiated. For a specific process of partial synchronization, refer to the description of partial synchronization in fig. 3d, which is not described herein again. When the complete synchronization is initiated, the retrieval main node can judge whether the node resource of the retrieval main node is idle, and if the node resource of the retrieval main node is idle, the retrieval main node sends binlog to the retrieval slave node. The retrieval slave node receives the binlog sent by the retrieval master node and deserializes the binlog to obtain the write request recorded in the binlog. And then merging the same write requests and deleting the invalid write requests to the write requests obtained by the reverse sequence to obtain the valid write requests. Further, the retrieval slave node can update the data stored in the retrieval database and the synchronization data identification Offset in the retrieval slave node by executing the valid write class request. If the node resources of the retrieval main node are idle, the retrieval main node persists the snapshot of the main retrieval database at the current time to the feature file, and acquires each write-type request received after the current time. And if the snapshot persistence is successful, the retrieval master node sends each write-type request received after the current time to the retrieval slave node together with the feature file. Wherein, the retrieval slave node receives the feature file and each write class request sent by the retrieval master node, and can update the data stored in the slave retrieval database and the synchronous data identification Offset by loading the feature file into the slave retrieval database and executing each write class request. If the snapshot persistence fails, the main node is searched to clear the feature file and the obtained writing requests, and the operations of persisting the snapshot of the main search database at the current moment to the feature file and obtaining the writing requests received after the current moment are executed again. Wherein, after performing full synchronization or partial synchronization, the retrieval master node may further send the service identification ID' of the currently running retrieval service to the retrieval slave node. The retrieval slave node receives the service identification ID ' sent by the retrieval master node, and can update the retrieval service identification ID to be the service identification ID ', namely, the service identification ID ' corresponding to the retrieval service operated in the retrieval master node is cached.

In the embodiment of the application, the retrieval master node receives a data synchronization request sent by the retrieval slave node, wherein the data synchronization request comprises a retrieval service identifier and a synchronization data identifier. And when the retrieval main node determines that the retrieval service identifier is the same as the service identifier of the retrieval service currently operated by the retrieval main node and the synchronous data identifier is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database of the retrieval main node and the auxiliary retrieval database of the retrieval auxiliary node. And when the retrieval main node determines that the retrieval service identifier is different from the service identifier of the retrieval service currently operated by the retrieval main node, or the synchronous data identifier is not contained in the synchronous cache queue, initiating complete synchronization of the main retrieval database and the auxiliary retrieval database. By adopting the embodiment of the application, the data backup speed of the master-slave retrieval database can be improved, and the occupation of machine resources can be reduced.

Referring to fig. 5a, fig. 5a is another schematic flow chart of a database synchronization method according to an embodiment of the present disclosure. The database synchronization method provided in the embodiment of the present application can be described by the following implementation manners provided in steps S201 to S204:

The specific implementation manner of step S203 may refer to the description of step S203 in the embodiment corresponding to fig. 4a, and is not described herein again.

S204, the retrieval main node receives a client writing request sent by the client, executes the client writing request to update data stored in the main retrieval database and a synchronous cache queue, and sends the client writing request to the retrieval slave node.

In some possible embodiments, after the partial synchronization of the master retrieval database of the retrieval master node and the slave retrieval database of the retrieval slave node is initiated, or after the full synchronization of the master retrieval database and the slave retrieval database is initiated, the retrieval master node may further receive a client write class request sent by the client. After the retrieval main node executes the client-side write request to update the data stored in the main retrieval database and the synchronous cache queue, the retrieval main node can also send the client-side write request to the retrieval slave node to trigger the retrieval slave node to update the data stored in the slave retrieval database and the synchronous data identifier in the retrieval slave node by executing the client-side write request, so that the data consistency between the main retrieval database and the slave retrieval database is ensured in real time after partial synchronization or complete synchronization. Optionally, after receiving the client write request sent by the client, the retrieval master node may also serialize the client write request, so as to store or write the serialized client write request into the binlog, and after the binlog is successfully written, update the data stored in the master retrieval database and the synchronization cache queue by executing the client write request. And then the client writing request is synchronized to the retrieval slave node, so that the retrieval slave node updates the data stored in the retrieval database and updates the synchronous data identifier in the retrieval node by executing the writing request, and the data consistency among the retrieval databases is ensured. For convenience of description, the client write class request may be referred to as a write class request hereinafter.

For ease of understanding, please refer to fig. 5b, and fig. 5b is a schematic flow chart illustrating synchronization of write class requests according to an embodiment of the present application. As shown in fig. 5b, in order to ensure that the slave search database can timely follow the state change of the master search database after completing the complete synchronization or the partial synchronization, and avoid frequently executing the partial synchronization or even the complete synchronization process, which affects the processing of the normal service of the search master node, the search master node may also actively initiate the synchronization of the write request after each complete synchronization or partial synchronization is executed. Specifically, the retrieval master node may first determine whether the received retrieval request is a write-type request by receiving a retrieval request sent by a client, or by receiving a client retrieval request forwarded by a retrieval agent. When the retrieval main node determines that the retrieval request is not a write request, namely the retrieval request is a read request, the retrieval main node directly executes the write request. When the retrieval main node determines that the retrieval request is a write request, the retrieval main node may serialize the write request and write the serialized write request into the binary file. If the serialized write-type request is successfully written into the binary file, the retrieval master node can further execute the write-type request to update the master retrieval database and the synchronous cache queue, and send the write-type request to the retrieval slave node. When the retrieval slave node receives the write request sent by the retrieval master node, the write request can be executed to update the data stored in the retrieval database and the synchronous data identifier Offset in the retrieval slave node, so that the state consistency of the retrieval database of the master node and the slave node after synchronization is ensured in real time.

In the embodiment of the application, the retrieval master node receives a data synchronization request sent by the retrieval slave node, wherein the data synchronization request comprises a retrieval service identifier and a synchronization data identifier. And when the retrieval main node determines that the retrieval service identifier is the same as the service identifier of the retrieval service currently operated by the retrieval main node and the synchronous data identifier is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database of the retrieval main node and the auxiliary retrieval database of the retrieval auxiliary node. And when the retrieval main node determines that the retrieval service identifier is different from the service identifier of the retrieval service currently operated by the retrieval main node, or the synchronous data identifier is not contained in the synchronous cache queue, initiating complete synchronization of the main retrieval database and the auxiliary retrieval database. Optionally, after performing partial synchronization of the master search database of the search master node and the slave search database of the search slave node, or after performing complete synchronization of the master search database and the slave search database, the search master node may further receive a client write request sent by the client, perform the client write request to update data stored in the master search database and the synchronization cache queue, and send the client write request to the search slave node to implement real-time synchronization of the slave search database and the master search database. By adopting the embodiment of the application, the data backup speed of the master-slave retrieval database can be improved, and the occupation of machine resources can be reduced.

Referring to fig. 6a, fig. 6a is another schematic flow chart of a database synchronization method provided in the embodiment of the present application. The database synchronization method provided in the embodiment of the present application can be described by the following implementation manners provided in steps S201 to S205:

The specific implementation manner of step S204 may refer to the description of step S204 in the embodiment corresponding to fig. 4a, and is not described herein again.

S205, the retrieval main node receives the heartbeat detection request sent by the retrieval slave node, and determines to initiate partial synchronization or complete synchronization according to the heartbeat detection request including the first synchronization data identifier corresponding to the slave retrieval database.

In some possible embodiments, the retrieving master node may also receive heartbeat detection requests periodically sent by the retrieving slave node. The heartbeat detection request comprises a first synchronization data identifier corresponding to the retrieval database. It is understood that the heartbeat detection request can be used for realizing the data supplement of the slave retrieval database after the retrieval master node and the retrieval slave node are partially or completely synchronized. Generally, if the retrieval master node determines that the total data amount identified by the first synchronization data identifier is smaller than the total data amount identified by the second synchronization data identifier stored last in the synchronization buffer queue, and the first synchronization data identifier is included in the synchronization buffer queue, then partial synchronization of the master retrieval database and the slave retrieval database may be initiated. If the retrieval main node determines that the total data volume identified by the first synchronous data identification is smaller than the total data volume identified by the second synchronous data identification and the first synchronous data identification is not contained in the synchronous cache queue, the complete synchronization of the main retrieval database and the auxiliary retrieval database can be initiated. It should be understood that when the first synchronization data identifier is equal to the second synchronization data identifier last stored in the synchronization buffer queue, it indicates that there is no difference data between the slave search database and the master search database at the current time, i.e. the total data amount in the slave search database is consistent with the total data amount in the master search database, and therefore, it is not necessary to initiate any synchronization.

Referring to fig. 6b, fig. 6b is a schematic view of an application scenario of the database synchronization method according to the embodiment of the present application. As shown in fig. 6b, when the retrieval master node receives a retrieval request sent by the client, or when the retrieval master node receives a retrieval request of the client forwarded by the retrieval agent module, the retrieval master node first serializes the write-type request in the retrieval request, and writes the serialized write-type request into the binary file. Wherein, when the write is successful, the retrieval master node may further execute the write-type request to update the master retrieval database and the synchronization buffer queue. At this time, if there is a retrieval slave node newly added to the retrieval service that needs to perform the first synchronization, the retrieval slave node may send a data synchronization request to the retrieval master node, and after receiving the data synchronization request, the retrieval master node may initiate full synchronization to the retrieval slave node, so as to synchronize the data in the retrieval master database with the data in the retrieval slave database. It should be understood that the above-mentioned full synchronization initiated by the retrieval master node may include binlog full synchronization or feature file full synchronization, etc., which is determined according to the actual application scenario and is not limited herein. After the complete synchronization between the retrieval master node and the retrieval slave node is completed, when the retrieval master node receives the write-class request sent by the client again, the retrieval master node not only serializes the newly received write-class request to the binlog, but also needs to send the newly received write-class request to the retrieval slave node to realize the synchronization of the write-class request by executing the new write-class request to update the master retrieval database and the synchronization cache queue. Furthermore, the retrieval slave node can also send a heartbeat detection request to the retrieval master node at regular time, so that the retrieval master node can determine whether synchronization needs to be initiated or not and what type of synchronization is initiated according to the heartbeat detection request. If the retrieval slave node suddenly breaks the network at a certain subsequent time and quickly recovers the network, or if the retrieval slave node is immediately restarted after being shut down due to power failure and the like, the detection slave node can send a data synchronization request to the retrieval master node again. The retrieval node receives the data synchronization request, determines that a retrieval service identifier included in the data synchronization request is the same as a service identifier of a retrieval service currently operated by the retrieval master node, and when a synchronous data identifier included in the data synchronization request is contained in a synchronous cache queue, the retrieval master node can initiate partial synchronization so as to realize the synchronization of data in the retrieval master database and balance data in the retrieval slave retrieval database. After the partial synchronization between the retrieval master node and the retrieval slave node is completed, when the retrieval master node receives the write-class request sent by the client again, the retrieval master node not only serializes the newly received write-class request to the binlog, but also needs to send the newly received write-class request to the retrieval slave node to realize the synchronization of the write-class request by executing the new write-class request to update the master retrieval database and the synchronization cache queue. Furthermore, the retrieval slave node can also send a heartbeat detection request to the retrieval master node at regular time, so that the retrieval master node can determine whether synchronization needs to be initiated or not and what type of synchronization is initiated according to the heartbeat detection request. Obviously, after the complete synchronization or the partial synchronization is executed, the operations of real-time synchronization of the write request, heartbeat detection and the like are further executed, so that the consistency of the states of the retrieval libraries of the master node and the slave node after the synchronization can be ensured in real time.

Please refer to fig. 6c, fig. 6c is a schematic flow chart of heartbeat detection according to an embodiment of the present disclosure. As shown in fig. 6c, the heartbeat detection mechanism is initiated periodically by the retrieval slave node, initiating a time configuration decision. Specifically, the retrieval slave node sends a heartbeat detection request carrying a first synchronization data identifier Offset1 corresponding to the slave retrieval database to the retrieval master node, so that when the retrieval master node receives the heartbeat detection request, the current state of the slave retrieval database can be determined according to the first synchronization data identifier Offset1 included in the heartbeat detection request and the latest synchronization data identifier Offset 2 in the synchronization buffer queue. It should be appreciated that the first synchronization data identification Offset1 identifies the total amount of data that has been currently synchronized binned from the search database, and the latest synchronization data identification Offset 2 in the synchronization buffer queue identifies the total amount of data currently in the primary search database. Thus, by comparing the magnitude relationship of offset1 and offset 2, it can be determined whether synchronization is required from the search database, and what kind of synchronization is performed. Specifically, if the retrieval master node determines that the first synchronization data flag offset1 is not less than the latest synchronization data flag offset 2 in the synchronization buffer queue (i.e., offset 1= offset 2), it means that the total data amount in the slave retrieval database is equal to the total data amount in the master retrieval database, i.e., the states of the master retrieval database and the slave retrieval database are consistent, and therefore, it is not necessary to perform synchronization. If the retrieval master node determines that the first synchronization data identification offset1 is less than the latest synchronization data identification offset 2 in the synchronization buffer queue and the first synchronization data identification offset1 is contained in the synchronization buffer queue, partial synchronization of the master retrieval database and the slave retrieval database is initiated, and if the retrieval master node determines that the first synchronization data identification offset1 is less than the latest synchronization data identification offset 2 in the synchronization buffer queue and the first synchronization data identification offset1 is not contained in the synchronization buffer queue, full synchronization of the master retrieval database and the slave retrieval database is initiated. The partial synchronization process refers to the partial synchronization process in fig. 3c, and the complete synchronization process refers to the complete synchronization process in fig. 4b, which is not described herein again.

In the embodiment of the application, the retrieval master node receives a data synchronization request sent by the retrieval slave node, wherein the data synchronization request comprises a retrieval service identifier and a synchronization data identifier. And when the retrieval main node determines that the retrieval service identifier is the same as the service identifier of the retrieval service currently operated by the retrieval main node and the synchronous data identifier is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database of the retrieval main node and the auxiliary retrieval database of the retrieval auxiliary node. And when the retrieval main node determines that the retrieval service identifier is different from the service identifier of the retrieval service currently operated by the retrieval main node, or the synchronous data identifier is not contained in the synchronous cache queue, initiating complete synchronization of the main retrieval database and the auxiliary retrieval database. Optionally, after performing partial synchronization of the master retrieval database of the retrieval master node and the slave retrieval database of the retrieval slave node, or after performing complete synchronization of the master retrieval database and the slave retrieval database, the retrieval master node may further receive a client write-class request sent by the client, perform the client write-class request to update data stored in the master retrieval database and the synchronization cache queue, and send the client write-class request to the retrieval slave node. Optionally, the retrieval master node may further receive a heartbeat detection request sent by the retrieval slave node, and then determine to initiate partial synchronization or complete synchronization according to that the heartbeat detection request includes a first synchronization data identifier corresponding to the slave retrieval database. By adopting the embodiment of the application, the data backup speed of the master-slave retrieval database can be improved, and the occupation of machine resources can be reduced.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a database synchronization apparatus according to an embodiment of the present application. The database synchronization device provided by the embodiment of the application comprises:

a data synchronization request obtaining module 31, configured to receive a data synchronization request sent by a retrieval slave node, where the data synchronization request includes a retrieval service identifier and a synchronization data identifier;

a first synchronization initiating module 32, configured to initiate partial synchronization between a master search database of the search master node and a slave search database of the search slave node when it is determined that the search service identifier is the same as a service identifier of a search service currently operated by the search master node and the synchronization data identifier is included in a synchronization cache queue;

Referring to fig. 8, fig. 8 is another schematic structural diagram of a database synchronization apparatus according to an embodiment of the present disclosure.

In some possible embodiments, the first synchronization initiating module 32 is configured to:

In some possible embodiments, the apparatus includes a second synchronization initiating module 33, and the second synchronization initiating module 33 includes:

a first full synchronization determining unit 331 configured to initiate full synchronization of the master search database and the slave search database when it is determined that the search service identifier is not identical to a service identifier of a search service currently operated by the search master node; or

A second full synchronization determining unit 332, configured to initiate full synchronization of the master search database and the slave search database when it is determined that the synchronization data id is not included in the synchronization buffer queue.

In some possible embodiments, the second synchronization initiating module 33 further includes:

the first fully-synchronized execution unit 333 is configured to send the binary log to the retrieval slave node, to trigger the retrieval slave node to obtain a write request recorded in the binary log by deserializing the binary log, and to update the data stored in the retrieval database and the synchronized data identifier by executing the write request.

a second full-synchronization executing unit 334, configured to persist the snapshot of the master search database at the current time to a feature file, and send the feature file and each write class request received after the current time to the search slave node, so as to trigger the search slave node to update the data stored in the slave search database and the synchronization data identifier by loading the feature file into the slave search database and executing each write class request.

In some possible embodiments, the apparatus further includes a service identifier updating module 34, where the service identifier updating module 34 is configured to:

In some possible embodiments, the apparatus further includes a write class request synchronization module 35, where the write class request synchronization module 35 includes:

a write request receiving unit 351, configured to receive a client write request sent by a client;

a main search database updating unit 352, configured to execute the client write request to update the data stored in the main search database and the synchronization buffer queue;

a write request sending unit 353, configured to send the client write request to the retrieval slave node, so as to trigger the retrieval slave node to execute the client write request to update the data stored in the slave retrieval database and the synchronization data identifier in the retrieval slave node.

In some possible embodiments, the apparatus further includes a heartbeat detection module 36, and the heartbeat detection module 36 includes:

a heartbeat detection request receiving unit 361, configured to receive a heartbeat detection request sent by the slave node, where the heartbeat detection request includes a first synchronization data identifier corresponding to the slave retrieval database;

a first heartbeat detection processing unit 362, configured to initiate partial synchronization between the master search database and the slave search database if it is determined that a total data amount identified by the first synchronization data identifier is smaller than a total data amount identified by a second synchronization data identifier last stored in the synchronization buffer queue and the first synchronization data identifier is included in the synchronization buffer queue;

a second heartbeat detection processing unit 363, configured to initiate complete synchronization between the master search database and the slave search database if it is determined that the total data amount identified by the first synchronization data identifier is smaller than the total data amount identified by the second synchronization data identifier and the first synchronization data identifier is not included in the synchronization buffer queue.

In a specific implementation, the database synchronization apparatus may execute the implementation manner provided in each step in fig. 3a to fig. 6c through each built-in functional module, which is not described herein again.

In the embodiment of the present application, the database synchronization apparatus receives a data synchronization request sent from a node, and obtains that the data synchronization request includes a retrieval service identifier and a synchronization data identifier. When the retrieval service identification is determined to be the same as the service identification of the retrieval service currently running in the database synchronization device and the synchronization data identification is contained in the synchronization cache queue, initiating partial synchronization of the master retrieval database and the slave retrieval database of the retrieval slave node. And when the retrieval service identification is determined to be different from the service identification of the retrieval service currently running in the database synchronization device, or the synchronization data identification is not contained in the synchronization cache queue, initiating complete synchronization of the master retrieval database and the slave retrieval database. Optionally, after performing partial synchronization of the master search database and the slave search database of the search slave node, or after performing full synchronization of the master search database and the slave search database, the database synchronization apparatus may further receive a client write-class request sent by the client, and perform the client write-class request to update data stored in the master search database and the synchronization buffer queue, and send the client write-class request to the search slave node. Optionally, the database synchronization apparatus may further receive a heartbeat detection request sent by the retrieval slave node, and then determine to initiate partial synchronization or full synchronization according to that the heartbeat detection request includes the first synchronization data identifier corresponding to the retrieval database. By adopting the embodiment of the application, the data backup speed of the master-slave retrieval database can be improved, and the occupation of machine resources can be reduced.

Referring to fig. 9, fig. 9 is a schematic structural diagram of a terminal device provided in an embodiment of the present application. As shown in fig. 9, the terminal device in this embodiment may include: one or more processors 401, memory 402, and a transceiver 403. The processor 401, memory 402 and transceiver 403 are connected by a bus 404. The memory 402 is used for storing a computer program comprising program instructions, and the processor 401 is used for executing the program instructions stored by the memory 402, in particular:

the transceiver 403 is configured to receive a data synchronization request sent from a node, where the data synchronization request includes a retrieval service identifier and a synchronization data identifier;

the processor 401 is configured to initiate partial synchronization between a master search database of the search master node and a slave search database of the search slave node when it is determined that the search service identifier is the same as a service identifier of a search service currently operated by the search master node and the synchronization data identifier is included in a synchronization buffer queue;

In some possible embodiments, the processor 401 is configured to determine, from the synchronization buffer queue, all write class requests stored after the write class request corresponding to the synchronization data identifier;

the transceiver 403 is configured to send the all-write-class request to the retrieving slave node to trigger the retrieving slave node to execute the all-write-class request to update the data stored in the slave retrieving database and the synchronous data identifier in the retrieving slave node.

In some possible embodiments, the processor 401 is configured to:

initiating complete synchronization of the master search database and the slave search database when it is determined that the search service identifier is not the same as the service identifier of the search service currently operated by the search master node; or

And when the synchronous data identification is determined not to be contained in the synchronous buffer queue, initiating complete synchronization of the main retrieval database and the auxiliary retrieval database.

In some possible embodiments, the transceiver 403 is configured to:

sending a binary log to the retrieval slave node to trigger the retrieval slave node to acquire a write class request recorded in the binary log by deserializing the binary log, and to update the data stored in the retrieval database and the synchronous data identifier by executing the write class request.

In some possible embodiments, the processor 401 is configured to persist the primary search database snapshot at the current time to a feature file;

the transceiver 403 is configured to send the profile and each write class request received after the current time to the retrieval slave node, so as to trigger the retrieval slave node to update the data stored in the retrieval database and the synchronization data identifier by loading the profile into the retrieval database and executing each write class request.

In some possible embodiments, the transceiver 403 is configured to:

receiving a client writing request sent by a client;

the processor 401 is configured to execute the client write request to update the data stored in the main search database and the synchronization buffer queue;

the transceiver 403 is configured to send the client write request to the retrieval slave node to trigger the retrieval slave node to execute the client write request to update the data stored in the retrieval database and the synchronization data identifier in the retrieval slave node.

In some possible embodiments, the transceiver 403 is configured to receive a heartbeat detection request sent by the retrieval slave node, where the heartbeat detection request includes a first synchronization data identifier corresponding to the slave retrieval database;

the processor 401 is configured to initiate partial synchronization between the master search database and the slave search database if it is determined that the total data amount identified by the first synchronization data identifier is smaller than the total data amount identified by the second synchronization data identifier last stored in the synchronization buffer queue and the first synchronization data identifier is included in the synchronization buffer queue;

the processor 401 is further configured to initiate full synchronization between the master search database and the slave search database if it is determined that the total data amount identified by the first synchronization data identifier is smaller than the total data amount identified by the second synchronization data identifier and the first synchronization data identifier is not included in the synchronization buffer queue.

It should be appreciated that in some possible implementations, the processor 401 may be a Central Processing Unit (CPU), and the processor may be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The memory 402 may include both read-only memory and random access memory, and provides instructions and data to the processor 401. A portion of the memory 402 may also include non-volatile random access memory. For example, the memory 402 may also store device type information.

In a specific implementation, the terminal device may execute the implementation manners provided in the steps in fig. 3a to fig. 6c through the built-in functional modules, which may specifically refer to the implementation manners provided in the steps, and no further description is given here.

In the embodiment of the present application, the terminal device may obtain that the data synchronization request includes the retrieval service identifier and the synchronization data identifier based on the data synchronization request sent from the node by receiving the retrieval. And when the retrieval service identification is determined to be the same as the service identification of the retrieval service currently running in the terminal equipment and the synchronous data identification is contained in the synchronous cache queue, initiating partial synchronization of the master retrieval database and the slave retrieval database of the retrieval slave node. And when the retrieval service identification is determined to be different from the service identification of the retrieval service currently running in the terminal equipment, or the synchronous data identification is not contained in the synchronous cache queue, initiating complete synchronization of the main retrieval database and the auxiliary retrieval database. Optionally, after performing partial synchronization of the master search database and the slave search database of the search slave node, or after performing full synchronization of the master search database and the slave search database, the terminal device may further receive a client write class request sent by the client, and perform the client write class request to update data stored in the master search database and the synchronization buffer queue, and send the client write class request to the search slave node. Optionally, the terminal device may further receive a heartbeat detection request sent by the retrieval slave node, and then determine to initiate partial synchronization or complete synchronization according to that the heartbeat detection request includes the first synchronization data identifier corresponding to the retrieval database. By adopting the embodiment of the application, the data backup speed of the master-slave retrieval database can be improved, and the occupation of machine resources can be reduced.

An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, where the computer program includes program instructions, and when the program instructions are executed by a processor, the database synchronization method provided in each step in fig. 3a to 6c is implemented.

The computer-readable storage medium may be the database synchronization apparatus provided in any of the foregoing embodiments or an internal storage unit of the terminal device, such as a hard disk or a memory of an electronic device. The computer readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) card, a flash card (flash card), and the like, which are provided on the electronic device. Further, the computer readable storage medium may also include both an internal storage unit and an external storage device of the electronic device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the electronic device. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.

The terms "first", "second", "third", "fourth", and the like in the claims and in the description and drawings of the present application are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The method and the related apparatus provided by the embodiments of the present application are described with reference to the flowchart and/or the structural diagram of the method provided by the embodiments of the present application, and each flow and/or block of the flowchart and/or the structural diagram of the method, and the combination of the flow and/or block in the flowchart and/or the block diagram can be specifically implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block or blocks of the block diagram. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block or blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block or blocks.

Claims

1. A database synchronization method, the method comprising:

the method comprises the steps that a retrieval main node receives a data synchronization request sent by a retrieval slave node, wherein the data synchronization request comprises a retrieval service identifier and a synchronization data identifier;

when the retrieval main node determines that the retrieval service identification is the same as the service identification of the retrieval service currently operated by the retrieval main node and the synchronous data identification is contained in a synchronous cache queue, initiating partial synchronization of a main retrieval database of the retrieval main node and a slave retrieval database of the retrieval slave node;

2. The method of claim 1, wherein said initiating partial synchronization of a master retrieval database of said retrieval master node and a slave retrieval database of said retrieval slave node comprises:

3. The method of claim 2, further comprising:

Initiating a full synchronization of the master search database and the slave search database when the search master node determines that the synchronized data identification is not contained in a synchronization buffer queue.

4. The method of claim 3, wherein said initiating full synchronization of said master search database and said slave search database comprises:

the retrieval main node sends the binary log to the retrieval slave node to trigger the retrieval slave node to acquire the writing class request recorded in the binary log by deserializing the binary log and update the data stored in the slave retrieval database and the synchronous data identification by executing the writing class request.

5. The method of claim 3, wherein said initiating full synchronization of said master search database and said slave search database comprises:

the retrieval master node persists the master retrieval base snapshot at the current moment to a feature file, and sends the feature file and each write-class request received after the current moment to the retrieval slave node so as to trigger the retrieval slave node to update the data stored in the slave retrieval database and the synchronous data identifier by loading the feature file to the slave retrieval database and executing each write-class request.

6. The method according to any of claims 1-5, wherein after said initiating a partial synchronization of a master search database of said search master node and a slave search database of said search slave node, or after said initiating a full synchronization of said master search database and said slave search database, the method further comprises:

and the retrieval master node sends the service identification to the retrieval slave node so that the retrieval slave node updates the retrieval service identification into the service identification.

7. The method according to any of claims 1-5, wherein after said initiating a partial synchronization of a master search database of said search master node and a slave search database of said search slave node, or after said initiating a full synchronization of said master search database and said slave search database, the method further comprises:

the retrieval main node receives a client writing request sent by a client;

the retrieval main node executes the client-side write request to update the data stored in the main retrieval database and the synchronous cache queue;

the retrieval master node sends the client write class request to the retrieval slave node to trigger the retrieval slave node to execute the client write class request to update the data stored in the slave retrieval database and the synchronous data identification in the retrieval slave node.

8. The method of claim 7, further comprising:

if the retrieval main node determines that the total data volume identified by the first synchronous data identification is less than the total data volume identified by a second synchronous data identification stored in the synchronous cache queue at last and the first synchronous data identification is contained in the synchronous cache queue, initiating partial synchronization of the main retrieval database and the auxiliary retrieval database;

if the retrieval master node determines that the total data volume identified by the first synchronization data identifier is less than the total data volume identified by the second synchronization data identifier and the first synchronization data identifier is not included in the synchronization cache queue, initiating full synchronization of the master retrieval database and the slave retrieval database.

9. A database synchronization apparatus, wherein the database synchronization apparatus is a retrieval master node, the apparatus comprising:

10. A terminal device, comprising a processor and a memory, the processor and the memory being interconnected;

the memory for storing a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of any of claims 1-8.

11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-8.