WO2021253688A1

WO2021253688A1 - Data synchronization method and apparatus, and data query method and apparatus

Info

Publication number: WO2021253688A1
Application number: PCT/CN2020/119711
Authority: WO
Inventors: 杨飞; 曹素杰
Original assignee: 北京旷视科技有限公司
Priority date: 2020-06-18
Filing date: 2020-09-30
Publication date: 2021-12-23
Also published as: CN111914020A

Abstract

Provided are a data synchronization method, a data synchronization apparatus, a data query method, a data query apparatus, an electronic device and a computer-readable storage medium. The data synchronization method comprises: acquiring data to be synchronized, wherein said data comprises data in database tables of one or more first databases; establishing a first index in a target database, and successively synchronizing, to the first index, data in database tables of all the first databases corresponding to the data to be synchronized; and if the current first index satisfies an index rollover policy, establishing a new first index, and continuing to synchronize data, which has not yet been synchronized, to the new first index. By means of successively writing data in database tables of a plurality of databases into one or more indexes in a rollover manner, occupied shard resources are saved.

Description

Data synchronization method and device, data query method and device

This application is filed based on a Chinese patent application with an application number of 202010561213.7 and an application date of June 18, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this application.

Technical field

The present disclosure generally relates to the field of database storage, and specifically relates to a data synchronization method, a data synchronization device, a data query method, a data query device, an electronic device, and a computer-readable storage medium.

Background technique

In the process of data storage and retrieval, especially in distributed storage data, the base database data needs to be synchronized to the retrieval engine to perform attribute retrieval. In order to improve retrieval efficiency, it is necessary to index the data stored in the retrieval engine. In some related technologies of distributed storage and distributed retrieval, the retrieval engine will be set with a certain number of shards. Each shard corresponds to an index, and an index corresponds to a database table written in a database, that is, to The database table of each database establishes an independent index. In this case, if the number of databases to be synchronized is large, a large amount of sharding resources will be required. If the number of shards is not enough, it will not be able to build The new index makes the data of some databases unable to be synchronized and searched.

Summary of the invention

In order to solve the above-mentioned problems in the prior art, the first aspect of the present disclosure provides a data synchronization method, wherein the method includes: acquiring data to be synchronized, and the data to be synchronized includes data in one or more database tables of the first database. Data; create a first index in the target database, and synchronize the data of all the database tables of the first database corresponding to the data to be synchronized to the first index in turn; if the current first index meets the index rolling strategy, create a new first index Index, and continue to synchronize the data that has not yet been synchronized to the new first index.

In an embodiment, the current first index satisfies the index rolling strategy, including at least one of the following: the storage space occupied by the current first index reaches the storage threshold; and the amount of data synchronized to the current first index reaches the capacity threshold.

In an embodiment, the current first index satisfies the index rolling strategy, which further includes: the time for data synchronization of the current first index reaches the time threshold.

In an embodiment, the data to be synchronized further includes data in one or more database tables of the second database, wherein the amount of data in the second database is greater than the amount of data in the first database; the method further includes: establishing in the target database One or more second indexes corresponding to the database table of each second database one-to-one; the content of the database table of each second database is synchronized to its corresponding second index respectively.

In an embodiment, the method further includes: determining the type of index creation based on the table name of the database table, wherein the table name of the database table of the first database includes a first identifier, and the database table of the second database includes a second identifier; if If the table name of the database table contains the first identifier, the first index is created, and the data of all the database tables of the first database corresponding to the data to be synchronized is synchronized to the first index in sequence; if the table name of the database table contains the second To identify, the step of establishing one or more second indexes corresponding to the database tables of each second database is executed.

In an embodiment, the method further includes: in response to a data deletion request for deleting any of the first databases, determining one or more first indexes corresponding to the first database to be deleted; deleting one or more first indexes Data corresponding to the deleted first database.

In an embodiment, the method further includes: in response to a data deletion request for deleting any second database, deleting a second index corresponding to the second database to be deleted.

In an embodiment, the method further includes: when an operation on the first data in the database table of any first database is detected, based on the first data, determining the first index corresponding to the first data among all the first indexes And according to the operation, synchronously correct the data in the first index corresponding to the first data; wherein, the operation includes adding data, modifying data, or deleting data.

In an embodiment, the method further includes: when an operation on the second data in the database table of any second database is detected, in the second index corresponding to the database table of the second database, synchronously revising the first data according to the operation. The data in the second index corresponding to the second data; where the operations include adding data, modifying data, or deleting data.

A second aspect of the present disclosure provides a data synchronization device, including: a data acquisition module for acquiring data to be synchronized, the data to be synchronized including data in one or more database tables of a first database; and a data synchronization module for Create a first index in the target database, and synchronize the data of all the database tables of the first database corresponding to the data to be synchronized to the first index in turn; if the current first index satisfies the index rolling strategy, a new first index is created, Continue to synchronize the unsynchronized data to the new first index.

A third aspect of the present disclosure provides a data query method. The method includes: obtaining query information of the data to be queried; querying synchronization data corresponding to the data to be queried in the index of the target database based on the query information; and determining the query to be queried based on the synchronization data The location of the data in its corresponding database; wherein the data to be synchronized in the database is synchronized to the index of the target database by the data synchronization method according to any one of claims 1-8.

A fourth aspect of the present disclosure provides a data query device including: a receiving module for obtaining query information of the data to be queried; a search module for querying synchronization data corresponding to the data to be queried in the index of the target database based on the query information The query module is used to determine the location of the data to be queried in its corresponding database based on the synchronization data; wherein, the data to be synchronized in the database is synchronized to the index of the target database through the data synchronization method as in the first aspect.

A fifth aspect of the present disclosure provides an electronic device including: a memory for storing instructions; and a processor for calling the instructions stored in the memory to execute the data synchronization method according to the first aspect or the data query method according to the third aspect .

A sixth aspect of the present disclosure provides a computer-readable storage medium in which instructions are stored. When the instructions are executed by a processor, the data synchronization method according to the first aspect or the data query method according to the third aspect is executed.

The data synchronization method, data synchronization device, data query method, data query device, electronic equipment, and computer readable storage medium provided by the present disclosure can write one or more data in the database tables of multiple databases in sequence by scrolling In the index, thereby saving the occupied fragmentation resources.

Description of the drawings

By reading the following detailed description with reference to the accompanying drawings, the above and other objectives, features, and advantages of the embodiments of the present disclosure will become easier to understand. In the drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

Fig. 1 shows a schematic flowchart of a data synchronization method according to an embodiment of the present disclosure;

FIG. 2 shows a schematic flowchart of a data synchronization method according to another embodiment of the present disclosure;

FIG. 3 shows a schematic flowchart of a data synchronization method according to another embodiment of the present disclosure;

FIG. 4 shows a schematic flowchart of a data synchronization method according to another embodiment of the present disclosure;

FIG. 5 shows a schematic flowchart of a data synchronization method according to another embodiment of the present disclosure;

Fig. 6 shows a schematic flowchart of a data query method according to an embodiment of the present disclosure;

Fig. 7 shows a schematic diagram of a data synchronization device according to an embodiment of the present disclosure;

FIG. 8 shows a schematic diagram of a data synchronization device according to another embodiment of the present disclosure;

Fig. 9 shows a schematic diagram of a data query device according to an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure;

In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

detailed description

Hereinafter, the principle and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are only provided to enable those skilled in the art to better understand and then implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

It should be noted that although expressions such as "first" and "second" are used herein to describe different modules, steps, data, etc. of the embodiments of the present disclosure, the expressions such as "first" and "second" are only used to describe Different modules, steps, data, etc. are distinguished, but do not indicate a specific order or degree of importance. In fact, expressions such as "first" and "second" can be used interchangeably.

The data to be synchronized can be the bottom library. The bottom library includes static library (large data volume, usually a static library contains more than 8000w data), dynamic library (small data volume, usually a dynamic library contains less than 100w data), one piece of data in the bottom library is usually It is a picture, name, ID number and other attribute information. In some related technologies, if you want to retrieve the data in the bottom database, you need to synchronize the bottom database data stored in a distributed file storage database (such as MongoDB) to a distributed search engine (such as Elasticsearch) through a synchronization tool such as Monstache. ), and then implement retrieval query functions such as fuzzy retrieval of attributes in the distributed retrieval engine. When synchronizing, you need to use the synchronization tool Monstache of the MongoDB database to synchronize the data in the bottom database of the MongoDB database, that is, the static database and dynamic database data, to the search engine Elasticsearch by reading the operation log (oplog). In related technologies, a static library is put into an index of Elasticsearch, and a dynamic library is also put into an index of Elasticsearch.

When synchronizing base database data from MongoDB to Elasticsearch, each database in MongoDB corresponds to an index in Elasticsearch. The creation of an index will occupy fragmented resources, and one index will occupy one fragment. The amount of data contained in the dynamic library is small. When a dynamic library corresponds to an index and occupies a fragment, the data in the dynamic library actually uses only a small part of the fragment, but because the dynamic library occupies the fragment Fragment, data in other databases cannot be written to this fragment. When the number of dynamic libraries to be synchronized is large, after the synchronization is completed, a large number of shards are in a state of being occupied but not full, which will cause a great waste of sharding resources in Elasticsearch. In addition, currently only 1,000 shards are available for use on a single Elasticsearch machine. If a database corresponds to one shard, a single Elasticsearch machine can only store up to 1,000 dynamic libraries/static libraries. Once the dynamic library/static library to be synchronized exceeds this Number, you need to expand a single Elasticsearch machine into a cluster.

In order to solve the above-mentioned problem, the embodiment of the present disclosure provides a data synchronization method 10, which can be applied to a database of distributed file storage. As shown in FIG. 1, the data synchronization method 10 may include: step S11 to step S13. The following is a detailed description of the above steps:

Step S11: Obtain data to be synchronized. The data to be synchronized includes data in one or more database tables of the first database.

The data to be synchronized in the present disclosure can be any form of data, for example, it can be used in the field of image recognition, and the data to be synchronized can be data stored in a base database, such as pictures, names, ID numbers, and personal attributes, etc. information. Among them, the first database may be a dynamic database, which is characterized by a relatively small amount of data. Generally, a first database contains 0 to 1 million pieces of data, but the number of first databases to be synchronized is relatively large. If the data of the database table of each first database is put into an index respectively, that is, one fragment is occupied, a large number of fragments will be in an occupied but not full state, and a large amount of fragmentation resources will be wasted. For each database data, when writing to the MongoDB database, it will be written into a database table of MongoDB, and each database corresponds to a database table.

The database table of the first database The database table of the first database In step S12, a first index is established in the target database, and the data of all the database tables of the first database corresponding to the data to be synchronized are sequentially synchronized to the first index.

In step S13, if the current first index satisfies the index rolling strategy, a new first index is created, and the unsynchronized data is continuously synchronized to the new first index.

In the embodiment of the present disclosure, the contents of the database tables of each first database are sequentially written into the first index. It realizes that the database tables of multiple first databases are written into one index; and after an index meets the index rolling strategy, a new index can be created, and the content that is not currently stored is continued to be stored, thereby reducing fragmentation Waste of resources. In this embodiment, a first index may store data in multiple database tables of the first database, and the data of a database table of a first database may be stored in different indexes (due to the amount of data in the first database) If it is smaller, the probability that part of the data in the database table of a first database exists in the previous first index and part of the data exists in the latter first index is relatively small).

In an embodiment, the current first index satisfies the index rolling strategy, including at least one of the following: the storage space occupied by the current first index reaches the storage threshold; and the amount of data synchronized to the current first index reaches the capacity threshold. In this example, the conditions for satisfying the rolling strategy can be set according to the storage space occupied by the index. For example, if a first index A stores the contents of 10 database tables of the first database, it only occupies one sharding resource, if When the contents of the database table of the eleventh first database are written in the first half, the current first index A has already occupied the fragment resources occupied by it, and a new first index B is created on a new fragment. , Continue to write the second half of the database table of the 11th first database that has not yet been written in the new first index B, thus realizing "multiple database tables are written into one index, and another one is automatically created when the index capacity is exceeded. The rolling storage of "index" saves a lot of storage resources. On the other hand, the rolling strategy can also be set according to the amount of data synchronized to the index, that is, after a certain amount of data is written in a first index, a new first index can be recreated to continue writing data, ensuring The amount of data in each first index. For example, if the data volume is set to 100 million, when the database table of the 10th first database synchronizes data in the first index C, the data volume in the first index C reaches 100 million, then a new first index is created D. Continue to synchronize the contents of the database table of the tenth first database that are not written in index C to the first index D. One of the above two methods can be selected, and either one can be satisfied to satisfy the index rolling strategy.

In an embodiment, on the basis of satisfying the aforementioned conditions of the index rolling strategy, it may further include: the time for data synchronization of the current first index reaches the time threshold, that is, the index rolling strategy is satisfied. In this embodiment, a time threshold is set. In the process of data synchronization for a first index, if the time exceeds the time threshold, and there are new data to be synchronized that need to be synchronized to the first index, a new one can be created The first index.

In an embodiment, the data to be synchronized further includes data in one or more database tables of the second database, wherein the amount of data in the second database is greater than the amount of data in the first database. Each second database also corresponds to a database table in MongoDB. It is understandable that a data volume threshold can be set. When the data volume of the database in the data to be synchronized is greater than the data volume threshold, the database is considered to be the second database, otherwise the database is the first database. As shown in FIG. 2, on the basis of the data synchronization method 10, the data synchronization method 20 may include: step S14, establishing in the target database one or more second indexes corresponding to the database tables of each second database. Step S15, synchronize the contents of the database table of each second database to its corresponding second index respectively. In this embodiment, the data to be synchronized may also include a second database, and the amount of data in the second database is greater than that of the first database. The second database, such as a static database, is characterized by a relatively large amount of data, usually more than 80 million. Therefore, even if a shard corresponds to the database table of a static library, the shard corresponding to the static library will be in a relatively full state; in addition, the number of static libraries is relatively small, even if the shards corresponding to the static library are not full , It will not cause a lot of fragmentation waste. At the same time, when an index is only stored in the database table of a static library, because the amount of data in the static library is large, when the data in the static library is retrieved, it can be retrieved directly in the corresponding index, so the efficiency can be improved. . When the static library is deleted, the index corresponding to the static library can be directly deleted, so the efficiency of data deletion can be improved. On the contrary, if the static library is also stored on a rolling basis, there will be a high probability that the data in a static library is stored in different indexes. When retrieving data in a static library, it is necessary to first determine which indexes the data in the static library is stored in, and then perform retrieval in these indexes, which will reduce the retrieval efficiency. Based on the above reasons, a second index corresponding to the database table of each second database is established in the target database, that is, the database table of each second database has a corresponding second index, and then each second index The contents of the database tables of the database are respectively synchronized to their corresponding second indexes. That is, for the first database with a large number and a small amount of data, the rolling storage strategy of "multiple database tables are written into one index, and another index is automatically created when the index capacity is exceeded" is used for the first database. The database adopts a storage strategy of one-to-one correspondence between database tables and indexes. By adopting different strategies for different databases, storage resources are saved, and retrieval efficiency is also taken into account.

In an embodiment, as shown in FIG. 3, based on the data synchronization method 20, the data synchronization method 30 may include: step S16, determining the type of index creation based on the table name of the database table, where the database of the first database The table name of the table contains the first identification, and the database table of the second database contains the second identification; if the table name of the database table contains the first identification, step S12 is executed; if the table name of the database table contains the second identification, step S12 is executed S14. In this embodiment, the naming rules of database tables corresponding to different database types can be different. The identifier contained in the table name can be used to determine the database type corresponding to the database table, so that the corresponding database table can be conveniently used according to different types of databases. Strategies to create indexes.

For example, in the case of externally writing a dynamic library data to MongoDB, the data of the same dynamic library is written to a corresponding database table of MongoDB. The database table name has monitor or other identifiers, and the operation occurs when writing. The log oplog is monitored in real time by the synchronization tool Monstache, so that the written data will be synchronized to the search engine Elasticsearch in real time. In the process of synchronizing to Elasticsearch, according to the identification of the database table name with the monitor identifier, the step S12 is adopted. The method is to use the synchronization template for the dynamic library set on Elasticsearch to write all the contents of this database table into the current index, and then roll the storage to the new index when the current index is full.

In the case of externally writing static database data to MongoDB, the data of the same static database is written to a corresponding database table of MongoDB. The table name of the database table contains statit or other identifiers. Similarly, oplog is generated when writing. If monitored by Monstache, the written data will be synchronized to Elasticsearch in real time. In the process of synchronizing to Elasticsearch, according to the identification of the database table name with static identification, the method of step S14 is adopted, that is, the setting on Elasticsearch is used. The synchronization template for static libraries writes all the contents of this database table into a separate index. In one embodiment, as shown in FIG. 4, on the basis of the data synchronization method 30, the data synchronization method 40 may include: step S171, in response to a data deletion request for deleting any first database, determining the first database to be deleted One or more first indexes corresponding to a database; step S172, deleting data corresponding to the deleted first database in the one or more first indexes. In this embodiment, when data in a certain first database needs to be deleted according to actual needs, because the first database and the index are not in a one-to-one correspondence, in response to a data deletion request to delete any first database, it is necessary to delete all data first. The first index corresponding to the first database to be deleted is determined in the first index, which may correspond to one first index, or may correspond to multiple first indexes based on synchronization based on a rolling strategy. After determining the corresponding one or more first indexes, delete the corresponding data in the first index, thereby completing the synchronization of the data in the index and the database.

In another embodiment, as shown in FIG. 5, based on the data synchronization method 40, the data synchronization method 50 may include: step S173, in response to a data deletion request for deleting any second database, deleting the first to be deleted The second index corresponding to the second database. In this embodiment, when data in a certain second database needs to be deleted according to actual needs, since the second database has a one-to-one correspondence with the second index, the second index corresponding to the second database to be deleted can be Determine in advance or in real time, and directly delete the second index corresponding to the second database to complete the data deletion. In this way, the efficiency of data deletion can be improved. In one example, the index name of the second index can be set to include the name of the second database or the database table of the second database, or the index name of the second index can be set to be the same as that of the second database or the database of the second database. The names of the tables are the same. In this way, the corresponding second index can be quickly determined according to the name of the second database to be deleted.

In an embodiment, the data synchronization method may further include: when an operation on the first data in the database table of any first database is detected, based on the first data, determining the first data corresponding to the first data in all the first indexes The position of the data in the first index; and according to the operation, synchronously correct the data in the first index corresponding to the first data; wherein, the operation includes adding data, modifying data, or deleting data. In this embodiment, when an operation is performed on the first data of a database table of any first database, since the database table of the first database is synchronized to the first index through a rolling strategy, it is necessary to determine the operated item first. In which first index and the position of the first data in the first index, synchronization can then be carried out. The position of the first data can be determined by searching in all the first indexes, and then according to the actual operation type, Such as adding data, modifying data, or deleting data, synchronize the content in the first index.

In another embodiment, the data synchronization method may further include: when an operation on the second data in the database table of any second database is detected, in the second index corresponding to the database table of the second database, according to the operation , Synchronously revise the data in the second index corresponding to the second data, where the operations include adding data, modifying data or deleting data. Different from the synchronization after the operation on the first data in the database table of the first database, after the operation on the second data in the database table of a second database, since each database table of the second database has a one-to-one correspondence Therefore, the data corresponding to the second data can be directly synchronized and corrected in accordance with the specific operation in the corresponding second index.

Based on the same inventive concept, the present disclosure also provides a data query method 60. As shown in FIG. 6, the data query method 20 may include: step S21 obtains query information of the data to be queried; Query the synchronization data corresponding to the data to be queried in; step S23, based on the synchronization data, determine the location of the data to be queried in its corresponding database; wherein, the data synchronization method 10 to 50 in any of the foregoing embodiments is used to synchronize the data in the database. The data is synchronized to the index of the target database. The index established through data synchronization methods 10 to 50 can reduce the occupation of fragmentation resources, improve efficiency, and can conveniently perform queries based on the data synchronized to the index.

Based on the same inventive concept, the present disclosure also provides a data synchronization device 100. As shown in FIG. 7, the data synchronization device 100 includes: a data acquisition module 110 for acquiring data to be synchronized. The data to be synchronized includes one or more first The data in the database table of the database; the data synchronization module 120 is used to establish a first index in the target database, and sequentially synchronize the data of all the database tables of the first database corresponding to the data to be synchronized to the first index; if the current first index If an index satisfies the index rolling strategy, a new first index is created, and the unsynchronized data is continuously synchronized to the new first index.

In one example, the current first index satisfies the index rolling strategy, including at least one of the following: the storage space occupied by the current first index reaches the storage threshold; and the amount of data synchronized to the current first index reaches the capacity threshold.

In one example, the current first index satisfies the index rolling strategy, which further includes: the time for data synchronization of the current first index reaches the time threshold.

In one example, the data to be synchronized further includes one or more second databases, wherein the data volume of the second database is greater than the data in the database table of the first database; the data synchronization module 120 is also used to: One or more second indexes corresponding to the database tables of each second database are established in one-to-one; and the contents of the database tables of each second database are respectively synchronized to its corresponding second indexes.

In one example, the data synchronization module 120 is further configured to determine the type of index creation based on the table name of the database table, where the table name of the database table of the first database contains the first identifier, and the table name of the database table of the second database contains the first identifier. Second identification; if the table name of the database table contains the first identification, the first index is established in the target database, and the data of all the database tables of the first database corresponding to the data to be synchronized are sequentially synchronized to the first index; if the database table If the table name of contains the second identifier, one or more second indexes corresponding to the database tables of each second database are established in the target database.

In one example, as shown in FIG. 8, on the basis of the data synchronization device 100, the data synchronization device 200 may further include: a synchronization correction module 130, configured to determine whether to delete data in response to a data deletion request to delete any of the first databases One or more first indexes corresponding to the first database; delete data corresponding to the deleted first database in one or more first indexes.

In one example, the synchronization correction module 130 is further configured to delete the second index corresponding to the second database to be deleted in response to a data deletion request for deleting any second database.

In one example, the synchronization correction module 130 is further configured to: when an operation on the first data in the database table of any first database is detected, based on the first data, determine the first data corresponding to the first data in all the first indexes. The position of the data in the index; and according to the operation, the data in the first index corresponding to the first data is synchronously revised; wherein, the operation includes adding data, modifying data, or deleting data.

In one example, the synchronization correction module 130 is further configured to: when an operation on the second data in the database table of any second database is detected, in the second index corresponding to the database table of the second database, synchronize according to the operation Modify the data in the second index corresponding to the second data; where the operations include adding data, modifying data, or deleting data.

Regarding the modules in the data synchronization device 100 in the foregoing embodiment, the specific manner in which each module executes operations has been described in detail in the embodiment of the method, and will not be elaborated here.

Based on the same inventive concept, the present disclosure also provides a data query device 300. As shown in FIG. 9, the data query device 300 includes: a receiving module 210 for obtaining query information of the data to be queried; a search module 220 for query-based Information, query the synchronization data corresponding to the data to be queried in the index of the target database; the query module 230 is used to determine the position of the data to be queried in its corresponding database based on the synchronization data; wherein, by The data synchronization method 10 synchronizes the data to be synchronized with the database to the index of the target database.

Regarding the data query device 300 in the foregoing embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment related to the method, and will not be elaborated here.

As shown in FIG. 10, an embodiment of the present disclosure provides an electronic device 400. The electronic device 400 includes a memory 401, a processor 402, and an input/output (Input/Output, I/O) interface 403. Among them, the memory 401 is used to store instructions. The processor 402 is configured to call the instructions stored in the memory 401 to execute the data synchronization method or the data query method of the embodiment of the present disclosure. The processor 402 is respectively connected to the memory 401 and the I/O interface 403, for example, through a bus system and/or other forms of connection mechanisms (not shown). The memory 401 can be used to store programs and data, including programs of the data synchronization method or data query method involved in the embodiments of the present disclosure. The processor 402 executes various functional applications and data of the electronic device 400 by running the programs stored in the memory 401 deal with.

In the embodiment of the present disclosure, the processor 402 may use any of digital signal processors (Digital Signal Processing, DSP), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and Programmable Logic Array (Programmable Logic Array, PLA). The processor 402 may be implemented in at least one form of hardware, and the processor 402 may be one or more of a central processing unit (Central Processing Unit, CPU) or other forms of processing units with data processing capabilities and/or instruction execution capabilities combination.

The memory 401 in the embodiment of the present disclosure may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (Random Access Memory, RAM) and/or cache memory (cache). The non-volatile memory may include, for example, Read-Only Memory (ROM), Flash Memory (Flash Memory), Hard Disk Drive (HDD), Solid-State Drive (SSD), etc. .

In the embodiment of the present disclosure, the I/O interface 403 can be used to receive input commands (for example, numeric or character information, and generate key signal inputs related to the user settings and function control of the electronic device 400, etc.), and can also output various external commands. Kind of information (for example, image or sound, etc.). The I/O interface 403 in the embodiment of the present disclosure may include one or more of a physical keyboard, function buttons (such as volume control buttons, switch buttons, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel, etc. Piece.

It can be understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, they should not be understood as requiring these operations to be performed in the specific order shown or in a serial order, or requiring execution. All the operations shown in order to get the desired result. In certain circumstances, multitasking and parallel processing may be advantageous.

The methods and devices involved in the embodiments of the present disclosure can be implemented using standard programming techniques, and various method steps can be implemented using rule-based logic or other logic. It should also be noted that the words "device" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and/or hardware implementations and/or devices for receiving input.

Any steps, operations, or programs described herein can be executed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer readable medium containing computer program code, which can be executed by a computer processor for executing any or all of the described steps, operations, or programs.

For the purposes of example and description, the foregoing description of the implementation of the present disclosure has been given. The foregoing description is not exhaustive and does not intend to limit the present disclosure to the exact form disclosed. Various variations and modifications may exist based on the above teachings, or may be derived from the practice of the present disclosure. These embodiments are selected and described in order to explain the principles of the present disclosure and its practical applications, so that those skilled in the art can utilize the present disclosure with various implementations and various modifications suitable for the specific purpose conceived.

Claims

A data synchronization method, wherein the method includes:

Acquiring data to be synchronized, where the data to be synchronized includes data in one or more database tables of the first database;

Establishing a first index in the target database, and sequentially synchronizing data of all database tables of the first database corresponding to the data to be synchronized to the first index;

If the current first index satisfies the index rolling strategy, a new first index is created, and the unsynchronized data is continuously synchronized to the new first index.
The method according to claim 1, wherein the current first index satisfies the index rolling strategy and includes at least one of the following:

The storage space occupied by the current first index reaches a storage threshold;

The amount of data synchronized to the current first index reaches the capacity threshold.
The method according to claim 2, wherein the current first index satisfies the index rolling strategy, further comprising:

The time for data synchronization of the current first index reaches the time threshold.
The method according to claim 1, wherein the data to be synchronized further includes data in one or more database tables of a second database, wherein the amount of data in the second database is greater than the data in the first database quantity;

The method also includes:

Establishing one or more second indexes in the target database corresponding to each database table of the second database;

Synchronize the contents of the database table of each second database to its corresponding second index respectively.
The method according to claim 4, wherein the method further comprises:

Determining the type of index creation based on the table name of the database table, wherein the table name of the database table of the first database includes a first identifier, and the database table of the second database includes a second identifier;

If the table name of the database table contains the first identifier, the first index creation is performed, and the data of all the database tables of the first database corresponding to the data to be synchronized are sequentially synchronized to the first index. step;

If the table name of the database table includes the second identifier, the step of establishing one or more second indexes corresponding to each database table of the second database is performed.
The method according to claim 4, wherein the method further comprises:

In response to a data deletion request to delete any first database, determine one or more first indexes corresponding to the first database to be deleted;

Deleting data corresponding to the first database to be deleted in the one or more first indexes.
The method according to any one of claims 4-6, wherein the method further comprises:

In response to a data deletion request for deleting any second database, the second index corresponding to the second database to be deleted is deleted.
The method according to claim 4, wherein the method further comprises:

When an operation on the first data in the database table of any first database is detected, based on the first data, determine the position of the data in the first index corresponding to the first data among all the first indexes ；

And according to the operation, synchronously correct the data in the first index corresponding to the first data;

Wherein, the operation includes adding data, modifying data or deleting data.
The method according to any one of claims 4-6, wherein the method further comprises:

When an operation on the second data in the database table of any second database is detected, in the second index corresponding to the database table of the second database, according to the operation, synchronously correct the second data corresponding to the second data Data in the second index;

Wherein, the operation includes adding data, modifying data or deleting data.
A data synchronization device, wherein the device includes:

A data acquisition module for acquiring data to be synchronized, where the data to be synchronized includes data in one or more database tables of the first database;

The data synchronization module is configured to establish a first index in the target database, and sequentially synchronize the data of all the database tables of the first database corresponding to the data to be synchronized to the first index; if the current first index satisfies The index rolling strategy is to establish a new first index, and continue to synchronize the unsynchronized data to the new first index.
A data query method, wherein the method includes:

Obtain query information of the data to be queried;

Based on the query information, query the synchronization data corresponding to the data to be queried in the index of the target database;

Based on the synchronization data, determine the position of the data to be queried in its corresponding database;

Wherein, the data to be synchronized in the database is synchronized to the index of the target database by the data synchronization method according to any one of claims 1-9.
A data query device, wherein the device includes:

The receiving module is used to obtain the query information of the data to be queried;

The search module is configured to query the synchronization data corresponding to the data to be queried in the index of the target database based on the query information;

A query module, configured to determine the location of the data to be queried in its corresponding database based on the synchronization data;

Wherein, the data to be synchronized in the database is synchronized to the index of the target database by the data synchronization method according to any one of claims 1-9.
An electronic device, wherein the electronic device includes:

Memory for storing instructions; and

The processor is configured to call the instructions stored in the memory to execute the data synchronization method according to any one of claims 1-9 or the data query method according to claim 11.
A computer-readable storage medium stores instructions therein, and when the instructions are executed by a processor, the data synchronization method according to any one of claims 1-9 or the data query method according to claim 11 is executed.