CN115617900A

CN115617900A - Data synchronization method, device, equipment and medium

Info

Publication number: CN115617900A
Application number: CN202110793385.1A
Authority: CN
Inventors: 王家彬; 缪洲光
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-07-12
Filing date: 2021-07-12
Publication date: 2023-01-17

Abstract

The application discloses a data synchronization method, a data synchronization device, data synchronization equipment and a data synchronization medium, wherein the method comprises the following steps: receiving a first query request sent by a synchronization client, wherein the first query request is used for querying target data used for data synchronization, the first query request comprises a data screening condition and a query mode, the data screening condition is used for screening the target data required to be synchronized, the data screening condition comprises a composite index, and the composite index comprises a time field and a sequence number field; according to the first query request, querying target data for data synchronization from a database; and returning the target data to the synchronous client so that the synchronous client writes the target data into a database of the synchronous client to realize data synchronization, and stable and quick data synchronization can be realized.

Description

Data synchronization method, device, equipment and medium

Technical Field

The present disclosure relates generally to the field of database technologies, and more particularly, to a data synchronization method, apparatus, device, and medium.

Background

With the development of technologies such as the internet of things, big data and artificial intelligence, data also grow explosively. How to stably and quickly realize data synchronization is a problem to be solved urgently in the field.

Disclosure of Invention

In view of the above-mentioned drawbacks and deficiencies of the prior art, it is desirable to provide a data synchronization method, apparatus, device and medium, which can achieve stable and fast data synchronization.

In a first aspect, an embodiment of the present application provides a data synchronization method, including:

receiving a first query request sent by a synchronization client, wherein the first query request is used for querying target data used for data synchronization, the first query request comprises a data screening condition and a query mode, the data screening condition is used for screening the target data required to be synchronized, the data screening condition comprises a composite index, and the composite index comprises a time field and a sequence number field;

according to the first query request, querying the target data for data synchronization from a database;

and returning the target data to the synchronization client so that the synchronization client writes the target data into a database of the synchronization client to realize data synchronization.

In a second aspect, an embodiment of the present application provides a data synchronization apparatus, including:

a receiving module, configured to receive a first query request sent by a synchronization client, where the first query request is used to query target data for data synchronization, the first query request includes a data screening condition and a query mode, the data screening condition is used to screen the target data to be synchronized, the data screening condition includes a composite index, and the composite index includes a time field and a sequence number field;

the screening module is used for inquiring the target data for data synchronization from a database according to the first inquiry request;

and the feedback module is used for returning the target data to the synchronous client so that the synchronous client writes the target data into a database of the synchronous client to realize data synchronization.

In some embodiments, the data filtering condition includes a first composite index and a second composite index connected by an or operation, the first composite index including the time field for filtering data generated after a last data generation time synchronized last time, the second composite index including the time field and the sequence number field for filtering data generated at the same time as the last data synchronized last time but ordered after the last data synchronized last time.

In some embodiments, the data screening condition may further include a classification sub-condition, where the classification sub-condition is used to perform secondary screening on the candidate data, and when the data screening condition does not include the classification sub-condition or the data screening condition includes the classification sub-condition and the time for querying the target data is less than or equal to a preset time, the querying manner employs direct querying; a screening module further configured to:

sending the data screening conditions to the database, wherein the database is used for inquiring according to the composite index in the data screening conditions to obtain the candidate data, and carrying out secondary screening on the candidate data according to the classification sub-conditions to obtain the target data meeting the classification sub-conditions;

and receiving the target data screened by the database according to the data screening conditions.

In some embodiments, the data filtering condition may further include a classification sub-condition, where the classification sub-condition is used to perform secondary filtering on the candidate data, and when the data filtering condition includes the classification sub-condition and the time for querying the target data is greater than a preset time, the querying manner is proxy querying, and the filtering module is further configured to:

splitting the data screening condition into the composite index and the classification sub-condition;

sending the composite index to the database;

receiving candidate data screened by the database according to the composite index;

and classifying and screening the candidate data according to the classification sub-conditions to obtain the target data meeting the classification sub-conditions.

In some embodiments, the compound index further includes a read data volume, and when the data volume of the candidate data is greater than the read data volume, the filtering module is further configured to:

updating the composite index according to a time field and a sequence number field corresponding to the last candidate data received currently;

and sending the updated composite index to the database.

In some embodiments, the screening module is further configured to:

adding 1 time granularity to the time field corresponding to the last candidate data to obtain an updated time field;

constructing a third compound index by using the updated time field, and constructing a fourth compound index by using the AND operation between the time field corresponding to the last candidate data and the sequence number field corresponding to the last candidate data;

and performing OR operation on the third composite index and the fourth composite index to obtain the updated composite index.

In some embodiments, the feedback module is further configured to: is connected with

Receiving a second query request, wherein the second query request is used for querying the screening state of the target data;

and when a preset condition is met, responding to the second query request, and returning the target data to the synchronous client, wherein the preset condition comprises at least one of screening the target data and finishing screening the candidate data.

In some embodiments, the data synchronization apparatus further comprises: and (5) a standardization module.

The normalization module is further configured to: and carrying out standardization processing on the data in the database according to the modification time and the modification sequence of the data in the database to obtain a time field label and a serial number field label corresponding to each data in the database.

In a third aspect, embodiments of the present application provide an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the method described in the embodiments of the present application.

In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the method as described in the embodiments of the present application.

According to the method and the device, the first query request which is sent by the synchronous client and comprises the data screening condition and the query mode is received, the target data is screened by utilizing the composite index which is composed of the time field and the serial number field in the data screening condition, so that the screened target data can comprise the data which meets the requirements of the time field and the serial number, and then the target data is returned to the synchronous client, so that the synchronous client writes the target data into the database of the synchronous client to realize data synchronization, the problem of data omission or repeated synchronization in the data synchronization process is effectively avoided, and the reliability of the online data synchronization process is greatly improved.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

fig. 1 is a schematic view of a scenario of a data synchronization method according to an embodiment of the present invention;

fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present application;

fig. 3 is a flowchart of another data synchronization method according to an embodiment of the present application;

fig. 4 is a flowchart of another data synchronization method according to an embodiment of the present application;

fig. 5 is a flowchart of another data synchronization method according to an embodiment of the present application;

fig. 6 is a flowchart of another data synchronization method according to an embodiment of the present application;

fig. 7 is a flowchart of another data synchronization method according to an embodiment of the present application;

fig. 8 is a schematic block diagram of a data synchronization apparatus according to an embodiment of the present application;

fig. 9 is a schematic block diagram of another data synchronization apparatus according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of a computer system suitable for implementing the electronic device or the server according to the embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

All internet applications have a need for data synchronization, especially in situations of capacity cut-in, big data service cooperation co-construction, etc.

For example, the school system contains information of teachers and students, classes, teaching and examinations of the whole school, and a data visualization developer needs the data to develop large-screen application.

Since some data is constantly changing, such as the examination scores of students, the card-punching situations of going to school and going to school, etc., it is necessary to perform online incremental synchronization on the changed data.

However, in the related art, synchronization methods such as paging synchronization and offline synchronization are generally used. Here, page synchronization is to pull the specified offset and size of the data table, but as the offset increases, the time of each query increases linearly, and the overhead on the database also increases linearly. And exporting the data table as a csv or json file during offline synchronization, transmitting the csv or json file by using a physical carrier such as a USB flash disk or an instant chat tool such as an ftp server or an object storage on the cloud or a Wechat tool, and manually importing data at a destination. In most scenes that offline synchronization has to be used, manual operation is needed, delay is large, workload is large, meanwhile, when data volume is large, files exported are large, a large file is difficult to process in any computer system, in addition, the large file generates a large amount of cost through cloud storage, a large amount of fragmented files are accumulated in the past, management is difficult in the later period, efficiency is limited by disk read-write performance, and synchronization efficiency is difficult to improve.

Based on this, embodiments of the present application provide a data synchronization method, apparatus, device, and medium, where the data synchronization apparatus may be specifically integrated in a database management system.

Referring to fig. 1, a schematic view of a scenario of a data synchronization method provided in an embodiment of the present invention, the scenario may include a synchronization client 1, a database management system 2, and a database 3. The synchronous client 1 is a target terminal and is mainly used for providing data synchronization requirements and receiving target data to be synchronized, the database management system 2 is used for standardizing the data in the database and providing proxy service when massive data and/or sparse data are synchronized, namely, a proxy module is established in the database management system 2 to replace the synchronous client to interact with the database 3, data delay is reduced by means of accessing the database 3 through an intranet, and data information can be returned within reasonable time.

The database management system 2 may be disposed in a server, and the server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing a cloud computing service. The synchronization client 1 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a smart television, and the like. The synchronization client 1 and the database management system 2 may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein.

Alternatively, the data synchronization method as disclosed herein, wherein the data may be stored on a blockchain.

For a clearer description of the present application, the following are explanations of terms of the related art:

the Database (Database), which can be regarded as an electronic file cabinet in short, is a place for storing electronic files, and a user can add, query, update, delete, etc. data in the files. A "database" is a collection of data that is stored together in a manner that can be shared by multiple users, has as little redundancy as possible, and is independent of the application.

A Database Management System (DBMS) is a computer software System designed for managing a Database, and generally has basic functions of storage, interception, security assurance, backup, and the like. The database management system may classify the database according to the database model it supports, such as relational, XML (Extensible Markup Language); or classified according to the type of computer supported, e.g., server cluster, mobile phone; or classified according to the Query Language used, such as SQL (Structured Query Language), XQuery; or by performance impulse emphasis, e.g., maximum size, maximum operating speed; or other classification schemes. Regardless of the manner of classification used, some DBMSs are capable of supporting multiple query languages across categories, for example, simultaneously.

Data synchronization: data in one database is migrated to another place to bring the data in both libraries into agreement.

Off-line synchronization: in contrast to online synchronization, source data is typically exported as a file, which is then transmitted over a network and imported at the destination. Such schemes are typically long and slow to complete synchronization tasks.

And (3) online synchronization: usually, both DBs (database files) are online (accessible through network calls), and the data synchronization module directly reads the data source and writes the data source to the destination.

Full quantity synchronization: the data synchronization reads out all data each time and writes the data to a destination.

Incremental synchronization: when data is synchronized, full synchronization is only carried out for the first time, and only the content which is changed or newly added later is read out and written to the destination end each time.

Mass data: with very large amounts of data, a data table may contain hundreds of millions of data.

Supporting screening conditions: only data satisfying the specified condition may be synchronized.

Fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present application. As shown in fig. 2, the data synchronization method includes the steps of:

step 101, receiving a first query request sent by a synchronization client, where the first query request is used to query target data for data synchronization, the first query request includes a data screening condition and a query mode, the data screening condition is used to screen the target data to be synchronized, the data screening condition includes a composite index, and the composite index includes a time field and a sequence number field.

The data screening condition may include at least one compound index, and if the compound index includes only a time field, that is, the index hits all sequence number fields corresponding to the time field included in the index, and if the compound index includes only a sequence number field, that is, the index hits data generated at any time having the sequence number, the compound index may further include both the time field and the sequence number field, that is, the index hits data conforming to both the time field and the sequence number field included in the index. It should be understood that when the compound index contains only one of the time field or the sequence number field, it can also be understood as an omitted writing to the other field.

It should be noted that, before receiving the first query request in step 101, the method further includes: and carrying out standardization processing on the data in the database according to the modification time and the modification sequence of the data in the database to obtain a time field label and a serial number field label corresponding to each data in the database. The addition, modification and deletion of the data belong to modification operation of the data, namely, data standardization processing is carried out according to the time and sequence of the addition, modification and deletion of the data.

It should be understood that, in the present application, the data in the database is standardized, and the purpose of the standardized data is to make each data in the data include a tag of the time field mtime and a tag of the serial number field id, specifically following the following rule:

specifically, for data already stored in the database, a time field tag and a sequence number field tag corresponding to each data are generated according to the time node when the data are stored and the stored sequence.

The data synchronization device provided by the application accesses data externally, so that the id and the mtime can keep the logic of the data in the whole life cycle of data service, namely, the id and the mtime corresponding to the data which are subjected to the data addition, deletion, modification and detection can meet the standardization rule, in other words, the data which are subjected to the data addition, deletion, modification can correspondingly generate corresponding time field labels and sequence number field labels when the data are added or modified.

The application is applicable to general databases such as mysql/mongoddb/postgresql and the like. Further, part of the database may be standardized by a setting command carried by itself, for example, mysql may set mtime to [ ON UPDATE CURRENT _ TIMESTAMP ] using the self-increment key id, so that the database may automatically conform to the standard. That is, the general database generally has a standardized function, and the data in the database can automatically generate the corresponding time field and the sequence number field by using the standardized command of the database.

It should be understood that, because data synchronization is an indispensable link in the internet field, and in consideration of universality, an index cannot be specially created for each synchronization task, a composite index combining time and a sequence number is designed, and online incremental synchronization of mass data can be completed with high performance and high reliability by managing the two fields.

And step 102, inquiring target data for data synchronization from a database according to the first inquiry request.

And 103, returning the target data to the synchronization client so that the synchronization client writes the target data into a database of the synchronization client to realize data synchronization.

That is to say, after the target data used for data synchronization is screened out from the database by using the data screening condition containing the composite index, the database management system returns the target data to the synchronization client, and the synchronization client receives the target data returned by the database management system and writes the target data into the database of the synchronization client, so that the data synchronization between the synchronization client and the database is realized.

The synchronization client can store each data in the target data to a corresponding storage position in a database of the synchronization client according to a preset data storage rule.

Specifically, the method and the device receive a first query request which is sent by a synchronization client and comprises a data screening condition and a query mode, screen target data by using a composite index which is composed of a time field and a sequence number field in the data screening condition, enable the screened target data to comprise both data meeting the requirements of the time field and data meeting the requirements of the sequence number, and then return the target data to the synchronization client, so that the synchronization client writes the target data into a database of the synchronization client to achieve data synchronization, effectively avoid the problem of missing data or repeated synchronization in the data synchronization process, and greatly improve the reliability of the online data synchronization process.

Further, the data filtering condition includes a first compound index and a second compound index connected by an or operation, the first compound index including a time field for filtering data generated after a last data generation time synchronized last time, the second compound index including a time field and a sequence number field for filtering data generated at the same time as the last data synchronized last time but ordered after the last data synchronized last time.

Wherein the generation time includes the time when the data is created and/or the time when the data is updated, and optionally, the data generation time is recorded in the second level.

As a possible embodiment, the time field and the sequence number field may be managed by the following rules, so as to form the data filtering condition:

for the general online increment synchronization, two general screening conditions are subjected to OR operation, and other conditions (sorting conditions, specified indexes and reading quantity) are normally spliced in the query request, so that a general online increment synchronization composite index is formed, and continuous online increment synchronization can be realized only by continuously updating a specific time field and a specific sequence number field when the universal online increment synchronization composite index is used.

For example, a first query request written in sql according to the above rules may be: select from t _ student force index (mtime _ id) where (mtime > 1614592239) or (mtime =1614592239and id > '53102b43bf1044ed8b0ba36 b') order by mtime ASC, id ASC limit 100; the first query request written in mongodb may be: (db) ({ $ or [ { mtime: { $ 1614592239} }, { mtime:1614592239, _ id { $ gt: '53102b43bf1044ed8b0ba36b' } }) ] } sort ({ mtime:1, _ id:1 }). Limit (100). Hit ({ mtime:1, _ id:1 }).

Specifically, it can be expressed as: and querying data which is created or modified after the time of 1614592239 from a student data table, or data which is created or rewritten at the time of 1614592232239 but is ranked after '53102b43bf1044ed8b0ba36b', wherein the queried data is ranked according to the update time stamp mtime, then ranked according to the serial number id, and 100 data results are returned.

For example, the index including the time field and the sequence number field may be visually understood as that all data are sorted according to the time field mtime first, and for the data with the same time field mtime, the data are sorted according to the sequence number id to obtain an array, as shown in the following table:

in this case, an or condition is set to connect the first compound index and the second compound index, so that all the first compound index and the second compound index can hit the index, and therefore, the query efficiency has reached the optimum, taking mtime =2,id = e (i.e., the last time the data is synchronized to the position of zhao xi).

For the first compound index: mtime >2, namely finding the position with the update time stamp of 3 to obtain two data of von ten and xiaoyu.

For the second compound index: mtime =2and id > 'e', namely, the data of the seventy-seven to zheng-nine are obtained by scanning from the first position after e, wherein the updating time stamp is 2and the serial number is.

It can be seen that, by using the composite index designed by the present application, data after the last query can be directly queried in a manner of hit index, that is, the next query can perform data synchronization from the position mtime =3,id = b, and data omission is effectively avoided while ensuring the performance of data synchronization each time. For example, with the composite index proposed in the present application, since the index is hit by the time field, the data that has completed data synchronization but is updated can be resynchronized, thereby avoiding data omission.

Moreover, because the index is hit, the target data does not need to be reordered, the search can be directly performed by 0 (logN), and the scanning index is all effective, so that the response time of data synchronization every time can be completely ensured no matter how large the data to be synchronized is.

Optionally, to avoid server time error, the latest time for synchronization may be limited to a period of time before the current time, for example, 3 seconds, to ensure that data is not missed due to time error. That is, when performing real-time data synchronization, data generated 3 seconds before and 3 seconds after the current time may be synchronized, for example, if the current time is the 5 th s, the first time field may be mtime =2, and the second time field may be mtime =1.

Note that the data synchronization also includes conditional data synchronization, for example, only target data that meets a certain condition among the data is synchronized.

For example, a type (type) attribute of the data is added, and the type is a second classification of the data, that is, the type corresponding to the data can be 0 or 1. The read data amount is 3. There is the following data list:

that is, in order to complete the query task, the database generally needs to scan the entire data table, and when the number of n and m is small, for example, m and n are less than or equal to 100, the time taken for the database to scan the entire data table to screen out the target data can be within the set return time (for example, when the set return time is 5 seconds), and the database can complete the data table scan within 5 seconds, thereby successfully completing the query requirement of the first query request.

However, as n and m increase, the scanning time of the database for the whole data table gradually increases, and the time that the synchronization client can receive the target data returned by the database management system or the database increases with the increase of n and m, even exceeds the preset time (5 seconds), thereby causing the problem of false death of the synchronization client.

Optionally, before determining to select the proxy query, the read volume per batch set in the first query request may be rewritten to reduce the time taken by the database to scan the database table each time, for example, the read volume per batch may be set to 1000 in a normal state, and when performing conditional filtering, the read volume per batch may be set to 100 and lower, so as to reduce the time taken by the database to read the target data per scan, and ensure that the target data can be returned within the set time, and a false death phenomenon caused by the time-out of the return time does not occur. Wherein, the reading amount of each batch can be adjusted according to a linear proportion, but the reading amount of each batch should be more than or equal to 1 and less than or equal to 500.

If the reading amount of each batch is reduced to 1, the target data still cannot be returned within the set time, for example, n is 100 ten thousand, and m is 1 hundred million, at this time, the database needs to scan hundreds of millions of data, which obviously exceeds the set time (5 seconds), that is, a false death phenomenon caused by the overtime of the return time inevitably occurs in the current situation, and at this time, proxy query may be adopted.

It should be understood that the screening condition may be sent to the database by using the synchronization client, to check whether the data return time of the database is over time, and if the data return time is still over time when the number of reads is reduced to 1, the query mode is set as the proxy query in the first query request.

That is to say, the data screening condition may further include a classification sub-condition, the classification sub-condition is used to perform secondary screening on the candidate data, and when the data screening condition does not include the classification sub-condition or the data screening condition includes the classification sub-condition and the time for querying the target data is less than or equal to the preset time, the query mode employs direct query.

As shown in fig. 3, when the query mode may be direct query, querying the target data for data synchronization from the database according to the first query request includes:

and 1021, the data synchronization device sends the data screening conditions to a database, and the database is used for querying according to the composite indexes in the data screening conditions to obtain candidate data and secondarily screening the candidate data according to the classification sub-conditions to obtain target data of the composite classification sub-conditions.

And step 1022, the data synchronization device receives the target data screened by the database according to the data screening condition.

That is to say, the data synchronization device receives a first query request sent by a target terminal, and determines to query the data screening conditions in a direct query manner according to a query manner in the first query request, at this time, the data synchronization device sends the data screening conditions in the first query request to the database, when the data screening conditions do not include a classification condition, the database performs data query according to a first composite index and a second composite index in the data screening conditions, so as to obtain a first target data set corresponding to the first composite index and a second target data set corresponding to the second composite index, then sorts the data in the first target data set and the second target data set according to time fields, sorts a plurality of data corresponding to each time field according to serial number fields, so as to obtain candidate data, selects data corresponding to the read number from the candidate data, and feeds the data back to the data synchronization device as final target data, and the data synchronization device returns the target data to the synchronization client, so as to achieve the purpose of synchronizing the data in the database to the synchronization client.

Or when the data screening condition comprises a classification sub-condition and the query time is less than or equal to the preset time, the database firstly queries data according to a first compound index and a second compound index in the data screening condition respectively to obtain a first target data set corresponding to the first compound index and a second target data set corresponding to the second compound index, secondly screens the first target data set and the second target data set respectively according to the classification sub-condition to obtain final target data, thirdly sorts the target data according to a first time field and a second sequence number field, and feeds back the sorting result to the synchronous client. And the first target data set and the second target data set can be sorted to obtain candidate data, and then the sorting result is screened for the second time, which is not described herein again.

Optionally, the first query request may further include address information of the synchronization client, and the database may directly return the target data to the synchronization client according to the address information of the synchronization client after querying the target data.

However, when the data filtering condition includes the classification sub-condition and the time for querying the target data is longer than the preset time, it indicates that the synchronization client inevitably appears false death because the synchronization client cannot receive the return information within the preset time, therefore, the proxy module in the data synchronization device can query in a proxy query manner, so as to maintain normal communication with the synchronization client, prevent the synchronization client from false death, and meanwhile, the proxy module waits for the database to query data according to the filtering condition, or the proxy module performs secondary filtering on the candidate data screened according to the first composite index and the second composite index by using the classification sub-condition, so as to ensure the reliability of the data query.

Further, as shown in fig. 4, the data filtering condition may further include a classification sub-condition, where the classification sub-condition is used to perform secondary filtering on the candidate data, and when the data filtering condition includes the classification sub-condition and the time for querying the target data is greater than the preset time, the querying manner is proxy querying, and the target data for data synchronization is queried from the database according to the first query request.

According to the first query request, target data used for data synchronization is queried from a database, and the method comprises the following steps:

and 1023, splitting the data screening condition into a compound index and a classification sub-condition.

It should be understood that the classification sub-condition may be set according to practical situations, and the classification sub-condition may be an infinite condition when performing general incremental synchronization.

Step 1024, the compound index is sent to a database.

And step 1025, receiving the candidate data screened by the database according to the composite index.

It should be appreciated that since the compound index mainly includes a time field and a sequence number field, a certain number of candidate data conforming to a time range and a sequence range can be hit and obtained by using the compound index.

It should be noted that, when the agent query is used, the data screening synchronization during the increment synchronization may be performed, that is, the incremental data is conditionally screened and then synchronized, or the data screening may be for destination data, that is, the screened database table may include data that has been synchronized, for example, first target data that has been synchronized in the database table a according to the first classification sub-condition and that meets the first classification sub-condition, at this time, the database table a is synchronized again according to the second classification sub-condition to synchronize second target data that meets the second classification sub-condition, and the second classification sub-condition may be different from, may be the same as, or partially the same as the first classification sub-condition. That is, the proxy query may not be limited to the case of incremental synchronization.

And step 1026, classifying and screening the candidate data according to the classification sub-conditions to obtain target data meeting the classification sub-conditions.

In conditional data screening, a composite index is usually used to limit a time range and a sequence number range of data to be screened, a data synchronization device first obtains candidate data satisfying the time range and the sequence number range by using the composite index, and then an agent module local to the data synchronization device performs secondary screening on the candidate data by using a classification sub-condition to obtain final target data.

Specifically, the data synchronization device receives a first query request sent by a synchronization client, and then splits a data screening condition in the first query request into a composite index and a classification sub-condition, wherein the composite index comprises a first composite index and a second composite index, the data synchronization device sends the composite index to a database, so that the database can query according to the first composite index and the second composite index respectively to obtain a first target data set and a second target data set, then the database sorts the data in the first target data set and the second target data set according to a time-first and sequence-second mode to obtain candidate data, and returns the candidate data to the data synchronization device, the data synchronization device receives the candidate data returned by the database, and performs secondary screening on the candidate data according to the classification sub-condition, screens out the data meeting the classification sub-condition as final target data, and feeds the target data back to the synchronization client.

Therefore, the access to the database is executed by using the data synchronization device in the database system, the access speed of mass data is effectively improved based on the database internal network, and the delay time of data acquisition is effectively reduced. And moreover, the data synchronization device in the database system is used for carrying out secondary screening on the candidate data, so that the calculation amount of the database can be effectively reduced, and the overall speed of data screening is effectively improved.

As a possible embodiment, the composite index further includes a read data volume, where the read data volume is the number of candidate data that can be read from the database in each batch when the candidate data are read from the database in batches.

When the data volume of the candidate data is larger than the read data volume, sending the composite index to the database, wherein the step of sending the composite index to the database comprises the following steps: and updating the composite index according to the time field and the sequence number field corresponding to the last candidate data received currently, and sending the updated composite index to the database.

That is, when the data amount of the candidate data is larger than the read data amount, it means that the candidate data cannot be read from the database to the data synchronization apparatus at one time, that is, the data synchronization apparatus needs to synchronize the candidate data to the data synchronization apparatus sequentially by a plurality of times. At this time, the process of multiple data synchronization may be converted into continuous online incremental synchronization operation, and therefore, the composite index of the next data synchronization needs to be updated according to the time field and the sequence number field of the last data after each synchronization, so as to achieve the purpose of continuously synchronizing the candidate data from the database.

Further, as shown in fig. 5, the updating of the compound index includes:

step 201, increasing the time field corresponding to the last candidate data by 1 time granularity to obtain the updated time field.

The time granularity is a time level adopted when the database data is standardized, for example, when the data is standardized by using time on the order of seconds, 1 time granularity may correspond to 1 second. For example, when the time field corresponding to the last candidate data is mtime =2, the updated time field is mtime =3. Alternatively, when data normalization is performed on the millisecond level, 1 time granularity may correspond to 1 millisecond, and in this case, the time field corresponding to the last data candidate is mtime =2ms, and then the updated time field is mtime =3ms. In the present embodiment, 2s and 3s refer to second and third seconds, and 2ms and 3ms refer to second and third milliseconds.

Step 202, a third compound index is constructed by using the updated time period, and a fourth compound index is constructed by using and operation between the time field corresponding to the last candidate data and the updated sequence number field.

It should be noted that the third compound index is used to filter data generated after the time of generating the last candidate data, and the fourth compound index is used to filter data generated at the same time as the last candidate data and ranked after the last data.

And 203, performing OR operation on the third composite index and the fourth composite index to obtain an updated composite index.

That is to say, by updating the composite index, the third composite index and the fourth composite index are obtained, the next synchronization requirement can be effectively converted to start after the last candidate data synchronized last time, and it is effectively ensured that the candidate data synchronized by the updated composite index can be kept continuous with the candidate data synchronized last time in time and sequence number, and no data is left.

Specifically, taking an example of data normalization using second-level time and an english alphabet sequence, when the time field corresponding to the last candidate data is 2and the serial number is a, that is, mtime =2, and the time field after updating is mtime =3, the third composite index is constructed to be mitime >3, and the fourth composite index is mtime =2and id >.

Therefore, the candidate data can be ensured to be synchronized from the database in sequence according to the time sequence and the sequence number sequence in the multiple data synchronization process by using the updated composite index.

As an alternative embodiment, as shown in fig. 6, returning the target data to the synchronization client includes:

step 301, receiving a first query request, where the first query request is used to query target data for data synchronization, the first query request includes a data screening condition and a query mode, the data screening condition is used to screen the target data to be synchronized, the data screening condition includes a composite index, and the composite index includes a time field and/or a sequence number field.

Step 302, according to the first query request, target data for data synchronization is queried from a database.

Step 303, receiving a second query request, where the second query request is used to query the screening status of the target data.

And 304, when a preset condition is met, responding to the second query request, and returning the target data to the synchronous client, wherein the preset condition comprises at least one of screening the target data and finishing the screening of the candidate data.

That is to say, the data synchronization device queries, after receiving the first query request, target data for data synchronization from the database according to the first query request, and then the target terminal may send a second query request to the data synchronization device according to a certain frequency to query a data synchronization state, and when the data synchronization device screens out at least one target data or completes the screening of all data corresponding to the composite index, the data synchronization device responds to the second query request and returns a screening result.

The screening of the target data may include screening all the target data, or screening at least one target data, that is, the data synchronization device may return one target data to the target terminal based on the second query request each time one target data is screened, or return the target data to the target terminal based on the second query request when the screening of all candidate data is completed, that is, when the data screening condition is completed (there is no more target data).

As a specific embodiment, as shown in fig. 7, a data synchronization method includes:

step 401, a target terminal carries a first query request including a data screening condition and a query mode to initiate a data synchronization task.

Step 402, the data synchronization device returns task confirmation information to the target terminal.

In step 403, the data synchronization apparatus initiates a data synchronization request to the database using the composite index in the data screening condition.

In step 404, the data synchronization device receives the candidate data returned by the database.

In step 405, the data synchronization apparatus performs secondary screening on the candidate data returned from the current batch by using the classification sub-condition.

The screening result can comprise target data and target data which are not obtained, and the target data obtained by screening are cached when the target data are obtained.

Step 406, the data synchronization device updates the composite index and initiates a data synchronization request to the database by using the updated composite index.

In step 407, the data synchronization apparatus performs secondary screening on the candidate data returned from the current batch by using the classification sub-condition.

And repeatedly executing the steps 406 to 407 until the screening condition is completely executed.

It should be understood that the composite index in step 403 includes a time field and a sequence number field of the initial data for data scanning sent by the target terminal, step 406 is to send the composite index for the second time after the initial data synchronization, that is, the time field and the sequence number field of the current initial position data need to be updated according to the candidate data returned in step 404, and the subsequent steps that are repeatedly executed are also to update the composite index according to the candidate data returned in the previous time, so as to achieve the effect of sequentially synchronizing the data in the database.

Step 408, the target terminal sends a second query request to the data synchronization apparatus after the first preset time.

In step 409, the data synchronization apparatus returns the incomplete status information to the target terminal.

Step 410, the target terminal sends a second query request to the data synchronization apparatus after a second preset time.

The second preset time is less than or equal to the first preset time, for example, the first preset time may be 5 seconds, and the second preset time may be 1 second.

In step 411, the data synchronizer returns the incomplete status information to the target terminal.

And repeatedly executing the steps 410 to 411 until receiving the status information that the data synchronization device returns completion and receiving the target data returned by the data synchronization device.

Wherein, there is no time sequence between steps 403-407 and steps 408-411, that is, any step in steps 403-407 may have the same time as any step in steps 408-411.

It should be noted that, after completing the pair screening, the data synchronization device may store the cached target data in the release queue, and when the storage time of the target data reaches the preset time or queues to the release position, the data synchronization device deletes the stored target data to complete the release.

To sum up, the method and the device for data synchronization receive a first query request which is sent by a synchronization client and comprises a data screening condition and a query mode, screen target data by utilizing a composite index which is composed of a time field and a sequence number field in the data screening condition, enable the screened target data to comprise both data meeting the requirements of the time field and data meeting the requirements of the sequence number, and then return the target data to the synchronization client, so that the synchronization client writes the target data into a database of the synchronization client to realize data synchronization, effectively avoid the problem of missing data or repeated synchronization in the data synchronization process, and greatly improve the reliability of the online data synchronization process.

It should be noted that while the operations of the methods of the present invention are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, in order to achieve desirable results.

Further, fig. 8 is a block diagram of a data synchronization apparatus according to an embodiment of the present application.

As shown in fig. 8, the data synchronization apparatus 10 includes:

a receiving module 11, configured to receive a first query request sent by a synchronization client, where the first query request is used to query target data for data synchronization, the first query request includes a data screening condition and a query mode, the data screening condition is used to screen the target data to be synchronized, the data screening condition includes a composite index, and the composite index includes a time field and a sequence number field;

the screening module 12 is configured to query, according to the first query request, target data for data synchronization from a database;

and the feedback module 13 is configured to return the target data to the synchronization client, so that the synchronization client writes the target data into a database of the synchronization client to implement data synchronization.

In some embodiments, the data filtering condition includes a first composite index and a second composite index connected by an or operation, the first composite index including a time field for filtering data generated after a last data generation time synchronized last time, the second composite index including a time field and a sequence number field for filtering data generated at the same time as the last data synchronized last time but ordered after the last data synchronized last time.

In some embodiments, the data screening condition may further include a classification sub-condition, where the classification sub-condition is used to perform secondary screening on the candidate data, and when the data screening condition does not include the classification sub-condition or the data screening condition includes the classification sub-condition and the time for querying the target data is less than or equal to a preset time, the querying manner employs direct querying;

sending the data screening conditions to a database, wherein the database is used for inquiring according to the composite index in the data screening conditions to obtain candidate data, and secondarily screening the candidate data according to the classification sub-conditions to obtain target data meeting the classification sub-conditions;

and receiving target data screened by the database according to the data screening conditions.

In some embodiments, the data filtering condition may further include a classification sub-condition, where the classification sub-condition is used to perform secondary filtering on the candidate data, and when the data filtering condition includes the classification sub-condition and the time for querying the target data is greater than the preset time, the querying manner is proxy querying, and the filtering module 12 is further configured to:

splitting the data screening condition into a composite index and a classification sub-condition;

sending the composite index to a database;

receiving candidate data screened out by the database according to the composite index;

and classifying and screening the candidate data according to the classification sub-conditions to obtain target data meeting the classification sub-conditions.

In some embodiments, screening module 12 is further configured to:

and sending the updated composite index to a database.

In some embodiments, screening module 12 is further configured to:

increasing 1 time granularity for a time field corresponding to the last candidate data to obtain an updated time field;

and performing OR operation on the third composite index and the fourth composite index to obtain an updated composite index.

In some embodiments, the receiving module 11 is further configured to: receiving a second query request, wherein the second query request is used for querying the screening state of the target data;

a feedback module 13, further configured to: and when the preset condition is met, responding to the second query request, and returning the target data to the synchronous client, wherein the preset condition comprises at least one of screening the target data and finishing screening the candidate data.

In some embodiments, the data synchronization apparatus 10 further includes:

and the standardization module 14 is used for carrying out standardization processing on the data in the database.

It should be understood that the units or modules recited in the data synchronization device 10 correspond to the various steps in the method described with reference to fig. 2. Thus, the operations and features described above with respect to the method are equally applicable to the data synchronization apparatus 10 and the units included therein, and are not described in detail here. The data synchronization apparatus 10 may be implemented in a browser or other security applications of the electronic device in advance, or may be loaded into the browser or other security applications of the electronic device by downloading or the like. The corresponding units in the data synchronization device 10 may cooperate with units in the electronic device to implement the solution of the embodiments of the present application.

Several modules or units have been mentioned in the above detailed description, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

It should be noted that details that are not disclosed in the data synchronization apparatus of the embodiment of the present application refer to details disclosed in the above embodiments of the present application, and are not described herein again.

Referring now to fig. 10, fig. 10 illustrates a schematic diagram of a computer system suitable for use to implement an electronic device or server of an embodiment of the present application,

as shown in fig. 10, the computer system includes a Central Processing Unit (CPU) 1001 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1002 or a program loaded from a storage section 1008 into a Random Access Memory (RAM) 1003. The RAM1003 also stores various programs and data necessary for operation instructions of the system. The CPU1001, ROM1002, and RAM1003 are connected to each other via a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.

The following components are connected to the I/O interface 1005; an input portion 1006 including a keyboard, a mouse, and the like; an output section 1007 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card, a modem, or the like. The communication section 1009 performs communication processing via a network such as the internet. A drive 1010 is also connected to the I/O interface 1005 as necessary. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1010 as necessary, so that a computer program read out therefrom is mounted into the storage section 1008 as necessary.

In particular, according to embodiments of the present application, the process described above with reference to the flowchart fig. 2 may be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program comprises program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication part 1009 and/or installed from the removable medium 1011. The computer program executes the above-described functions defined in the system of the present application when executed by the Central Processing Unit (CPU) 1001.

It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operational instructions of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules described in the embodiments of the present application may be implemented by software or hardware. The described units or modules may also be provided in a processor, and may be described as: a processor includes a receiving module and a screening module. The names of these units or modules do not form a limitation on the units or modules themselves in some cases, for example, a receiving module may also be described as "receiving a first query request, the first query request is used for querying target data for data synchronization, the first query request includes a data screening condition and a query mode, the data screening condition is used for screening the target data needing synchronization, the data screening condition includes a composite index, and the composite index includes a time field and/or a sequence number field".

As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment, or may exist separately without being assembled into the electronic device. The computer-readable storage medium stores one or more programs which, when executed by one or more processors, perform the data synchronization methods described herein.

The foregoing description is only exemplary of the preferred embodiments of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other combinations of features described above or their equivalents without departing from the spirit of the disclosure. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

Claims

1. A method of data synchronization, comprising:

2. The method of claim 1, wherein the data filtering condition comprises a first composite index and a second composite index connected by an or operation, the first composite index comprising the time field for filtering data generated after a last data generation time synchronized last time, the second composite index comprising the time field and the sequence number field for filtering data generated at the same time as the last data synchronized last time but ordered after the last data synchronized last time.

3. The method according to claim 1, wherein the data filtering condition further includes a classification sub-condition, the classification sub-condition is used for performing secondary filtering on the candidate data, and when the data filtering condition does not include the classification sub-condition or the data filtering condition includes the classification sub-condition and the time for querying the target data is less than or equal to a preset time, the query mode is direct query;

the querying the target data for data synchronization from a database according to the first query request includes:

4. The method according to claim 1, wherein the data filtering condition further includes a classification sub-condition, the classification sub-condition is used for performing secondary filtering on the candidate data, when the data filtering condition includes the classification sub-condition and the time for querying the target data is greater than a preset time, the querying manner is proxy querying, and querying the target data for data synchronization from the database according to the first query request includes:

sending the composite index to the database;

5. The method of claim 4, wherein the compound index further comprises a read data volume, and wherein sending the compound index to the database when the data volume of the candidate data is greater than the read data volume comprises:

and sending the updated composite index to the database.

6. The method of claim 5, wherein the updating the composite index comprises:

7. The method according to any of claims 4-6, wherein said returning said target data to said synchronization client comprises:

8. The method of claim 1, prior to said receiving the first query request, further comprising:

and carrying out standardization processing on the data in the database according to the modification time and the modification sequence of the data in the database to obtain a time field label and a serial number field label corresponding to each data in the database.

9. A data synchronization apparatus, comprising:

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, implements the data synchronization method of any of claims 1-8.

11. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the data synchronization method according to any one of claims 1 to 8.