CN112214500A

CN112214500A - Data comparison method and device, electronic equipment and storage medium

Info

Publication number: CN112214500A
Application number: CN201910613849.9A
Authority: CN
Inventors: 邓力凡
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2019-07-09
Filing date: 2019-07-09
Publication date: 2021-01-12

Abstract

The invention discloses a data comparison method, a data comparison device, electronic equipment and a storage medium, and relates to the technical field of computers. One embodiment of the method comprises: receiving a data updating message pushed by a first database, wherein the data updating message comprises updated data and a target field corresponding to the updated data; determining a target comparison strategy corresponding to the target field according to the mapping relation between the field and the comparison strategy; acquiring comparison data corresponding to the target field from a second database according to the target field; wherein the second database has completed data synchronization based on the first database; and comparing the comparison data with the updated data through a target comparison strategy to obtain a comparison result. The embodiment can reduce the access to the service system database during data comparison, and avoid the loss of the service system.

Description

Data comparison method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for data comparison, an electronic device, and a storage medium.

Background

With the rapid development of computer network services, a single database storage technology cannot support the requirement of data storage during the operation of a business system, so technicians usually use a multi-data source storage manner to support data storage of the business system, such as a Remote Dictionary Server (Remote Dictionary Server) distributed cache manner. However, when data is stored in a multi-data source storage manner, in order to ensure normal operation of the service system, the data of the service system needs to be synchronously stored in the original database and other databases, and the stored data in the original database is ensured to be consistent with the stored data in the other databases. Therefore, in order to ensure the normal operation of the service system, the consistency comparison between the data stored in the database and the data stored in the multiple data sources is required.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

at present, the data comparison is usually realized by comparing the original database with the full data or the timing increment of other databases in a timing manner. When data is compared, data to be compared is usually extracted from each database, so that the original database of the business system needs to be frequently accessed, which brings great pressure on resource usage of the original database. Especially, when the service system has a high resource utilization frequency for the original database, if the data to be compared is often obtained from the original database, the storage resource of the service system is seized, so that the performance of the service system accessing the original database is reduced, and the performance of the service system is lost.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method, an apparatus, a system, and a storage medium for data comparison, which can reduce access to an original database of a service system during data comparison, and avoid performance degradation of the original database accessed by the service system due to preemption of storage resources of the service system.

To achieve the above object, according to an aspect of an embodiment of the present invention, a method of data comparison is provided.

The data comparison method of the embodiment of the invention comprises the following steps: receiving a data updating message pushed by a first database, wherein the data updating message comprises updated data and a target field corresponding to the updated data; determining a target comparison strategy corresponding to the target field according to the mapping relation between the field and the comparison strategy; acquiring comparison data corresponding to the target field from a second database according to the target field; wherein the second database has completed data synchronization based on the first database; and comparing the comparison data with the updated data through the target comparison strategy to obtain a comparison result.

In one embodiment, the target field comprises a plurality of fields, and the updated data comprises updated data corresponding to each field in the plurality of fields; determining a target comparison strategy corresponding to the target field according to the mapping relation between the field and the comparison strategy, wherein the target comparison strategy comprises the following steps: respectively determining a target comparison strategy corresponding to each field according to the mapping relation between the field and the comparison strategy; comparing the comparison data with the updated data by the target comparison policy, including: and for each field in the plurality of fields, comparing the comparison data corresponding to each field with the updated data corresponding to each field through the target comparison strategy corresponding to each field.

In one embodiment, after the obtaining the comparison result, the method further includes: judging whether the comparison data and the updated data are consistent or not based on the comparison result; if not, sending an alarm message to an alarm system, and storing the comparison result; if yes, the comparison result is saved.

In one embodiment, before the comparing the comparison data and the updated data by the target comparison policy, the method further includes: and loading the target comparison strategy from a comparison strategy database, wherein the comparison strategy database is configured in advance.

To achieve the above object, according to another aspect of the present invention, there is provided an apparatus for data comparison.

The data comparison device of the invention comprises: the receiving module is used for receiving a data updating message pushed by a first database, wherein the data updating message comprises updated data and a target field corresponding to the updated data; the determining module is used for determining a target comparison strategy corresponding to the target field according to the mapping relation between the field and the comparison strategy; the acquisition module is used for acquiring comparison data corresponding to the target field from a second database according to the target field; wherein the second database has completed data synchronization based on the first database; and the comparison module is used for comparing the comparison data with the updated data through the target comparison strategy to obtain a comparison result.

In one embodiment, the target field comprises a plurality of fields, and the updated data comprises updated data corresponding to each field in the plurality of fields; the determination module is further to: respectively determining a target comparison strategy corresponding to each field according to the mapping relation between the field and the comparison strategy; the comparison module is further configured to: and for each field in the plurality of fields, comparing the comparison data corresponding to each field with the updated data corresponding to each field through the target comparison strategy corresponding to each field.

In one embodiment, further comprising: the judging module is used for judging whether the comparison data and the updated data are consistent or not based on the comparison result; the alarm module is used for sending an alarm message to the alarm system if the alarm module does not send the alarm message to the alarm system; the storage module is used for storing the comparison result if the comparison result is not the same as the comparison result; and the storage module is also used for storing the comparison result if the comparison result is positive.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided an electronic apparatus.

An electronic device of an embodiment of the present invention includes: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the data comparison method provided by the embodiment of the invention.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer-readable medium.

A computer-readable medium of an embodiment of the present invention stores thereon a computer program, which when executed by a processor implements the method for data comparison provided by an embodiment of the present invention.

One embodiment of the above invention has the following advantages or benefits: in the embodiment of the invention, a data updating message pushed by the first database can be received, wherein the data updating message comprises updated data and target fields corresponding to the updated data, so that which fields in the first database are updated and the updated data can be determined according to the data updating message; when the subsequent data comparison is carried out, the comparison data is only obtained from the second database for carrying out the data comparison, and the updated data does not need to be obtained from the first database, so that the access to the first database is reduced, the occupation of the storage resource of the service system is reduced, and the performance of the service system for accessing the original database due to the occupation of the storage resource of the service system is avoided.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of a system architecture of a method of data comparison according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a main flow of a method of data comparison according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating an alternative system architecture for a method of data comparison according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of yet another major flow of a method of data comparison according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of yet another major flow of a method of data comparison according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of the main blocks of a data comparison apparatus according to an embodiment of the present invention;

FIG. 7 is a diagram of yet another exemplary system architecture to which embodiments of the present invention may be applied;

FIG. 8 is a schematic block diagram of a computer system suitable for use in implementing embodiments of the present invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.

The embodiment of the invention provides a data comparison system which can be used for comparing data stored in an original database with data stored in other databases when data storage with multiple data sources is adopted. In particular as shown in figure 1. In the system architecture shown in fig. 1, a service system 101 performs data storage by using a multi-data-source storage manner, where during operation, the service system 101 writes data to be stored into an original database, that is, a database 102, and simultaneously needs to store updated data into other databases, where the other databases may include one or more databases, and fig. 1 illustrates an example that one database (that is, a database 103) is included. During the operation of the service system 101, the data to be stored may be directly written into the database 102, and the database 103 may complete data synchronization based on the data in the database 102, that is, synchronize the data in the database 102. In order to ensure the normal operation of the business system, the data in the database 102 and the database 103 need to be consistent, so the data in the database 102 and the database 103 need to be compared in a consistent manner.

The data comparison system 104 is respectively connected with the database 102 and the database 103, and can perform data interaction. In the embodiment of the present invention, after data update occurs in the service system 101, the service system 101 stores the updated data in the database 102, and the database 103 synchronizes the data in the database 102. The database 102 stores the updated data, and may also push a data update message to the data comparison system 104, where the data update message includes specific data of the data update. The data comparison system 104 may determine specific data of data update in the database 102 according to the data update message, and further may obtain comparison data for comparing with the specific data updated in the database 102 from the database 103 based on the data update message, and perform data comparison to determine whether the data in the database 102 is consistent with the data in the database 103. In this way, the data comparison system 104 can avoid obtaining updated data of the service system 101 from the database 102, reduce access to the database 102, reduce occupation of storage resources of the service system 101, and reduce influence on the service system.

An embodiment of the present invention provides a data comparison method, which may be executed by a data comparison system in a system architecture shown in fig. 1, as shown in fig. 2, where the method includes:

s201: and receiving a data updating message pushed by the first database.

Wherein the data update message is pushed by the first database. When the data in the first database is updated, a data update message can be sent to the data comparison system. The data update message includes the updated data and a target field corresponding to the updated data. In connection with the system architecture shown in fig. 1, the first database may be database 102.

The data update may include operations such as adding data, or modifying or deleting original data. Data updates often occur in business systems during the course of processing a business. When the data is updated, the service system writes the updated data into the first database and simultaneously stores the updated data into other databases. When the first database writes the updated data, the first database may be triggered to generate a data update message, where the data update message includes a field where the data update occurs and the updated data. After the first database generates the data update message, the first database may push the data update message to the data comparison system. The data comparison system and the first database can establish communication connection directly or through message middleware, and further can receive data updating messages from the first database.

Take an example that the service system stores data in a distributed cache manner, and the first database is mysql (relational database management system). When data in the mysql is changed, a binlog (binary log) mechanism can be triggered in the mysql to generate the binlog, and the change condition of the data in the database is recorded in the binlog, so the binlog can be used as a data updating message in the step. mysql, upon triggering the generation of the binlog, may send the generated binlog to the data comparison system, i.e., the data comparison system receives the binlog. Specifically, the binlog includes data update records, for example, a target field corresponding to the data to be updated, data before the update, data after the update, and the like. Due to the fact that the binlog mechanism in mysql is used, the binlog can be rapidly transmitted to the data comparison system, and therefore the requirement of data comparison in near real time can be met.

S202: and determining a target comparison strategy corresponding to the target field according to the mapping relation between the field and the comparison strategy.

The data comparison policy may include a plurality of policies, for example, a k-v data structure comparison policy, an out-of-order array structure comparison policy, a string comparison policy, an ordered array structure comparison policy, and the like. Because different data structures or different data types adopt different data comparison strategies during data comparison, the mapping relation between the fields and the comparison strategies can be configured in advance in the data comparison system, and then the target comparison strategies corresponding to the target fields are determined according to the mapping relation between the fields and the comparison strategies.

In an implementation manner of the embodiment of the present invention, the data comparison system may also pre-configure a comparison policy required for data comparison. For example, the data comparison policy required for data comparison may be stored by pre-establishing a comparison policy database, so that after the target comparison policy is determined in step S202, the target comparison policy may be directly loaded from the comparison policy database to perform a subsequent data comparison process.

It should be noted that, for reasons such as different naming rules or different database storage manners, the name of the field in the database of the service system may be different from the name of the same field in the data comparison system, so that after the data comparison system receives the data update message in step S201, the data comparison system may also perform preprocessing on the target field in the data update message, for example, convert the name of the target field into a name that can be recognized by the data comparison system, so as to perform the subsequent data comparison process.

S203: and acquiring comparison data corresponding to the target field from the second database according to the target field.

Wherein the second database has completed data synchronization based on the first database.

The service system stores data by adopting a multi-data-source data storage mode, when the data is updated, the service system writes the updated data into the first database, meanwhile, other databases perform data synchronization based on the first database, the second database may be one or more of the other databases, and in combination with the system architecture shown in fig. 1, the second database may be the database 103.

The data comparison system can determine the updated data in the first database through the data update message, and then needs to acquire the data for data comparison in other databases storing the service system data, that is, the comparison data corresponding to the target field is acquired from the second database according to the target field.

S204: and comparing the comparison data with the updated data through the target comparison strategy to obtain a comparison result.

After the comparison data is obtained in step S203, the data comparison system may compare the comparison data with the updated data by using the determined target comparison policy, so as to obtain a comparison result, and determine whether the data in the first database is consistent with the data in the second database based on the comparison result.

In an implementation manner of the embodiment of the present invention, since the service system may update multiple items of data during running, the data update message may further include content of updating multiple pieces of data, that is, the target field includes multiple fields, and the updated data includes updated data corresponding to each field in the multiple fields. At this time, step S202 may be specifically executed as: determining a target comparison strategy corresponding to each field in the fields according to the mapping relation between the fields and the comparison strategy; step S204 may be specifically performed as: and for each field in the plurality of fields, comparing the comparison data corresponding to each field with the updated data corresponding to each field through the target comparison strategy corresponding to each field.

When the target field includes a plurality of fields, the comparison policy corresponding to each field is not necessarily the same, so the data comparison system in step S202 needs to determine the target comparison policy corresponding to each field according to the mapping relationship between the fields and the comparison policy, so as to perform data comparison on each field. Then, the data comparison system obtains the comparison data corresponding to each field from the second database through step S203, and compares the comparison data corresponding to each field with the updated data corresponding to each field through the target comparison policy corresponding to each field, thereby determining whether the data in the first database is consistent with the data in the second database. After the data comparison system compares the comparison data corresponding to each field with the updated data corresponding to each field, the comparison results may be summarized to determine whether the data in the first database is consistent with the data in the second database.

In an implementation manner of the embodiment of the present invention, after the data comparison system executes step S204, the comparison result between the comparison data and the updated data in step S204 may also be saved.

The data comparison system can store the result of the data comparison in the step S204, and thus, the data comparison system stores the result of each data comparison, so that each data comparison has a record which can be traced, and the data comparison system can be convenient for the subsequent check of workers and the maintenance of the data in the first database and the second database. The data comparison system may store the comparison result in various ways, for example, the comparison result may be stored by storing a snapshot.

In an implementation manner of the embodiment of the present invention, after the data comparison system executes step S204, it may further determine whether the comparison data is consistent with the updated data based on the comparison result; and when the comparison data is inconsistent with the target data, executing an alarm process.

After the data comparison result is obtained in step S204, the data comparison system may determine whether the comparison data is consistent with the updated data according to the comparison result. When the comparison data is inconsistent with the updated data, the data stored in the first database is inconsistent with the data stored in the second database, which easily affects the normal operation of the service system, so that the data comparison system can also execute an alarm process, so that the staff can be informed to maintain in time, and the normal operation of the service system is ensured.

In an implementation manner of the embodiment of the present invention, after the data comparison result is obtained by the data comparison system in step S204, the comparison result may be displayed, so that the comparison result may be visually presented to the worker, and the worker may simply and conveniently view the comparison result.

It should be noted that, in the embodiment of the present invention, the manner of pushing the data update message by the first database may be real-time pushing, that is, the first database pushes the data update message to the data comparison system when data update occurs, so that the data comparison system can compare the updated data in the first database with the updated data in the second database as soon as possible, and thus, when the data in the first database is inconsistent with the data in the second database, the updated data can be found in time, thereby avoiding the influence on the service system as much as possible.

In the embodiment of the invention, a data updating message pushed by the first database can be received, wherein the data updating message comprises updated data and target fields corresponding to the updated data, so that which fields in the first database are updated and the updated data can be determined according to the data updating message; when the subsequent data comparison is carried out, the comparison data is only obtained from the second database for carrying out the data comparison, and the updated data does not need to be obtained from the first database, so that the access to the first database is reduced, the occupation of the storage resource of the service system is reduced, and the performance of the service system for accessing the original database due to the occupation of the storage resource of the service system is avoided.

In addition, in the embodiment of the invention, the data comparison among the multiple databases of the business system is realized in an asynchronous mode without code intrusion and any business intrusion, and meanwhile, other complete matching capabilities such as historical comparison record storage and query, abnormal alarm and the like can be provided. After the business system is accessed to the data comparison system, the data comparison system can operate the data comparison function, configuration change needs to be carried out on the data comparison system only when the business system increases or deletes the storage fields of the database, and the data comparison system can automatically complete the data comparison peer-to-peer operation at other times, so that the workload of development and testing personnel is reduced, and the operation quality of the business system is improved.

With reference to the system architecture shown in fig. 1, in the embodiment of the present invention, based on the system architecture shown in fig. 1, an example is described in which the database 102 is a mysql database, and the database 103 is Redis, which may be specifically shown in fig. 3. In the system architecture shown in fig. 3, the business system writes data to be stored into mysql during operation, and the Redis may complete data synchronization based on the data in the mysql. The data comparison system is respectively connected with mysql and Redis, and data interaction can be carried out. In the embodiment of the invention, after data is updated in the service system, the updated data is stored to the mysql, the Redis synchronizes the data in the mysql, when the mysql stores the data of the updated data, the binlog can be pushed to the data comparison system, the specific data of the data update is recorded in the binlog, the data comparison system can determine the specific data of the data update in the mysql according to the binlog, and then the data comparison system can acquire the comparison data for comparing with the specific data updated in the mysql from the Redis based on the binlog and perform data comparison, so as to determine whether the data in the mysql is consistent with the data in the Redis, so that the data comparison system can avoid acquiring the updated data of the service system from the mysql, reduce the access to the mysql, reduce the occupation of storage resources of the service system, and reduce the influence on the service system.

In addition, in the system architecture shown in fig. 3, a comparison policy database for storing the comparison policy, a database for storing the comparison result, and an alarm system are further included, where the database for storing the comparison result may be HBase (a distributed and column-oriented open source database), and the like, and fig. 3 illustrates HBase as an example. And the comparison strategy database, the HBase and the alarm system are connected with the data comparison system.

Fields used for data comparison in the mysql and the data comparison strategies corresponding to the fields can be configured in advance in the comparison strategy database, so that the data comparison system can directly load the required comparison strategies from the fields during data comparison. The comparison strategy database can adopt a distributed cache mode to store the pre-configured comparison strategy. In order to enable the comparison strategy database to support flexible configuration, support storage of various data structures and improve the performance of the comparison strategy database, the comparison strategy database can adopt Redis for data storage. Specifically, the data format of key-field-value can be used for storage. Wherein, key represents the name of the database table, field-value represents the specific information of the field, for example, field represents the name of the field, value represents the contrast policy corresponding to the field, and so on. In addition, the field-value may also include multiple records to enable the storage of specific information for multiple fields.

The HBase can store comparison results after data comparison of the data comparison system every time, so that data comparison executed by the data comparison system every time is recorded and traceable, and subsequent checking by workers and maintenance of data in mysql and Redis can be facilitated.

After the data comparison system completes the data comparison, if the comparison result is that the data in the mysql is inconsistent with the data in the Redis, an alarm message needs to be sent to the alarm system, so that the alarm system can notify a corresponding system responsible person to process the data in time.

It should be noted that, in the architecture shown in fig. 3, a comparison page view viewing function may also be provided, so that a worker may view the process and the result of data comparison. The development or testing personnel may also send requests (e.g., HTTP (hypertext Transfer Protocol) requests) to the data comparison system to view the results of the data comparison, or to configure and modify the data comparison system.

In the system architecture shown in fig. 3, the database 103 is described by taking Redis as an example, and the database 103 may also be other types of databases, for example, databases such as HBase and ES.

The following describes the data comparison method in the embodiment of the present invention in detail with reference to the system architecture shown in fig. 3. An embodiment of the present invention provides a data comparison method, which may be executed by a data comparison system in a system architecture shown in fig. 3, as shown in fig. 4, where the method includes:

s401: and receiving and resolving binlog pushed by mysql.

When data in the mysql are changed, a binlog mechanism can be triggered in the mysql database to generate the binlog. mysql, upon triggering generation of the binlog, may push the generated binlog to the data comparison system, i.e., the data comparison system receives the binlog. After receiving the binlog, the data comparison system analyzes the binlog, and analyzes a field in which data updating occurs, namely a target field, and data after the field is updated, namely updated data from the binlog.

Due to the fact that the binlog mechanism in the mysql database is used, the binlog can be rapidly transmitted to the data comparison system, and therefore near real-time data transmission can be achieved.

S402: and loading the configuration of data comparison.

The comparison strategy database stores fields and data comparison strategies corresponding to the fields in advance, and the data comparison strategies corresponding to the fields can be loaded in the step so as to facilitate data comparison in the subsequent step.

It should be noted that some processing flows performed on the fields, for example, a flow for processing the out-of-order fields, may also be configured in the comparison policy database. These configurations may be used to pre-process the target field after it is derived.

S403: the contrast data in Redis is read.

After obtaining the target field and the updated data corresponding to the target field by analyzing the binlog, the data comparison system needs to read the comparison data for comparing the updated data corresponding to the target field from the Redis. Redis data storage is distributed storage that may store data in a variety of data structures, such as k-v data structures, set data structures, hash data structures, and so forth.

The Redis data stores data of a multi-data structure, so that the position of the data to be read in the Redis can be determined based on the data structure of the data to be read, and the data can be acquired based on the data acquisition mode of the data structure. For example, as shown in the sub-flow executed in this step in fig. 4, it is determined that the data structures of the data to be read are respectively a k-v data structure, a set data structure, and a hash data structure according to the target field, and it is necessary to read the data in the k-v data structure, the set data structure, and the hash data structure in the Redis in a data acquisition manner corresponding to the data structures, respectively, so that the data comparison system reads the comparison data corresponding to the target field according to the position of the target field at which each data structure is stored in the Redis.

It should be noted that, in the system architecture shown in fig. 3, the database 103 is described by taking Redis as an example, and when the database 103 is another type of database, the manner of reading the comparison data in this step may be performed according to specific requirements of the database.

S404: and (6) data comparison.

After reading the comparison data in Redis, the data comparison system can perform data comparison, that is, perform data comparison between the updated data parsed from the binlog and the comparison data read from Redis, and determine whether the updated data parsed from the binlog is consistent with the comparison data read from Redis.

S405: and judging whether the updated data analyzed from the binlog is consistent with the comparison data read from the Redis.

After the data comparison is completed in step S404, it can be determined whether the updated data analyzed in the binlog is consistent with the comparison data read in the Redis according to the comparison result. If the analyzed updated data in the binlog is consistent with the comparison data read in Redis, step S407 may be performed; step S406 may be performed if the parsed updated data in binlog and the compared data read in Redis are inconsistent.

S406: and if the updated data analyzed from the binlog is inconsistent with the contrast data read from the Redis, an abnormal alarm is given.

Through the data comparison in step S404, it can be obtained whether the updated data parsed out from the binlog is consistent with the comparison data read out from the Redis. If the updated data parsed from binlog is inconsistent with the comparison data read from Redis, the normal operation of the service system will be affected. Therefore, in this step, if the updated data analyzed from the binlog is inconsistent with the comparison data read from the Redis, in order to avoid affecting the normal operation of the service system, an abnormal warning is required, that is, an alarm message is sent to the warning system, and the warning message informs the staff to process the data in time.

S407: and storing the comparison result.

Whether the updated data analyzed from the binlog in the comparison result is consistent with the comparison data read from the Redis or not, the comparison result needs to be stored. After the step S405 determines whether the updated data parsed from the binlog is consistent with the comparison data read from the Redis, if the determination result is yes, the step may be directly executed, and the comparison result is saved; if the determination result is negative, step S406 may be executed first, and then the step is executed to store the comparison result. In the specific step, the result may be stored in the HBase shown in fig. 3, so that the staff can conveniently check the result.

It should be noted that, since it is necessary to store the comparison result if the updated data parsed from the binlog in the comparison result is consistent with the comparison data read from the Redis, in the embodiment of the present invention, after step S405 is executed, step S407 is executed regardless of the determination result, and when the updated data parsed from the binlog is inconsistent with the comparison data read from the Redis, step S406 is executed at the same time.

In the embodiment of the invention, the binlog pushed by the mysql can be received, and the binlog comprises the updated data and the target fields corresponding to the updated data, so that in the embodiment of the invention, which fields in the mysql are subjected to data updating and the updated data can be determined according to the binlog; when data comparison is carried out subsequently, the comparison data is only obtained from the Redis for data comparison, and updated data does not need to be obtained from the mysql, so that the access to the mysql is reduced, the occupation of the storage resources of the service system is reduced, and the performance of the service system for accessing the mysql, which is reduced due to the occupation of the storage resources of the service system, is avoided.

Based on the system architecture shown in fig. 3, the present invention provides an embodiment to explain the specific implementation procedure of step S404 in the data comparison method shown in fig. 4. The Binlog may include a plurality of fields in which data is updated, so that the present invention is described in that the target field includes a plurality of fields, as shown in fig. 5, the method includes:

s501: and converting the mapping alignment field.

Because the name rules are different or the database storage modes are different, the names of the fields in the database of the business system may be different from the names of the same fields in the data comparison system, so the data comparison system may also perform preprocessing, for example, conversion mapping processing, that is, the names of the target fields are converted and mapped into names that can be recognized by the data comparison system, so as to perform the subsequent data comparison process.

For example, for a field of a user identity, usually the naming convention of the database is that words are separated by underlines, so the field is defined in mysql as user _ id, but in java the naming convention is in the first letter of the hump named word, so the field is defined in java as userId, i.e. the field defining the user identity when the data alignment system uses java is named userId. The data alignment system as an independent system does not know that the user _ id in mysql is the same as the userId in the system, so the field with the field name of userId in mysql needs to be converted and mapped into the userId which can be identified by the data alignment system in the step.

It should be noted that the method and rule of the conversion mapping used in this step may be configured in advance, for example, may be completed in step S402 of the flow shown in fig. 4, and may be directly loaded from the comparison policy database when this step is executed.

S502: the fields are compared one by one.

Since the target field analyzed in the binlog includes a plurality of fields, and the data types corresponding to each field are not necessarily the same, it is necessary to compare each field one by one to determine the data type thereof, and then subsequently determine the data comparison policy.

S503: a comparison strategy is selected.

In step S502, the data structure of each field is determined, and thus the target comparison policy of each field can be determined. For example, the A field is a character string type, the selection comparison strategy is a character string comparison strategy, the B field is a k-v data structure, the selection comparison strategy is a k-v data structure comparison strategy, the C field is an ordered data structure type, the selection comparison strategy is an ordered data structure comparison strategy, the D field is a disorder data structure type, and the selection comparison strategy is a disorder data comparison strategy.

S504: and (6) data comparison.

After the target matching policy of each field is determined in step S503, data matching may be performed on each field at the same time. For example, the A, B, C and D fields described in step S503 all determine the corresponding target comparison policy, and then data comparison may be performed in this step.

It should be noted that, in the implementation of the present invention, there are various data comparison manners, and the data comparison general comparison reference may include the number, the sequence, the content, and the like, and meanwhile, in combination with various comparison strategies, the flexible configuration may be extended horizontally.

S505: and summarizing comparison results.

After the step S504 is completed, the comparison results corresponding to the fields may be summarized.

S506: and judging whether the comparison is finished.

After the comparison results are summarized in step S505, it can be determined whether all the fields have been subjected to data comparison according to whether the summarized results include the comparison results of all the fields, that is, whether the comparison is finished. If the summarized result does not include the comparison results of all the fields, there may be fields for which data comparison is not performed, so it is determined that the comparison is not finished, and step S502 needs to be executed; if the summarized result includes the comparison results of all the fields, it can be determined that the comparison is finished, that is, the entire process can be finished.

In order to solve the problems in the prior art, an embodiment of the present invention provides an apparatus 600 for data comparison, as shown in fig. 6, the apparatus 600 includes:

a receiving module 601, configured to receive a data update message pushed by a first database, where the data update message includes updated data and a target field corresponding to the updated data;

a determining module 602, configured to determine, according to a mapping relationship between a field and a comparison policy, a target comparison policy corresponding to the target field;

an obtaining module 603, configured to obtain, according to the target field, comparison data corresponding to the target field from at least one second database; wherein the second database has completed data synchronization based on the first database;

a comparison module 604, configured to compare the comparison data with the updated data through the target comparison policy to obtain a comparison result.

It should be understood that the manner of implementing the embodiment of the present invention is the same as the manner of implementing the embodiment shown in fig. 2, and the description thereof is omitted.

In an implementation manner of the embodiment of the present invention, the target field includes a plurality of fields, and the updated data includes updated data corresponding to each of the plurality of fields; the determining module 602 is further configured to: respectively determining a target comparison strategy corresponding to each field according to the mapping relation between the field and the comparison strategy; the comparison module 604 is further configured to: and for each field in the plurality of fields, comparing the comparison data corresponding to each field with the updated data corresponding to each field through the target comparison strategy corresponding to each field.

In an implementation manner of the embodiment of the present invention, the method further includes: the judging module is used for judging whether the comparison data and the updated data are consistent or not according to the comparison result of the target comparison strategy on the comparison data and the updated data;

the alarm module is used for sending an alarm message to the alarm system if the alarm module does not send the alarm message to the alarm system;

the storage module is used for storing the comparison result if the comparison result is not the same as the comparison result;

and the storage module is also used for storing the comparison result if the comparison result is positive.

In an implementation manner of the embodiment of the present invention, the method further includes: and the loading module is used for loading the target comparison strategy from a comparison strategy database, and the comparison strategy database is configured in advance.

In the embodiment of the present invention, after the data in the first database is updated, the data comparison apparatus 600 may receive a data update message, where the data update message includes the updated data and the target field corresponding to the updated data, so that in the embodiment of the present invention, the data comparison apparatus 600 may determine, according to the data update message, which data in the first database is updated and the updated data, so that when performing subsequent data comparison, only the comparison data needs to be obtained from the second database in which the data comparison is to be performed, and the updated data does not need to be obtained from the first database, thereby reducing access to the first database, reducing occupation of resources of the service system, and reducing influence on the service system.

The invention also provides an electronic device and a readable storage medium according to the embodiment of the invention.

The electronic device of the present invention includes: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform the method for data comparison provided by the embodiment of the invention.

Fig. 7 illustrates an exemplary system architecture 700 of a method or apparatus for data comparison to which embodiments of the present invention may be applied.

As shown in fig. 7, the system architecture 700 may include

terminal devices

701, 702, 703, a network 704, and a server 705. The network 704 serves to provide a medium for communication links between the

terminal devices

701, 702, 703 and the server 705. Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

A user may use the

terminal devices

701, 702, 703 to interact with a server 705 over a network 704, to receive or send messages or the like. The

terminal devices

701, 702, 703 may have installed thereon various communication client applications, such as a shopping-like application, a web browser application, a search-like application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only).

The

terminal devices

701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 705 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the

terminal devices

701, 702, 703. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, product information — just an example) to the terminal device.

It should be noted that the method for data comparison provided by the embodiment of the present invention is generally executed by the server 705, and accordingly, the apparatus for data comparison is generally disposed in the server 705.

It should be understood that the number of terminal devices, networks, and servers in fig. 7 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 8, a block diagram of a computer system 800 suitable for use in implementing embodiments of the present invention is shown. The computer system illustrated in FIG. 8 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the invention.

As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU)801 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.

The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program executes the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 801.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a receiving module, a determining module, an obtaining module, and a comparing module. The names of these modules do not in some cases constitute a limitation of the module itself, and for example, a receiving module may also be described as a "module that receives the function of the module".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to perform the method of data comparison provided by the present invention.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of data comparison, comprising:

receiving a data updating message pushed by a first database, wherein the data updating message comprises updated data and a target field corresponding to the updated data;

determining a target comparison strategy corresponding to the target field according to the mapping relation between the field and the comparison strategy;

acquiring comparison data corresponding to the target field from a second database according to the target field; wherein the second database has completed data synchronization based on the first database;

and comparing the comparison data with the updated data through the target comparison strategy to obtain a comparison result.

2. The method of claim 1, wherein the target field comprises a plurality of fields, and wherein the updated data comprises updated data corresponding to each of the plurality of fields;

determining a target comparison strategy corresponding to the target field according to the mapping relation between the field and the comparison strategy, wherein the target comparison strategy comprises the following steps:

respectively determining a target comparison strategy corresponding to each field according to the mapping relation between the field and the comparison strategy;

comparing the comparison data with the updated data by the target comparison policy, including:

and for each field in the plurality of fields, comparing the comparison data corresponding to each field with the updated data corresponding to each field through the target comparison strategy corresponding to each field.

3. The method of claim 1, further comprising, after obtaining the comparison result:

judging whether the comparison data and the updated data are consistent or not based on the comparison result;

if not, sending an alarm message to an alarm system, and storing the comparison result;

if yes, the comparison result is saved.

4. The method of claim 1, further comprising, prior to said comparing said comparison data and said updated data with said target comparison policy:

and loading the target comparison strategy from a comparison strategy database, wherein the comparison strategy database is configured in advance.

5. An apparatus for data comparison, comprising:

the receiving module is used for receiving a data updating message pushed by a first database, wherein the data updating message comprises updated data and a target field corresponding to the updated data;

the determining module is used for determining a target comparison strategy corresponding to the target field according to the mapping relation between the field and the comparison strategy;

the acquisition module is used for acquiring comparison data corresponding to the target field from a second database according to the target field; wherein the second database has completed data synchronization based on the first database;

and the comparison module is used for comparing the comparison data with the updated data through the target comparison strategy to obtain a comparison result.

6. The apparatus of claim 5, wherein the target field comprises a plurality of fields, and wherein the updated data comprises updated data corresponding to each of the plurality of fields; the determination module is further to:

the comparison module is further configured to:

7. The apparatus of claim 5, further comprising:

the judging module is used for judging whether the comparison data and the updated data are consistent or not based on the comparison result;

8. The apparatus of claim 5, further comprising:

and the loading module is used for loading the target comparison strategy from a comparison strategy database, and the comparison strategy database is configured in advance.

9. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-4.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-4.