CN113656432A - Data comparison method and device - Google Patents
Data comparison method and device Download PDFInfo
- Publication number
- CN113656432A CN113656432A CN202110935850.0A CN202110935850A CN113656432A CN 113656432 A CN113656432 A CN 113656432A CN 202110935850 A CN202110935850 A CN 202110935850A CN 113656432 A CN113656432 A CN 113656432A
- Authority
- CN
- China
- Prior art keywords
- query
- comparison
- result
- database
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000012544 monitoring process Methods 0.000 claims abstract description 21
- 238000004590 computer program Methods 0.000 claims description 14
- 238000004458 analytical method Methods 0.000 claims description 9
- 238000011161 development Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 239000000872 buffer Substances 0.000 description 4
- 230000006978 adaptation Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A data comparison method and device can be used in the financial field or other fields. The method comprises the following steps: obtaining base table comparison information from the first message queue by using a multithreading real-time monitoring mode, analyzing the base table comparison information, generating a first database query statement, querying the first database by using the first database query statement to obtain a first query result, and sending the base table comparison information to the second message queue; obtaining base table comparison information from the second message queue and analyzing the base table comparison information to generate a second database query statement, querying the second database by using the second database query statement, comparing the first query result to obtain a second query result, and sending the base table comparison information to a third message queue; and acquiring and analyzing the base table comparison information from the third message queue, and generating a data comparison result according to the first query result and the second query result. The invention realizes the pipeline type data comparison, improves the efficiency of the data comparison, configures the table comparison information and reduces the development cost.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a data comparison method and apparatus.
Background
In the existing technical scheme for data comparison of databases, data comparison of two databases is generally performed according to a certain specific scene. The general implementation method is to query all data of the two databases respectively, then compare the data table by table and item by item, and record the comparison result. The prior art has the problems of poor expandability, no universality, inflexible applicable scene and the like, and low contrast efficiency, and particularly relates to the conditions of multiple tables and large data volume.
Disclosure of Invention
In view of the problems in the prior art, embodiments of the present invention mainly aim to provide a data comparison method and apparatus, which implement pipelined data comparison and improve efficiency and universality of data comparison.
In order to achieve the above object, an embodiment of the present invention provides a data comparison method, where the method includes:
acquiring base table comparison information from a first message queue by using a multithreading real-time monitoring mode, analyzing the base table comparison information, generating a first database query statement, querying a first database by using the first database query statement to obtain a first query result, and sending the base table comparison information to a second message queue;
acquiring the base table comparison information from the second message queue by using a multithread real-time monitoring mode, analyzing the base table comparison information, generating a second database query statement, querying a second database by using the second database query statement, comparing the first query result to obtain a second query result, and sending the base table comparison information to a third message queue;
and acquiring and analyzing the base table comparison information from the third message queue by utilizing a multithreading real-time monitoring mode, and generating a data comparison result according to the first query result and the second query result.
Optionally, in an embodiment of the present invention, the method further includes: and acquiring preset base table comparison information, and sending the base table comparison information to the first message queue.
Optionally, in an embodiment of the present invention, the base table comparison information includes a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a table-associative query increment comparison table name, and a table-associative query increment comparison table field.
Optionally, in an embodiment of the present invention, the analyzing the library table comparison information and generating the first database query statement includes:
analyzing the base table comparison information to obtain a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a linked table query increment comparison table name and a linked table query increment comparison linked table field;
and generating the first database query statement according to the data format of the first database by using the table name, the query field, the comparison type, the sorting field, the increment comparison control field type, the linked list query increment comparison table name and the linked list query increment comparison linked list field.
Optionally, in an embodiment of the present invention, the querying a second database by using the second database query statement, and comparing the first query result to obtain a second query result includes:
querying a second database by using the second database query statement to obtain a query record HASH value;
and comparing the HASH value of the query record with the HASH value in the first query result to generate a comparison result, and generating a second query result according to the comparison result.
Optionally, in an embodiment of the present invention, the obtaining and analyzing the library table comparison information from the third message queue, and generating a data comparison result according to the first query result and the second query result includes:
obtaining and analyzing the base table comparison information from the third message queue, and obtaining the first query result and the second query result from a distributed cache according to the obtained analysis result;
and generating the data comparison result according to the first query result and the second query result.
The embodiment of the invention also provides a data comparison device, which comprises:
the first query result module is used for acquiring base table comparison information from a first message queue by utilizing a multithreading real-time monitoring mode, analyzing the base table comparison information, generating a first database query statement, querying a first database by utilizing the first database query statement to obtain a first query result, and sending the base table comparison information to a second message queue;
the second query result module is used for acquiring the base table comparison information from the second message queue by utilizing a multithreading real-time monitoring mode, analyzing the base table comparison information, generating a second database query statement, querying a second database by utilizing the second database query statement, comparing the first query result to obtain a second query result, and sending the base table comparison information to a third message queue;
and the data comparison result module is used for acquiring and analyzing the base table comparison information from the third message queue by utilizing a multithreading real-time monitoring mode, and generating a data comparison result according to the first query result and the second query result.
Optionally, in an embodiment of the present invention, the apparatus further includes: and the information acquisition module is used for acquiring preset base table comparison information and sending the base table comparison information to the first message queue.
Optionally, in an embodiment of the present invention, the base table comparison information includes a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a table-associative query increment comparison table name, and a table-associative query increment comparison table field.
Optionally, in an embodiment of the present invention, the first query result module includes:
the information analysis unit is used for analyzing the base table comparison information to obtain a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a linked table query increment comparison table name and a linked table query increment comparison linked table field;
and the query statement unit is used for generating the first database query statement according to the data format of the first database by utilizing the table name, the query field, the comparison type, the sorting field, the increment comparison control field type, the linked list query increment comparison table name and the linked list query increment comparison linked list field.
Optionally, in an embodiment of the present invention, the second query result module includes:
the HASH value unit is used for querying the second database by using the second database query statement to obtain a query record HASH value;
and the query result unit is used for comparing the query record HASH value with the HASH value in the first query result to generate a comparison result, and generating a second query result according to the comparison result.
Optionally, in an embodiment of the present invention, the data comparison result module includes:
a result obtaining unit, configured to obtain and analyze the library table comparison information from the third message queue, and obtain the first query result and the second query result from a distributed cache according to an obtained analysis result;
and the comparison result unit is used for generating the data comparison result according to the first query result and the second query result.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method when executing the program.
The present invention also provides a computer-readable storage medium storing a computer program for executing the above method.
The invention overcomes the defects of the prior art through the message queue, realizes the pipeline type data comparison, improves the efficiency of the data comparison, configures the table comparison information, realizes the general adaptation of the scheme and reduces the development cost.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a flow chart of a data comparison method according to an embodiment of the present invention;
FIG. 2 is a flow chart of generating a first database query statement in an embodiment of the present invention;
FIG. 3 is a flowchart of obtaining a second query result in an embodiment of the present invention;
FIG. 4 is a flow chart of generating a data comparison result in an embodiment of the present invention;
FIG. 5 is a flow chart of data comparison in an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a data comparison apparatus according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a data comparison apparatus according to an embodiment of the present invention;
FIG. 8 is a block diagram of a first query result module according to an embodiment of the present invention;
FIG. 9 is a diagram illustrating a second query result module according to an embodiment of the present invention;
FIG. 10 is a block diagram of a data comparison module according to an embodiment of the present invention;
fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a data comparison method and device, which can be used in the financial field and other fields, and it should be noted that the data comparison method and device can be used in the financial field and any fields except the financial field, and the application fields of the data comparison method and device are not limited.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a flowchart illustrating a data comparison method according to an embodiment of the present invention, where an execution subject of the data comparison method includes, but is not limited to, a computer. The method shown in the figure comprises the following steps:
step S1, obtaining the base table comparison information from the first message queue by using a multithread real-time monitoring mode, analyzing the base table comparison information, generating a first database query statement, querying the first database by using the first database query statement to obtain a first query result, and sending the base table comparison information to the second message queue.
Wherein, the message queue is a container for storing messages in the transmission process of the messages. The message queue of the invention is used for storing the base table comparison information, and in the whole data comparison process, the base table comparison information is consumed and forwarded from one message queue to another message queue until the comparison result is obtained.
Further, the table comparison information is preset and includes table name, query field, comparison type, sorting field, increment comparison control field type, table-linked query increment comparison table name, table-linked query increment comparison table field and other information. The table comparison information represents which tables are compared in the database, which fields are compared by each table, the comparison mode (increment or full amount) and the like, and can be pre-configured in a database recording mode, and the configuration is stored in the first message queue.
Furthermore, the first message queue is monitored in real time by multiple threads, and when messages exist in the first message queue, the messages are consumed by the multiple threads. The base table comparison information of the first message queue is obtained and analyzed, and the SQL query statement, namely the first query statement, is assembled according to the format of the first database, and generally in a paging cycle query mode.
Further, the obtained first query result is stored in a first distributed cache, and the data format is HASH. Thus, all the base tables of the comparison store one HASH type data result set, i.e., the first query result. And storing the database table comparison information into a second message queue after the query of one database table is completed.
Step S2, obtaining the library table comparison information from the second message queue by using a multi-thread real-time monitoring manner, analyzing the library table comparison information, generating a second database query statement, querying the second database by using the second database query statement, comparing the first query result to obtain a second query result, and sending the library table comparison information to a third message queue.
And when the second message queue has messages, the second message queue is consumed by multiple threads. The base table comparison information of the second message queue is obtained and analyzed, and an SQL query statement, i.e., a second query statement, is assembled according to the format of the second database, generally in a paging cycle query manner (the same as the first query statement).
Further, the HASH value of the query record is computed, and data of the same HASH of the same table of the first database is removed from the first distributed cache, indicating that the data of the table is not different. And if the same database table of the first database in the first distributed cache does not have the same HASH record, storing the data in the second distributed cache according to the same rule as the first query result. And after each comparison completes one database table, storing the comparison information of the database table into a third message queue.
And step S3, acquiring and analyzing the base table comparison information from the third message queue by using a multi-thread real-time monitoring mode, and generating a data comparison result according to the first query result and the second query result.
And the third message queue is monitored in real time by multiple threads, and when the third message queue has messages, the messages are consumed by the multiple threads. And acquiring and analyzing the base table comparison information in the third message queue, and then respectively acquiring data in the first distributed cache and the second distributed cache, namely a first query result and a second query result. And storing the data into a database table comparison result table in a database, thereby obtaining a data comparison result.
As an embodiment of the invention, the method further comprises: and acquiring preset base table comparison information, and sending the base table comparison information to the first message queue.
The table comparison information represents which tables are compared in the database, which fields are compared in each table, the comparison mode (increment or full amount) and the like, and the table comparison information can be pre-configured in a database recording mode and stores the configuration in the first message queue.
As an embodiment of the invention, the base table comparison information comprises a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a linked table query increment comparison table name and a linked table query increment comparison linked table field.
The specific table comparison information is shown in table 1, and the table comparison information represents which tables are compared in the database, which fields are compared by each table, the comparison mode (increment or full amount) and the like, and can be configured in a database recording mode, so that configuration modification can be performed according to actual conditions, and data comparison is more flexible and controllable.
TABLE 1
In this embodiment, as shown in fig. 2, analyzing the library table comparison information, and generating the first database query statement includes:
step S21, analyzing the table comparison information to obtain table name, query field, comparison type, sort field, increment comparison control field type, linked table query increment comparison table name and linked table query increment comparison linked table field;
and step S22, generating the first database query statement according to the data format of the first database by using the table name, the query field, the comparison type, the sorting field, the increment comparison control field type, the linked list query increment comparison table name and the linked list query increment comparison linked list field.
The method comprises the steps of obtaining base table comparison information of a first message queue, analyzing the base table comparison information to obtain a table name, a query field, a comparison type, a sequencing field, an increment comparison control field type, a linked table query increment comparison table name and a linked table query increment comparison linked table field. Then, the SQL query statements, i.e. the first query statement, are assembled according to the format of the first database, generally in a paging cycle query manner.
As an embodiment of the present invention, as shown in fig. 3, querying a second database by using the second database query statement, and comparing the first query result to obtain a second query result includes:
step S31, using the second database query statement to query the second database to obtain a query record HASH value;
step S32, comparing the HASH value of the query record with the HASH value in the first query result to generate a comparison result, and generating a second query result according to the comparison result.
And the HASH value of the query record is calculated, the data of the same HASH of the same base table of the first database is removed from the first distributed cache, and the data of the table is shown to have no difference. And if the same database table of the first database in the first distributed cache does not have the same HASH record, storing the data in the second distributed cache according to the same rule as the first query result.
As an embodiment of the present invention, as shown in fig. 4, acquiring and analyzing the library table comparison information from the third message queue, and generating a data comparison result according to the first query result and the second query result includes:
step S41, obtaining and analyzing the library table comparison information from the third message queue, and obtaining the first query result and the second query result from a distributed cache according to the obtained analysis result;
step S42, generating the data comparison result according to the first query result and the second query result.
And acquiring the base table comparison information in the third message queue, analyzing the base table comparison information, and then respectively acquiring the data in the first distributed cache and the second distributed cache, namely the first query result and the second query result. And storing the data into a database table comparison result table in a database, thereby obtaining a data comparison result.
In one embodiment of the present invention, as shown in FIG. 5, the present invention utilizes message queues to pipeline data comparison.
The assembly line, also called assembly line, is a production mode in industry, and refers to that each production unit only focuses on the work of processing a certain segment, so as to improve the work efficiency and the yield. The pipeline of the invention is that each program unit only focuses on processing a certain data comparison function point, and finally obtains a data comparison result through the processing of a plurality of pipeline programs. In addition, the data comparison of the invention refers to two different databases, and the data difference of the two databases is compared under the condition that the database table structures are completely the same.
Further, pipeline 1: caching source data of database A (first database)
The multithread monitors message queue a (first message queue) in real time and consumes messages in message queue a when they are available. The base table comparison information of the message queue A is obtained and analyzed, and SQL query statements (first query statements) are assembled according to the format of the database A, and a paging cycle query mode is generally adopted. If the MySQL database is assembled as follows:
and (3) full quantity comparison:
SELECT${select_columns}FROM${table_name}limit${startNo},${querySize}
and (4) incremental comparison:
SELECT${select_columns}FROM${table_name}where${raise_columns}>${raise_colums_value}limit${startNo},${querySize}
and (4) performing linked table query increment comparison:
select${select_columns}from${table_name}a left join${join_table}b on${join_columns}where b.${raise_columns}>${raise_colums_value}limit${startNo},${querySize}
and storing the query result (first query result) into a distributed cache A (first distributed cache), wherein the stored data format is HASH, KEY is the database type + table name, and VALUE is MAP < KEY, VALUE >. KEY in the MAP is the HASH VALUE of the record, and VALUE is the record identification field VALUE. Thus, all the base tables of the comparison store one HASH type data result set.
And when one database table is queried, storing the database table comparison information into another message queue B (a second message queue).
Further, pipeline 2: database B (second database) target data comparison
And the multithreading real-time monitoring message queue B is used for consuming the messages by multithreading when the messages exist in the message queue B. And acquiring and analyzing the base table comparison information of the message queue B, and assembling an SQL query statement (a second query statement) according to the format of the database B, wherein a paging cycle query mode (the same as the pipeline 1) is generally adopted.
The HASH value of the query record is computed, and the data of the same HASH of the same table of database a is removed from the distributed cache a, indicating that the data of the table is not different. And if the same database table of the database A in the distributed cache A does not have the same HASH record, storing the data into a distributed cache B (a second distributed cache) according to the rule of the pipeline 1 as a second query result.
And after each match completes one database table, storing the database table comparison information into another message queue C (a third message queue).
Further, a third assembly line: database AB comparison result storage
And the multithreading real-time monitoring message queue C consumes the messages in the message queue C in a multithreading mode when the messages exist in the message queue C. And acquiring and analyzing the base table comparison information in the message queue C, then respectively acquiring the data in the distributed cache A, B, and storing the data in the base table comparison result table in the database, thereby acquiring a data comparison result.
The invention overcomes the defects of the prior art through the message queue, realizes the pipeline type data comparison, improves the efficiency of the data comparison, configures the table comparison information, realizes the general adaptation of the scheme and reduces the development cost.
Fig. 6 is a schematic structural diagram of a data comparison apparatus according to an embodiment of the present invention, where the apparatus includes:
the first query result module 10 is configured to acquire the base table comparison information from the first message queue in a multi-thread real-time monitoring manner, analyze the base table comparison information, generate a first database query statement, query the first database by using the first database query statement, obtain a first query result, and send the base table comparison information to the second message queue.
Wherein, the message queue is a container for storing messages in the transmission process of the messages. The message queue of the invention is used for storing the base table comparison information, and in the whole data comparison process, the base table comparison information is consumed and forwarded from one message queue to another message queue until the comparison result is obtained.
Further, the table comparison information is preset and includes table name, query field, comparison type, sorting field, increment comparison control field type, table-linked query increment comparison table name, table-linked query increment comparison table field and other information. The table comparison information represents which tables are compared in the database, which fields are compared by each table, the comparison mode (increment or full amount) and the like, and can be pre-configured in a database recording mode, and the configuration is stored in the first message queue.
Furthermore, the first message queue is monitored in real time by multiple threads, and when messages exist in the first message queue, the messages are consumed by the multiple threads. The base table comparison information of the first message queue is obtained and analyzed, and the SQL query statement, namely the first query statement, is assembled according to the format of the first database, and generally in a paging cycle query mode.
Further, the obtained first query result is stored in a first distributed cache, and the data format is HASH. Thus, all the base tables of the comparison store one HASH type data result set, i.e., the first query result. And storing the database table comparison information into a second message queue after the query of one database table is completed.
And a second query result module 20, configured to acquire the library table comparison information from the second message queue in a multi-thread real-time monitoring manner, analyze the library table comparison information, generate a second database query statement, query the second database by using the second database query statement, compare the first query result, obtain a second query result, and send the library table comparison information to a third message queue.
And when the second message queue has messages, the second message queue is consumed by multiple threads. The base table comparison information of the second message queue is obtained and analyzed, and an SQL query statement, i.e., a second query statement, is assembled according to the format of the second database, generally in a paging cycle query manner (the same as the first query statement).
Further, the HASH value of the query record is computed, and data of the same HASH of the same table of the first database is removed from the first distributed cache, indicating that the data of the table is not different. And if the same database table of the first database in the first distributed cache does not have the same HASH record, storing the data in the second distributed cache according to the same rule as the first query result. And after each comparison completes one database table, storing the comparison information of the database table into a third message queue.
And the data comparison result module 30 is configured to acquire and analyze the base table comparison information from the third message queue in a multi-thread real-time monitoring manner, and generate a data comparison result according to the first query result and the second query result.
And the third message queue is monitored in real time by multiple threads, and when the third message queue has messages, the messages are consumed by the multiple threads. And acquiring and analyzing the base table comparison information in the third message queue, and then respectively acquiring data in the first distributed cache and the second distributed cache, namely a first query result and a second query result. And storing the data into a database table comparison result table in a database, thereby obtaining a data comparison result.
As an embodiment of the present invention, as shown in fig. 7, the apparatus further includes: and the information acquisition module 40 is configured to acquire preset library table comparison information and send the library table comparison information to the first message queue.
As an embodiment of the invention, the base table comparison information comprises a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a linked table query increment comparison table name and a linked table query increment comparison linked table field.
In this embodiment, as shown in fig. 8, the first query result module 10 includes:
the information analysis unit 11 is configured to analyze the library table comparison information to obtain a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a linked table query increment comparison table name, and a linked table query increment comparison linked table field;
and the query statement unit 12 is configured to generate the first database query statement according to the data format of the first database by using the table name, the query field, the comparison type, the sorting field, the incremental comparison control field type, the linked list query incremental comparison table name, and the linked list query incremental comparison linked list field.
As an embodiment of the present invention, as shown in fig. 9, the second query result module 20 includes:
a HASH value unit 21, configured to query the second database by using the second database query statement, to obtain a query record HASH value;
and the query result unit 22 is configured to compare the HASH value of the query record with the HASH value in the first query result, generate a comparison result, and generate a second query result according to the comparison result.
As an embodiment of the present invention, as shown in fig. 10, the data comparison result module 30 includes:
a result obtaining unit 31, configured to obtain and analyze the library table comparison information from the third message queue, and obtain the first query result and the second query result from a distributed cache according to an obtained analysis result;
and a comparison result unit 32, configured to generate the data comparison result according to the first query result and the second query result.
Based on the same application concept as the data comparison method, the invention also provides the data comparison device. Because the principle of solving the problems of the data comparison device is similar to that of the data comparison method, the implementation of the data comparison device can refer to the implementation of the data comparison method, and repeated parts are not described again.
The invention overcomes the defects of the prior art through the message queue, realizes the pipeline type data comparison, improves the efficiency of the data comparison, configures the table comparison information, realizes the general adaptation of the scheme and reduces the development cost.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method when executing the program.
The present invention also provides a computer-readable storage medium storing a computer program for executing the above method.
As shown in fig. 11, the electronic device 600 may further include: communication module 110, input unit 120, audio processing unit 130, display 160, power supply 170. It is noted that the electronic device 600 does not necessarily include all of the components shown in FIG. 11; furthermore, the electronic device 600 may also comprise components not shown in fig. 11, which may be referred to in the prior art.
As shown in fig. 11, the central processor 100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, the central processor 100 receiving input and controlling the operation of the various components of the electronic device 600.
The memory 140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 100 may execute the program stored in the memory 140 to realize information storage or processing, etc.
The input unit 120 provides input to the cpu 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display an object to be displayed, such as an image or a character. The display may be, for example, an LCD display, but is not limited thereto.
The memory 140 may be a solid state memory such as Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 140 may also be some other type of device. Memory 140 includes buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application/function storage section 142, and the application/function storage section 142 is used to store application programs and function programs or a flow for executing the operation of the electronic device 600 by the central processing unit 100.
The memory 140 may also include a data store 143, the data store 143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by the electronic device. The driver storage portion 144 of the memory 140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging application, address book application, etc.).
The communication module 110 is a transmitter/receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter/receiver) 110 is coupled to the central processor 100 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132 to implement general telecommunications functions. Audio processor 130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, an audio processor 130 is also coupled to the central processor 100, so that recording on the local can be enabled through a microphone 132, and so that sound stored on the local can be played through a speaker 131.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (14)
1. A method of data comparison, the method comprising:
acquiring base table comparison information from a first message queue by using a multithreading real-time monitoring mode, analyzing the base table comparison information, generating a first database query statement, querying a first database by using the first database query statement to obtain a first query result, and sending the base table comparison information to a second message queue;
acquiring the base table comparison information from the second message queue by using a multithread real-time monitoring mode, analyzing the base table comparison information, generating a second database query statement, querying a second database by using the second database query statement, comparing the first query result to obtain a second query result, and sending the base table comparison information to a third message queue;
and acquiring and analyzing the base table comparison information from the third message queue by utilizing a multithreading real-time monitoring mode, and generating a data comparison result according to the first query result and the second query result.
2. The method of claim 1, further comprising: and acquiring preset base table comparison information, and sending the base table comparison information to the first message queue.
3. The method of claim 1, wherein the library table comparison information comprises a table name, a query field, a comparison type, a sort field, an incremental comparison control field type, a linked-table query incremental comparison table name, and a linked-table query incremental comparison linked-table field.
4. The method of claim 3, wherein parsing the library table comparison information to generate a first database query statement comprises:
analyzing the base table comparison information to obtain a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a linked table query increment comparison table name and a linked table query increment comparison linked table field;
and generating the first database query statement according to the data format of the first database by using the table name, the query field, the comparison type, the sorting field, the increment comparison control field type, the linked list query increment comparison table name and the linked list query increment comparison linked list field.
5. The method of claim 1, wherein the querying a second database using the second database query statement and comparing the first query result to obtain a second query result comprises:
querying a second database by using the second database query statement to obtain a query record HASH value;
and comparing the HASH value of the query record with the HASH value in the first query result to generate a comparison result, and generating a second query result according to the comparison result.
6. The method of claim 1, wherein the obtaining and parsing the library table comparison information from the third message queue, and generating a data comparison result according to the first query result and the second query result comprises:
obtaining and analyzing the base table comparison information from the third message queue, and obtaining the first query result and the second query result from a distributed cache according to the obtained analysis result;
and generating the data comparison result according to the first query result and the second query result.
7. A data comparison apparatus, said apparatus comprising:
the first query result module is used for acquiring base table comparison information from a first message queue by utilizing a multithreading real-time monitoring mode, analyzing the base table comparison information, generating a first database query statement, querying a first database by utilizing the first database query statement to obtain a first query result, and sending the base table comparison information to a second message queue;
the second query result module is used for acquiring the base table comparison information from the second message queue by utilizing a multithreading real-time monitoring mode, analyzing the base table comparison information, generating a second database query statement, querying a second database by utilizing the second database query statement, comparing the first query result to obtain a second query result, and sending the base table comparison information to a third message queue;
and the data comparison result module is used for acquiring and analyzing the base table comparison information from the third message queue by utilizing a multithreading real-time monitoring mode, and generating a data comparison result according to the first query result and the second query result.
8. The apparatus of claim 7, further comprising: and the information acquisition module is used for acquiring preset base table comparison information and sending the base table comparison information to the first message queue.
9. The apparatus of claim 7, wherein the library table comparison information comprises a table name, a query field, a comparison type, a sort field, an incremental comparison control field type, a linked-table query incremental comparison table name, and a linked-table query incremental comparison linked-table field.
10. The apparatus of claim 9, wherein the first query result module comprises:
the information analysis unit is used for analyzing the base table comparison information to obtain a table name, a query field, a comparison type, a sorting field, an increment comparison control field type, a linked table query increment comparison table name and a linked table query increment comparison linked table field;
and the query statement unit is used for generating the first database query statement according to the data format of the first database by utilizing the table name, the query field, the comparison type, the sorting field, the increment comparison control field type, the linked list query increment comparison table name and the linked list query increment comparison linked list field.
11. The apparatus of claim 7, wherein the second query result module comprises:
the HASH value unit is used for querying the second database by using the second database query statement to obtain a query record HASH value;
and the query result unit is used for comparing the query record HASH value with the HASH value in the first query result to generate a comparison result, and generating a second query result according to the comparison result.
12. The apparatus of claim 7, wherein the data comparison result module comprises:
a result obtaining unit, configured to obtain and analyze the library table comparison information from the third message queue, and obtain the first query result and the second query result from a distributed cache according to an obtained analysis result;
and the comparison result unit is used for generating the data comparison result according to the first query result and the second query result.
13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 6 when executing the computer program.
14. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110935850.0A CN113656432B (en) | 2021-08-16 | 2021-08-16 | Data comparison method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110935850.0A CN113656432B (en) | 2021-08-16 | 2021-08-16 | Data comparison method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113656432A true CN113656432A (en) | 2021-11-16 |
| CN113656432B CN113656432B (en) | 2024-10-18 |
Family
ID=78479183
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110935850.0A Active CN113656432B (en) | 2021-08-16 | 2021-08-16 | Data comparison method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113656432B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114969098A (en) * | 2022-06-21 | 2022-08-30 | 中银金融科技有限公司 | Data processing method and device, electronic equipment and computer storage medium |
| CN115314751A (en) * | 2022-08-08 | 2022-11-08 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN117131058A (en) * | 2023-10-23 | 2023-11-28 | 天津神舟通用数据技术有限公司 | Parallel comparison database table data method based on data slicing |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107291827A (en) * | 2017-05-27 | 2017-10-24 | 北京思特奇信息技术股份有限公司 | A kind of data comparison method and system |
| CN110704475A (en) * | 2019-09-29 | 2020-01-17 | 中国银行股份有限公司 | Method and system for comparing ETL loading table structures |
| US10635718B1 (en) * | 2016-10-13 | 2020-04-28 | Jpmorgan Chase Bank, N.A. | Method and system for implementing a data compare tool |
| CN111143406A (en) * | 2018-11-06 | 2020-05-12 | 北京京东尚科信息技术有限公司 | Database data comparison method and database data comparison system |
| CN111177119A (en) * | 2019-12-23 | 2020-05-19 | 平安资产管理有限责任公司 | Database-based full-data comparison method, device, equipment and storage medium |
| WO2020139282A1 (en) * | 2018-12-24 | 2020-07-02 | Turkcell Teknoloji̇ Araştirma Ve Geli̇şti̇rme Anoni̇m Şi̇rketi̇ | A data comparison system |
| CN111597164A (en) * | 2020-04-02 | 2020-08-28 | 深圳壹账通智能科技有限公司 | Database structure testing method, database structure testing device and terminal equipment |
| US20210064637A1 (en) * | 2019-09-04 | 2021-03-04 | American Express Travel Related Services Company, Inc. | Systems and methods for comparing data across data sources and platforms |
| CN112749541A (en) * | 2021-01-14 | 2021-05-04 | 京东数字科技控股股份有限公司 | Data verification system, method, device, electronic equipment and computer readable medium |
-
2021
- 2021-08-16 CN CN202110935850.0A patent/CN113656432B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10635718B1 (en) * | 2016-10-13 | 2020-04-28 | Jpmorgan Chase Bank, N.A. | Method and system for implementing a data compare tool |
| CN107291827A (en) * | 2017-05-27 | 2017-10-24 | 北京思特奇信息技术股份有限公司 | A kind of data comparison method and system |
| CN111143406A (en) * | 2018-11-06 | 2020-05-12 | 北京京东尚科信息技术有限公司 | Database data comparison method and database data comparison system |
| WO2020139282A1 (en) * | 2018-12-24 | 2020-07-02 | Turkcell Teknoloji̇ Araştirma Ve Geli̇şti̇rme Anoni̇m Şi̇rketi̇ | A data comparison system |
| US20210064637A1 (en) * | 2019-09-04 | 2021-03-04 | American Express Travel Related Services Company, Inc. | Systems and methods for comparing data across data sources and platforms |
| CN110704475A (en) * | 2019-09-29 | 2020-01-17 | 中国银行股份有限公司 | Method and system for comparing ETL loading table structures |
| CN111177119A (en) * | 2019-12-23 | 2020-05-19 | 平安资产管理有限责任公司 | Database-based full-data comparison method, device, equipment and storage medium |
| CN111597164A (en) * | 2020-04-02 | 2020-08-28 | 深圳壹账通智能科技有限公司 | Database structure testing method, database structure testing device and terminal equipment |
| CN112749541A (en) * | 2021-01-14 | 2021-05-04 | 京东数字科技控股股份有限公司 | Data verification system, method, device, electronic equipment and computer readable medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114969098A (en) * | 2022-06-21 | 2022-08-30 | 中银金融科技有限公司 | Data processing method and device, electronic equipment and computer storage medium |
| CN115314751A (en) * | 2022-08-08 | 2022-11-08 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN115314751B (en) * | 2022-08-08 | 2024-07-23 | 北京达佳互联信息技术有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN117131058A (en) * | 2023-10-23 | 2023-11-28 | 天津神舟通用数据技术有限公司 | Parallel comparison database table data method based on data slicing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113656432B (en) | 2024-10-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113656432B (en) | Data comparison method and device | |
| CN111782470B (en) | Distributed container log data processing method and device | |
| US20210365421A1 (en) | Data analysis method, computer device and storage medium | |
| CN108572789B (en) | Disk storage method and device, message pushing method and device and electronic equipment | |
| CN112783793B (en) | Automatic interface test system and method | |
| CN105589783A (en) | Application program lag problem data obtaining method and device | |
| CN107423404B (en) | Flow instance data synchronous processing method and device | |
| CN112732650A (en) | File fragmentation method and device | |
| CN106648839A (en) | Method and device for processing data | |
| CN113095782A (en) | Automatic approval decision-making method and device | |
| CN112905461B (en) | Method and device for executing automatic interface test case | |
| CN112786032A (en) | Display content control method, device, computer device and readable storage medium | |
| CN112084210B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN113468196A (en) | Method, apparatus, system, server and medium for processing data | |
| CN112800063A (en) | Automatic label passing method and device based on data structure | |
| CN112052391A (en) | Data pushing method, device and system | |
| CN113515447B (en) | Automatic testing method and device for system | |
| CN116244202A (en) | Automatic performance test method and device | |
| CN112988603B (en) | Big data test case generation method and device | |
| CN110806874A (en) | Analysis method, storage medium, electronic device and system for configuration file of live broadcast room | |
| CN113010514B (en) | Thermal loading method and device | |
| CN112783633A (en) | Data updating system and method based on resource mutual exclusion scheduling model | |
| CN115495519A (en) | Report data processing method and device | |
| CN112486979B (en) | Data processing method, device and system, electronic equipment and computer readable storage medium | |
| CN114490402A (en) | Method and device for checking validity of message field |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |