CN110059071B - Data comparison monitoring method, system, equipment and medium based on storage process - Google Patents
Data comparison monitoring method, system, equipment and medium based on storage process Download PDFInfo
- Publication number
- CN110059071B CN110059071B CN201910190040.XA CN201910190040A CN110059071B CN 110059071 B CN110059071 B CN 110059071B CN 201910190040 A CN201910190040 A CN 201910190040A CN 110059071 B CN110059071 B CN 110059071B
- Authority
- CN
- China
- Prior art keywords
- compared
- data table
- data
- comparison
- basic data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 139
- 238000012544 monitoring process Methods 0.000 title claims abstract description 107
- 230000008569 process Effects 0.000 title claims abstract description 78
- 230000001960 triggered effect Effects 0.000 claims abstract description 18
- 230000002159 abnormal effect Effects 0.000 claims description 11
- 230000005012 migration Effects 0.000 claims description 10
- 238000013508 migration Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 8
- 238000002360 preparation method Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 description 12
- 230000007547 defect Effects 0.000 description 5
- JASONGFGOLHLGB-UHFFFAOYSA-N Atranol Chemical compound CC1=CC(O)=C(C=O)C(O)=C1 JASONGFGOLHLGB-UHFFFAOYSA-N 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000005856 abnormality Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000010295 mobile communication Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Abstract
The disclosure relates to a data comparison monitoring method based on a stored procedure, which is characterized by comprising the following steps: monitoring and knowing that a comparison switch in the main storage process is triggered, and determining a target basic data table on which the comparison depends from at least one basic data table; storing the table name of the target basic data table into a monitoring record table; triggering a corresponding secondary storage process, acquiring a first data table to be compared associated with a target basic data table and a second data table to be compared associated with the target basic data table, and comparing the first data table to be compared with the second data table to be compared; determining comparison results of the first data table to be compared and the second data table to be compared; and recording the comparison result in a position associated with the table name in the monitoring record table. The present disclosure also provides a data comparison monitoring system, a computer device and a computer readable storage medium based on the stored procedure.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data comparison monitoring method, system, computer device, and computer readable storage medium based on a storage process.
Background
In the use of databases, data migration is often required, possibly due to insufficient server resources, network isolation, data disaster recovery, etc. Data migration essentially refers to the migration of a database in system a to system B. However, since the database in system a is updated in real time, it is also generally necessary to compare the database in system a with the database in system B after migration is completed.
However, in the process of implementing the inventive concept, the inventor has found that at least the following drawbacks exist in the related art: the existing database contrast has no monitoring mechanism, which is unfavorable for observing the state of data contrast.
Disclosure of Invention
The disclosure aims to provide a data comparison monitoring method, a system, a computer device and a computer readable storage medium based on a storage process, so as to at least solve the defects that a monitoring mechanism does not exist in database comparison in the prior art and the data comparison state is not easy to observe.
One aspect of the present disclosure provides a data comparison monitoring method based on a stored procedure, which is characterized by comprising: monitoring and knowing that a comparison switch in the main storage process is triggered, and determining a target basic data table on which the comparison depends from at least one basic data table; storing the table name of the target basic data table into a monitoring record table; triggering a corresponding secondary storage process, acquiring a first data table to be compared associated with the target basic data table and a second data table to be compared associated with the target basic data table, and comparing the first data table to be compared with the second data table to be compared, wherein the target basic data table is used for storing basic data associated with data to be compared in the first data table to be compared and basic data associated with data to be compared in the second data table to be compared; determining comparison results of the first data table to be compared and the second data table to be compared; and recording the comparison result in the monitoring record table at a position associated with the table name.
According to an embodiment of the present disclosure, the determining the comparison result of the first to-be-compared data table and the second to-be-compared data table includes: determining the number of the data to be compared between the first data table to be compared and the second data table to be compared; determining the number of matching failure pieces of data which are failed to match from the number of the comparison pieces; and determining the successful matching number of the successfully matched data from the comparison number.
According to an embodiment of the present disclosure, after the storing the table name of the target base data table in the monitoring record table, the method further includes: the state of the monitoring record table is set to a first state for characterizing that a comparison operation is being performed depending on the target underlying data table.
According to an embodiment of the present disclosure, the above method further includes: and if the first data table to be compared is different from the second data table to be compared, setting the state of the second data table to be compared to be a second state used for representing that data updating is needed.
According to an embodiment of the present disclosure, the above method further includes: judging whether the comparison process of the first data table to be compared and the second data table to be compared is abnormal or not; if yes, setting the state of the monitoring record table to be a third state used for representing the comparison failure of the first data table to be compared and the second data table to be compared; or if not, setting the state of the monitoring record table to be a fourth state for representing that the first data table to be compared and the second data table to be compared are successfully compared.
According to an embodiment of the present disclosure, the at least one base data table includes a plurality of base data tables, and the method further includes: judging whether the comparison operation depending on the plurality of basic data tables is completed or not; and if not, determining a basic data table on which the next comparison operation depends from the plurality of basic data tables.
Another aspect of the present disclosure provides a data comparison monitoring system based on a stored procedure, comprising: the first determining module is used for monitoring and knowing that a comparison switch in the main storage process is triggered, and determining a target basic data table on which the comparison depends from at least one basic data table; the storage module is used for storing the table names of the target basic data table into a monitoring record table; the processing module is used for triggering a corresponding secondary storage process, acquiring a first data table to be compared associated with the target basic data table and a second data table to be compared associated with the target basic data table, and comparing the first data table to be compared with the second data table to be compared, wherein the target basic data table is used for storing basic data associated with data to be compared in the first data table to be compared and basic data associated with data to be compared in the second data table to be compared; the second determining module is used for determining the comparison result of the first data table to be compared and the second data table to be compared; and a recording module for recording the comparison result at a position associated with the table name in the monitoring record table.
According to an embodiment of the present disclosure, the second determining module includes: a first determining unit, configured to determine a number of comparison pieces of data that are compared by the first to-be-compared data table and the second to-be-compared data table; a second determining unit configured to determine a number of matching failure pieces of data that failed to match from the number of comparison pieces; and a third determining unit, configured to determine a matching success number of the data that is successfully matched from the comparison number.
According to an embodiment of the present disclosure, the above system further includes: and the first setting module is used for setting the state of the monitoring record table to be a first state used for representing that the comparison operation is being performed depending on the target basic data table after the table name of the target basic data table is stored in the monitoring record table.
According to an embodiment of the present disclosure, the above system further includes: and the second setting module is used for setting the state of the second data table to be compared to a second state for representing the data update required under the condition that the first data table to be compared and the second data table to be compared are different.
According to an embodiment of the present disclosure, the above system further includes: the first judging module is used for judging whether the comparison process of the first data table to be compared and the second data table to be compared is abnormal or not; the third setting module is configured to set a state of the monitoring record table to a third state for representing failure in comparing the first to-be-compared data table with the second to-be-compared data table when an abnormality occurs in a comparison process of the first to-be-compared data table with the second to-be-compared data table; or a fourth setting module, configured to set, in a case where no abnormality occurs in the comparison process of the first to-be-compared data table and the second to-be-compared data table, a state of the monitoring record table to be a fourth state for characterizing that the first to-be-compared data table and the second to-be-compared data table are successfully compared.
According to an embodiment of the present disclosure, the at least one base data table includes a plurality of base data tables, and the system further includes: the second judging module is used for judging whether the comparison operation depending on the plurality of basic data tables is completed or not; and a third determining module, configured to determine, from the plurality of basic data tables, a basic data table on which the next comparison operation depends, if the comparison operation that depends on the plurality of basic data tables is not completed entirely.
A further aspect of the present disclosure provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor is adapted to implement the steps of the method as claimed in any one of the preceding claims when executing the computer program.
A further aspect of the present disclosure provides a computer readable storage medium having stored thereon a computer program, characterized in that the computer program is for implementing the steps of the method according to any of the preceding claims when being executed by a processor.
According to the data comparison monitoring method based on the storage process, when the comparison switch in the main storage process is triggered, a target basic data table on which the comparison depends is found from at least one basic data table for storing basic data, and the table name of the target basic data table is stored in the monitoring record table, so that the data compared at the present time is monitored and executed based on which basic data table. Further, triggering a corresponding secondary storage process, finding a first data table to be compared and a second data table to be compared which are associated with the target basic data table, comparing the first data table to be compared with the second data table to be compared, and further recording the comparison result at a position which is associated with the table name in the monitoring record table. According to the embodiment of the disclosure, the data comparison condition can be mastered in time by adding the monitoring record to the data comparison process, and the related information of the data comparison process is unified and generalized, so that subsequent analysis is convenient, and the defects that a monitoring mechanism does not exist in database comparison in the prior art and the data comparison state is not easy to observe are overcome.
Drawings
FIG. 1 schematically illustrates a flow chart of a stored procedure based data contrast monitoring method in accordance with an embodiment of the present disclosure;
FIG. 2 schematically illustrates a flow chart of a stored procedure based data contrast monitoring method according to another embodiment of the present disclosure;
FIG. 3 schematically illustrates a block diagram of a stored procedure based data comparison monitoring system in accordance with an embodiment of the present disclosure; and
Fig. 4 schematically illustrates a hardware architecture diagram of a computer device adapted to implement a stored procedure based data contrast monitoring method according to an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present disclosure more apparent, the present disclosure will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present disclosure. Based on the embodiments in this disclosure, all other embodiments that a person of ordinary skill in the art would obtain without making any inventive effort are within the scope of protection of this disclosure.
The defect that the database comparison in the prior art does not have a monitoring mechanism and is unfavorable for observing the data comparison state is considered. According to the data comparison monitoring method based on the storage process, when the comparison switch in the main storage process is triggered, a target basic data table on which the comparison depends is found from at least one basic data table for storing basic data, and the table name of the target basic data table is stored in the monitoring record table, so that the data compared at the present time is monitored and executed based on which basic data table. Further, triggering a corresponding secondary storage process, finding a first data table to be compared and a second data table to be compared which are associated with the target basic data table, comparing the first data table to be compared with the second data table to be compared, and further recording the comparison result at a position which is associated with the table name in the monitoring record table. According to the embodiment of the disclosure, the data comparison condition can be mastered in time by adding the monitoring record to the data comparison process, and the related information of the data comparison process is unified and generalized, so that the subsequent analysis is convenient.
Fig. 1 schematically illustrates a flow chart of a stored procedure based data contrast monitoring method according to an embodiment of the present disclosure.
As shown in fig. 1, the data comparison monitoring method based on the stored procedure may include steps S101 to S105, wherein:
Step S101, monitoring and knowing that a comparison switch in a main storage process is triggered, and determining a target basic data table on which the comparison depends from at least one basic data table.
It should be noted that, before performing data comparison, the embodiment of the present disclosure actually migrates the database in the system a (such as the first system) to the system B (such as the second system), but, because the database in the system a is updated in real time, in order to ensure that the database in the system B is consistent with the database in the system a, the database in the system a needs to be compared with the database in the system B, where the database in the system a may include a first data table to be compared, and the database in the system B may include a second data table to be compared.
In the embodiments of the present disclosure, a data migration operation has been performed before the data comparison is performed, wherein a work of basic data preparation is also required before the data migration operation. Specifically, basic data (such as a customer name, an identification card number, and a customer number) corresponding to data to be migrated (such as application information of a customer, loan information of a customer, and address information of a customer) in the data table to be migrated is stored in at least one basic data table, and then the data to be migrated can be migrated from the system a to the system B based on the at least one basic data table. At this time, the data migrated to the system B may be stored in the second data table to be compared, and the data to be migrated in the system a may be referred to as the first data table to be compared, and the data in the first data table to be compared may or may not be updated. And, the basic data stored in each basic data table are respectively associated with the first data table to be compared and the second data table to be compared. Preferably, in order to scatter data, the embodiment of the present disclosure reduces the data amount of a single table, improves the efficiency of the data migration process, and may store the above basic data into a plurality of basic data tables. In addition, the state of each basic data table may be set to a preset state, such as state U, for characterizing that each piece of data needs to be compared.
For example, there are 10 basic data tables with the same table structure, and the basic data tables are respectively: the 1 st basic data table is cis_merge_ deamon _base1, the 2 nd basic data table is cis_merge_ deamon _base2, …, the 10 th basic data table is cis_merge_ deamon _base10, and then the basic data such as customer name, identity card number, customer number and the like can be evenly dispersed into the 10 basic data tables. The basic data stored in cis_merge_ deamon _base1 is a customer name Zhao Yi, an identity card number 110110196210102563 and a customer number 1, and cis_merge_ deamon _base1 can be associated with a first data table 1-1 to be compared for storing application information of the customer in a system A, a second data table 1-1' to be compared for storing application information of the customer in a system B, a first data table 1-2 to be compared for storing loan information of the customer in a system A, a second data table 1-2' to be compared for storing loan information of the customer in a system B, and a first data table 1-3 to be compared for storing address information of the customer in a system A, and a second data table 1-3' to be compared for storing address information of the customer in a system B; …, and so on.
After the basic data preparation is completed, a data comparison operation may be performed, and embodiments of the present disclosure may monitor the data comparison process.
In particular, in performing data comparison, the embodiments of the present disclosure include a primary storage process and a secondary storage process, which may be performed based on the storage process. In the main storage process, once monitoring and knowing that a comparison switch of the main storage process is triggered, a target basic data table on which the comparison depends can be determined from at least one basic data table. For example, triggering a circulation mechanism, determining a first basic data table from at least one basic data table to perform circulation operation, triggering a secondary storage process, comparing a first data table to be compared with a second data table to be compared, which are associated with the first basic data table, and monitoring the comparison process; then, a similar operation to the above loop is performed again until the loop is completed, wherein the loop is completed based on the comparison of the last underlying data table.
In the following, a specific example will be described in detail how the target underlying data table on which the comparison depends is determined from at least one underlying data table.
For example, monitoring and knowing that a comparison switch in the main storage process is triggered, starting to execute a cycle mechanism of data comparison, if the cycle mechanism is entered for the first time, selecting a 1 st basic data table cis_merge_ deamon _base1 from 10 basic data tables as a target basic data table on which the comparison depends. If the data comparison operation based on the 1 st basic data table cis_merge_ deamon _base1 is completed, then the 2 nd basic data table cis_merge_ deamon _base2 can be selected from the 10 th basic data tables, and cis_merge_ deamon _base2 is taken as a target basic data table on which the comparison depends.
Step S102, the table name of the target basic data table is stored in the monitoring record table.
According to embodiments of the present disclosure, after determining the target base data table in the primary storage process stage, in order to facilitate monitoring on which base data table the data compared at this time is performed based on, embodiments of the present disclosure may further store the table name of the target base data table in the monitoring record table, for example, in cis_merge_ deamon _summary. Further, in order to intuitively understand what data is being stored in the monitoring record table, the state of the monitoring record table may be set to a first state, such as state CR, for characterizing that a comparison operation is being performed depending on the target underlying data table.
For example, in the process of performing data comparison based on the scheme of the present disclosure, the user finds that the first state of the monitoring record table is CR, so that it can be known that the data related to the data comparison process, but not the data related to the data migration process, is stored in the monitoring record table at this time.
How the table names of the target base data table are stored in the monitoring record table is described in detail below with a specific example.
For example, if the target base data table is cis_merge_ deamon _base1, the table name "cis_merge_ deamon _base1" may be stored into cis_merge_ deamon _summary, and the state of cis_merge_ deamon _summary may also be changed to CR.
Step S103, triggering a corresponding secondary storage process, obtaining a first data table to be compared associated with a target basic data table and a second data table to be compared associated with the target basic data table, and comparing the first data table to be compared with the second data table to be compared, wherein the target basic data table is used for storing basic data associated with data to be compared in the first data table to be compared and basic data associated with data to be compared in the second data table to be compared.
In embodiments of the present disclosure, after storing the table name of the target base data table into the monitoring record table, the corresponding secondary storage process may be triggered. After triggering the corresponding secondary storage process, firstly, obtaining preset table types (such as an application table, a loan table, a customer family information table and a customer bank card table) from a preset configuration table, then, circulating the preset table types, triggering the data comparison storage process corresponding to the preset table types, and finding a first data table to be compared and a second data table to be compared which are associated with a target basic data table according to the preset table types and association relations, wherein the first data table to be compared can comprise: a table for storing application information in the system A, a table for storing loan information in the system A, a table for storing customer family information in the system A, and a table for storing customer bank card information in the system A; the second data table to be compared may include: a table for storing application information in the system B, a table for storing loan information in the system B, a table for storing customer family information in the system B, and a table for storing customer bank card information in the system B.
How to acquire the first to-be-compared data table and the second to-be-compared data table associated with the target base data table and how to compare the first to-be-compared data table and the second to-be-compared data table are described in detail below with a specific example.
For example, if the target basic data table is cis_merge_ deamon _base1, the preset table types, which are preset, may be obtained from the preset configuration tables cis_parameters, and are respectively an application table, a loan table, a customer home information table, and a customer bank card table. Secondly, the preset table types can be circulated, if the first circulation is the application table, a data comparison storage process of the application table can be triggered, an application table (also called a first data table to be compared) related to the cis_merge_ deamon _base1 is found from the system A according to the association relation, an application table (also called a second data table to be compared) related to the cis_merge_ deamon _base1 is found from the system B, and then the application data in the application table of the system A and the application data in the application table of the system B are compared; if the second cycle is a loan table, a data comparison storage process of the loan table can be triggered, a loan table (also called a first data table to be compared) associated with the cis_merge_ deamon _base1 is found from the system A according to the association relation, a loan table (also called a second data table to be compared) associated with the cis_merge_ deamon _base1 is found from the system B, and then the loan data in the loan table of the system A and the loan data in the loan table of the system B are compared; … until all preset table types are looped through, indicating that the data comparison dependent on cis_merge_ deamon _base1 has been completed. The target base data table may further be determined to be cis_merge_ deamon _base2, and operations similar to cis_merge_ deamon _base1 may be performed … until the data comparison operation that relies on cis_merge_ deamon _base10 is complete.
It should be noted that, in the secondary storage process, when the comparison operation of a first data table to be compared and a second data table to be compared is completed, the c_count value in the monitoring record table may be added by 1 to record a total of several pieces of data. For example, if there are 2 first tables to be compared and 2 second tables to be compared associated with the 1 st basic data table, after the loop mechanism of the first data comparison is completed, the c_count corresponding to the 1 st basic data table should be 2. If there are 6 first tables to be compared and 6 second tables to be compared associated with the 2 nd basic data table, after the loop mechanism of the second data comparison is completed, the m_count corresponding to the 2 nd basic data table should be 6, and so on.
Step S104, determining comparison results of the first data table to be compared and the second data table to be compared.
Wherein, the comparison result comprises: in the comparison operation, several pieces of data (i.e. the comparison number of the data to be compared) are compared together, several pieces of data are different (i.e. the matching failure number of the data which fails to match), and several pieces of data are the same (i.e. the matching success number of the data which succeeds in matching).
According to an embodiment of the present disclosure, determining a comparison result of a first data table to be compared and a second data table to be compared includes: determining the number of comparison bars of data to be compared between the first data table to be compared and the second data table to be compared; determining the number of matching failure pieces of data which are failed to match from the number of comparison pieces; and determining the successful matching number of the successfully matched data from the comparison number.
The number of comparison bars of the data to be compared between the first data table to be compared and the second data table to be compared may be determined based on the characteristics of the orcal database, that is, after each comparison is completed, the number of comparison bars of the data to be compared, for example, 20 bars may be counted based on the characteristics of the orcal database.
The number of failed match bars of the data with failed match determined from the number of comparison bars may also be determined based on the characteristics of the orcal database, where the number of failed match bars is determined based on the characteristics of the orcal database, that is, the number of different data, for example, 6 bars, in the first data table to be compared and the second data table to be compared.
Further, the number of successful match pieces of data for which the match is successful may be determined from the number of comparison pieces: and determining the number of successful matching strips based on the number of comparison strips and the number of failed matching strips. Specifically, the difference between the number of comparison bars and the number of matching failure bars may be calculated, and the difference is taken as the number of matching success bars. For example, in combination with the above example, the number of matching success bars is 20-6=14.
It should be noted that, if the first to-be-compared data table and the second to-be-compared data table are different, that is, the matching fails, it indicates that the first to-be-compared data table has been updated in the system a, and at this time, the state of the second to-be-compared data table of the data system B may be set to a second state, for example, N, for characterizing that the data update is required.
Step S105, the comparison result is recorded at the position associated with the table name in the monitor recording table.
In embodiments of the present disclosure, after determining the comparison result, the comparison result may be recorded in the monitoring record table, in particular, to associate the comparison result with the table name of the target underlying data table.
How the comparison result is recorded in the monitor recording table at a position associated with the table name is described in detail below with a specific example.
For example, when the comparison result is recorded in the monitoring record table, the "20 comparison bars, 6 matching failure bars, 14 matching success bars" may be associated with the "client information" first, and then the "20 comparison bars, 6 matching failure bars, 14 matching success bars" may be recorded in the monitoring record table. More specifically, it may be that "20 pieces of comparison pieces, 6 pieces of matching failure pieces, 14 pieces of matching success pieces" are recorded at the same line position as "customer information" in the monitor recording table.
According to the embodiment of the disclosure, when data comparison is performed, whether the data comparison is successful can be further judged. Taking comparison of the first data table to be compared and the second data table to be compared as an example, the embodiment of the disclosure can judge whether the comparison process of the first data table to be compared and the second data table to be compared is abnormal; if yes, setting the state of the monitoring record table as a third state used for representing the comparison failure of the first data table to be compared and the second data table to be compared; or if not, setting the state of the monitoring record table as a fourth state for representing successful comparison of the first data table to be compared and the second data table to be compared.
Specifically, determining whether the comparison process of the first to-be-compared data table and the second to-be-compared data table is abnormal may be that, during the data comparison process, if the data comparison is abnormal, that is, the data comparison fails, the system returns a flag for indicating that the comparison fails, where the state of the monitoring record table may be set to a third state, for example, CF, for indicating that the comparison of the first to-be-compared data table and the second to-be-compared data table fails. If the data comparison is not abnormal, i.e. the data comparison is successful, the system returns a flag for indicating that the comparison is successful, at this time, the state of the monitoring record table may be set to a fourth state for indicating that the comparison of the first data table to be compared and the second data table to be compared is successful, for example, CS.
For example, flag 1 indicates failed contrast, flag 2 indicates successful contrast, if flag 1 is received, the state of cis_merge_ deamon _summary may be changed to CF, and if flag 2 is received, the state of cis_merge_ deamon _summary may be changed to CS.
According to the embodiment of the disclosure, if the comparison needs to be performed based on multiple basic data tables, after the completion of the cycle mechanism for performing data comparison based on the current basic data table (e.g., the target basic data table), it is further required to determine whether the comparison operation that depends on the multiple basic data tables has been completed completely; and if not, determining a basic data table on which the next comparison operation depends from the plurality of basic data tables.
For example, in combination with the above example, after the comparison operation based on cis_merge_ deamon _base1 is completed, since there are 10 basic data tables in total and comparison is currently performed based on the 1 st basic data table, it can be determined that the comparison operation corresponding to the 9 basic data tables is not completed. At this time, the 2 nd base table may be used as a base data table on which the next comparison operation depends. However, if the target underlying data table is cis_merge_ deamon _base10, it is known that the comparison is currently already performed based on the 10 th underlying data table, and at this time, the comparison operation based on the 10 underlying data tables is considered to be completed.
According to the data comparison monitoring method based on the storage process, when the comparison switch in the main storage process is triggered, a target basic data table on which the comparison depends is found from at least one basic data table for storing basic data, and the table name of the target basic data table is stored in the monitoring record table, so that the data compared at the present time is monitored and executed based on which basic data table. Further, triggering a corresponding secondary storage process, finding a first data table to be compared and a second data table to be compared which are associated with the target basic data table, comparing the first data table to be compared with the second data table to be compared, and further recording the comparison result at a position which is associated with the table name in the monitoring record table. According to the embodiment of the disclosure, the data comparison condition can be mastered in time by adding the monitoring record to the data comparison process, and the related information of the data comparison process is unified and generalized, so that subsequent analysis is convenient, and the defects that a monitoring mechanism does not exist in database comparison in the prior art and the data comparison state is not easy to observe are overcome.
Fig. 2 schematically illustrates a flow chart of a stored procedure based data contrast monitoring method according to another embodiment of the present disclosure.
As shown in fig. 2, the data comparison monitoring method based on the stored procedure may include steps S201 to S216, wherein:
in step S201, basic data preparation;
In step S202, basic data allocation;
In step S203, the state of the base data table becomes U;
in step S204, the data comparison main storage process is triggered;
in step S205, it is determined whether the switch for data comparison is turned on, if not, step S216 is executed, and if yes, step S206 is executed;
in step S206, the state of the monitoring record table is set to CR;
in step S207, a data comparison auxiliary storage process is triggered;
In step S208, a first data table to be compared and a second data table to be compared are obtained;
in step S209, triggering a record data update;
In step S210, triggering a data comparison storage process of each specific data table to be compared;
In step S211, data in the first data table to be compared and the second data table to be compared are compared;
in step S212, a comparison result is recorded;
in step S213, it is determined whether the comparison operation is abnormal, if yes, step S214 is executed, and if no, step S215 is executed;
in step S214, the state of the monitoring record table is set to CF;
In step S215, the state of the monitoring record table is set to CS;
In step S216, the process ends.
In the embodiment of the present disclosure, before performing data comparison, the basic data preparation work may be performed first, the basic data is allocated to at least one basic data table, and the state of each piece of data in each basic data table is set to U. Triggering a main storage process of data comparison, judging whether a switch in a preset configuration table is opened or not, and if not, stopping the comparison operation; if open, the state of the monitoring record table is set to CR. Triggering a data comparison auxiliary storage process, acquiring preset table types from a preset configuration table, circulating the table types, executing data comparison operation on data in a first data table to be compared and a second data table to be compared corresponding to each table type, and then recording comparison results. Judging whether the comparison operation is abnormal or not, if so, setting the state in the monitoring record table as CF; if not, the state in the monitoring record table is set as CS.
FIG. 3 schematically illustrates a block diagram of a stored procedure based data alignment monitoring system in accordance with an embodiment of the present disclosure.
As shown in fig. 3, the stored procedure-based data comparison monitoring system 300 may include a first determining module 310, a storing module 320, a processing module 330, a second determining module 340, and a recording module 350, wherein:
The first determining module 310 is configured to monitor that a comparison switch in the main storage process is triggered, and determine a target basic data table on which the comparison depends from at least one basic data table;
the storage module 320 is configured to store a table name of the target basic data table into the monitoring record table;
the processing module 330 is configured to trigger a corresponding secondary storage process, obtain a first to-be-compared data table associated with the target basic data table and a second to-be-compared data table associated with the target basic data table, and compare the first to-be-compared data table with the second to-be-compared data table, where the target basic data table is used to store basic data associated with to-be-compared data in the first to-be-compared data table and basic data associated with to-be-compared data in the second to-be-compared data table;
the second determining module 340 is configured to determine a comparison result of the first data table to be compared and the second data table to be compared; and
The recording module 350 is configured to record the comparison result in a position associated with the table name in the monitoring record table.
According to the data comparison monitoring scheme based on the storage process, when the comparison switch in the main storage process is triggered, a target basic data table on which the comparison depends is found from at least one basic data table for storing basic data, and the table name of the target basic data table is stored in a monitoring record table so as to monitor which basic data table the data compared at the present time is executed based on. Further, triggering a corresponding secondary storage process, finding a first data table to be compared and a second data table to be compared which are associated with the target basic data table, comparing the first data table to be compared with the second data table to be compared, and further recording the comparison result at a position which is associated with the table name in the monitoring record table. According to the embodiment of the disclosure, the data comparison condition can be mastered in time by adding the monitoring record to the data comparison process, and the related information of the data comparison process is unified and generalized, so that subsequent analysis is convenient, and the defects that a monitoring mechanism does not exist in database comparison in the prior art and the data comparison state is not easy to observe are overcome.
As an alternative embodiment, the second determining module includes: the first determining unit is used for determining the number of the data to be compared of the first data table to be compared and the second data table to be compared; the second determining unit is used for determining the number of matching failure pieces of data which are failed to match from the number of comparison pieces; and a third determining unit for determining the number of successful matching strips of the data which are successful in matching from the number of comparison strips.
As an alternative embodiment, the above system further comprises: and the first setting module is used for setting the state of the monitoring record table to be a first state used for representing that the comparison operation is being performed depending on the target basic data table after the table name of the target basic data table is stored in the monitoring record table.
As an alternative embodiment, the above system further comprises: the second setting module is configured to set, when the first to-be-compared data table and the second to-be-compared data table are different, a state of the second to-be-compared data table to be a second state for representing that data update is required.
As an alternative embodiment, the above system further comprises: the first judging module is used for judging whether the comparison process of the first data table to be compared and the second data table to be compared is abnormal or not; the third setting module is used for setting the state of the monitoring record table to be a third state used for representing the comparison failure of the first data table to be compared and the second data table to be compared under the condition that the comparison process of the first data table to be compared and the second data table to be compared is abnormal; or the fourth setting module is used for setting the state of the monitoring record table to be a fourth state for representing successful comparison of the first data table to be compared and the second data table to be compared under the condition that no abnormality occurs in the comparison process of the first data table to be compared and the second data table to be compared.
As an alternative embodiment, the at least one base data table comprises a plurality of base data tables, and the system further comprises: the second judging module is used for judging whether the comparison operation depending on the plurality of basic data tables is completed or not; and the third determining module is used for determining the basic data table on which the next comparison operation depends from the plurality of basic data tables under the condition that the comparison operation depending on the plurality of basic data tables is not completed completely.
Fig. 4 schematically illustrates a hardware architecture diagram of a computer device adapted to implement a stored procedure based data contrast monitoring method according to an embodiment of the present disclosure. In this embodiment, the computer device 400 is a device capable of automatically performing numerical calculation and/or information processing in accordance with instructions set or stored in advance. For example, it may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a rack server (including a stand-alone server or a server cluster composed of a plurality of servers), etc. As shown, computer device 400 includes at least, but is not limited to: the memory 410, processor 420, and network interface 430 may be communicatively coupled to each other via a system bus. Wherein:
the memory 410 includes at least one type of computer-readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 410 may be an internal storage module of the computer device 400, such as a hard disk or memory of the computer device 400. In other embodiments, the memory 410 may also be an external storage device of the computer device 400, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, abbreviated as SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the computer device 400. Of course, the memory 410 may also include both internal memory modules of the computer device 400 and external memory devices. In this embodiment, the memory 410 is typically used to store an operating system and various types of application software installed on the computer device 400, such as program codes of a data comparison monitoring method based on a stored procedure. In addition, the memory 410 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 420 may be a central processing unit (Central Processing Unit, simply CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 420 is generally used to control the overall operation of the computer device 400, such as performing control and processing related to data interaction or communication with the computer device 400, and the like. In this embodiment, the processor 420 is used to execute program code or process data stored in the memory 410.
The network interface 430 may include a wireless network interface or a wired network interface, the network interface 430 typically being used to establish a communication connection between the computer device 400 and other computer devices. For example, the network interface 430 is used to connect the computer device 400 to an external terminal through a network, establish a data transmission channel and a communication connection between the computer device 400 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a global system for mobile communications (Global System of Mobile communication, abbreviated as GSM), wideband code division multiple access (Wideband Code Division Multiple Access, abbreviated as WCDMA), a 4G network, a 5G network, bluetooth (Bluetooth), wi-Fi, etc.
It should be noted that FIG. 4 only shows a computer device having components 410-430, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead.
In this embodiment, the data comparison monitoring method based on the stored procedure stored in the memory 410 may be further divided into one or more program modules and executed by one or more processors (the processor 420 in this embodiment) to complete the present invention.
The present embodiment also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the stored procedure based data comparison monitoring method in the embodiment.
In this embodiment, the computer-readable storage medium includes a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, abbreviated as SMC), a Secure Digital (abbreviated as SD) card, a flash memory card (FLASH CARD), or the like, which are provided on the computer device. Of course, the computer-readable storage medium may also include both internal storage units of a computer device and external storage devices. In this embodiment, the computer readable storage medium is typically used to store an operating system and various types of application software installed on a computer device, such as program code of a data comparison monitoring method based on a stored procedure in the embodiment, and the like. Furthermore, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described, or they may be separately fabricated into individual integrated circuit modules, or a plurality of modules or steps in them may be fabricated into a single integrated circuit module. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. A stored procedure based data comparison monitoring method, comprising:
Monitoring and knowing that a comparison switch in the main storage process is triggered, and determining a target basic data table on which the comparison depends from at least one basic data table;
storing the table name of the target basic data table into a monitoring record table;
Triggering a corresponding secondary storage process, acquiring a first data table to be compared associated with the target basic data table and a second data table to be compared associated with the target basic data table, and comparing the first data table to be compared with the second data table to be compared, wherein the target basic data table is used for storing basic data associated with data to be compared in the first data table to be compared and basic data associated with data to be compared in the second data table to be compared;
determining comparison results of the first data table to be compared and the second data table to be compared; and
Recording the comparison result in a position associated with the table name in the monitoring record table;
Before data comparison, migrating a database in a system A to a system B, wherein the database in the system A comprises a first data table to be compared, and the database in the system B comprises a second data table to be compared; before data migration operation, a basic data preparation operation is needed, specifically, basic data corresponding to data to be migrated in a data table to be migrated is stored in at least one basic data table, the data to be migrated is migrated from a system A to a system B based on the at least one basic data table, at this time, the data migrated to the system B is stored in a second data table to be compared, and the data table to be migrated in the system A is called a first data table to be compared.
2. The method of claim 1, wherein the determining the comparison result of the first table to be compared with the second table to be compared comprises:
Determining the number of the data to be compared between the first data table to be compared and the second data table to be compared;
determining the number of matching failure pieces of data which are failed to match from the number of comparison pieces; and
And determining the successful matching number of the successfully matched data from the comparison number.
3. The method of claim 1, wherein after said storing the table names of the target base data table into a monitoring record table, the method further comprises:
the state of the monitoring record table is set to a first state for characterizing a comparison operation being performed in dependence on the target underlying data table.
4. The method according to claim 1, wherein the method further comprises:
And if the first data table to be compared is different from the second data table to be compared, setting the state of the second data table to be compared to be a second state used for representing that data updating is needed.
5. The method according to claim 1, wherein the method further comprises:
Judging whether the comparison process of the first data table to be compared and the second data table to be compared is abnormal or not; and
If yes, setting the state of the monitoring record table to be a third state used for representing the comparison failure of the first data table to be compared and the second data table to be compared; or alternatively
If not, setting the state of the monitoring record table to be a fourth state used for representing that the first data table to be compared and the second data table to be compared are successfully compared.
6. The method of claim 1, wherein the at least one base data table comprises a plurality of base data tables, the method further comprising:
judging whether the comparison operation depending on the plurality of basic data tables is completed or not; and
If not, determining a basic data table on which the next comparison operation depends from the plurality of basic data tables.
7. A stored procedure based data migration monitoring system for implementing the method of any one of claims 1 to 6, comprising:
The first determining module is used for monitoring and knowing that a comparison switch in the main storage process is triggered, and determining a target basic data table on which the comparison depends from at least one basic data table;
The storage module is used for storing the table name of the target basic data table into a monitoring record table;
The processing module is used for triggering a corresponding secondary storage process, acquiring a first data table to be compared associated with the target basic data table and a second data table to be compared associated with the target basic data table, and comparing the first data table to be compared with the second data table to be compared, wherein the target basic data table is used for storing basic data associated with data to be compared in the first data table to be compared and basic data associated with data to be compared in the second data table to be compared;
The second determining module is used for determining the comparison result of the first data table to be compared and the second data table to be compared; and
And the recording module is used for recording the comparison result at a position associated with the table name in the monitoring record table.
8. The system of claim 7, wherein the second determination module comprises:
The first determining unit is used for determining the number of the data to be compared of the first data table to be compared and the second data table to be compared;
a second determining unit, configured to determine a matching failure number of the data that fails to match from the comparison number; and
And the third determining unit is used for determining the number of successful matching strips of the data which are successful in matching from the number of the comparison strips.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor is adapted to implement the steps of the method of any one of claims 1 to 6 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program is for implementing the steps of the method according to any one of claims 1 to 6 when being executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910190040.XA CN110059071B (en) | 2019-03-13 | Data comparison monitoring method, system, equipment and medium based on storage process |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910190040.XA CN110059071B (en) | 2019-03-13 | Data comparison monitoring method, system, equipment and medium based on storage process |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110059071A CN110059071A (en) | 2019-07-26 |
| CN110059071B true CN110059071B (en) | 2024-05-31 |
Family
ID=
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722521A (en) * | 2012-04-24 | 2012-10-10 | 深圳市神盾信息技术有限公司 | Method and system for monitoring data comparison |
| EP2801943A1 (en) * | 2013-05-08 | 2014-11-12 | Wisetime Pty Ltd | A system and method for generating a chronological timesheet |
| CN104915599A (en) * | 2015-05-30 | 2015-09-16 | 广东欧珀移动通信有限公司 | Application program monitoring method and terminal |
| CN108415990A (en) * | 2018-02-12 | 2018-08-17 | 苏州朗动网络科技有限公司 | Data quality monitoring method, device, computer equipment and storage medium |
| CN108847977A (en) * | 2018-06-14 | 2018-11-20 | 平安科技（深圳）有限公司 | A kind of monitoring method of business datum, storage medium and server |
| CN108874644A (en) * | 2018-06-06 | 2018-11-23 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
| CN109189367A (en) * | 2018-06-28 | 2019-01-11 | 重庆小雨点小额贷款有限公司 | A kind of data processing method, device, server and storage medium |
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722521A (en) * | 2012-04-24 | 2012-10-10 | 深圳市神盾信息技术有限公司 | Method and system for monitoring data comparison |
| EP2801943A1 (en) * | 2013-05-08 | 2014-11-12 | Wisetime Pty Ltd | A system and method for generating a chronological timesheet |
| CN104915599A (en) * | 2015-05-30 | 2015-09-16 | 广东欧珀移动通信有限公司 | Application program monitoring method and terminal |
| CN108415990A (en) * | 2018-02-12 | 2018-08-17 | 苏州朗动网络科技有限公司 | Data quality monitoring method, device, computer equipment and storage medium |
| CN108874644A (en) * | 2018-06-06 | 2018-11-23 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
| CN108847977A (en) * | 2018-06-14 | 2018-11-20 | 平安科技（深圳）有限公司 | A kind of monitoring method of business datum, storage medium and server |
| CN109189367A (en) * | 2018-06-28 | 2019-01-11 | 重庆小雨点小额贷款有限公司 | A kind of data processing method, device, server and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113448862B (en) | Software version testing method and device and computer equipment | |
| CN110928802A (en) | Test method, device, equipment and storage medium based on automatic generation of case | |
| CN113238924B (en) | Chaotic engineering realization method and system in distributed graph database system | |
| CN110515795B (en) | Big data component monitoring method and device and electronic equipment | |
| WO2019148657A1 (en) | Method for testing associated environments, electronic device and computer readable storage medium | |
| CN107644041B (en) | Policy settlement processing method and device | |
| CN112256672B (en) | Database change approval method and device | |
| CN110059071B (en) | Data comparison monitoring method, system, equipment and medium based on storage process | |
| CN112416648A (en) | Data verification method and device | |
| CN110442466B (en) | Method, device, computer equipment and storage medium for preventing repeated access request | |
| CN112256532A (en) | Test interface generation method and device, computer equipment and readable storage medium | |
| CN109298928B (en) | Service processing method and device | |
| CN115098362B (en) | Page test method, page test device, electronic equipment and storage medium | |
| CN110647463A (en) | Method and device for restoring test breakpoint and electronic equipment | |
| CN110969430B (en) | Suspicious user identification method, suspicious user identification device, computer equipment and storage medium | |
| CN110489208B (en) | Virtual machine configuration parameter checking method, system, computer equipment and storage medium | |
| CN110674214A (en) | Big data synchronization method and device, computer equipment and storage medium | |
| CN110990052A (en) | Configuration saving method and device | |
| CN111224861B (en) | Method, system, equipment and storage medium for opening instant message service | |
| CN112464088B (en) | Recommendation request response method and device and recommendation system development method | |
| CN111400309B (en) | Data updating method, electronic device and readable storage medium | |
| CN114598755B (en) | Network protocol start sequence control method, device, equipment and storage medium | |
| CN110647526A (en) | Batch data processing method and device, computer equipment and storage medium | |
| CN117234596A (en) | Index value calculation method, device, equipment and storage medium for real-time data | |
| CN116431564A (en) | Intelligent data migration method and device based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240508 Address after: 518000, 4232, Building D1, Kexing Science Park, No. 15 Keyuan Road, Science Park Community, Yuehai Street, Nanshan District, Shenzhen City, Guangdong Province Applicant after: Shenzhen Dameng Longtu Network Technology Co.,Ltd. Country or region after: China Address before: 518052 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.) Applicant before: PING AN PUHUI ENTERPRISE MANAGEMENT Co.,Ltd. Country or region before: China |
|
| GR01 | Patent grant |