CN114510536A - Data verification method and device, computer equipment and storage medium - Google Patents
Data verification method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN114510536A CN114510536A CN202210140993.7A CN202210140993A CN114510536A CN 114510536 A CN114510536 A CN 114510536A CN 202210140993 A CN202210140993 A CN 202210140993A CN 114510536 A CN114510536 A CN 114510536A
- Authority
- CN
- China
- Prior art keywords
- data
- field
- acquiring
- file
- verification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to the field of data processing, and discloses a data checking method, a data checking device, computer equipment and a storage medium, wherein the data checking method comprises the following steps: receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; acquiring a first data file of a first system, and extracting a first field in a first table from the first data file according to a table name and a table field to be verified; mapping the first field into a standard field based on a preconfigured field mapping rule, and acquiring first data corresponding to the first field; acquiring a second field mapped by the standard field in a second system and corresponding second data; and calling a check rule to compare the first data with the second data, and if the first data and the second data meet the check rule, synchronizing the first data in the first system and the second data in the second system respectively. The method and the device can improve the efficiency and the accuracy of data synchronization among different systems.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data verification method, apparatus, computer device, and storage medium.
Background
The data verification occupies a large part of test work, batch data synchronization can be performed between different systems every day at present, due to the factors of huge data volume, complex data structure and the like, link abnormity can occur in the data synchronization process, so that data of a downstream system is frequently lost, the current scheme is that a tester audits and processes result data after each synchronization, a large amount of manpower is consumed for troubleshooting, the efficiency is low, the quality cannot be completely guaranteed, and the data verification efficiency and the accuracy are low during data synchronization.
Disclosure of Invention
The present application mainly aims to provide a data verification method, an apparatus, a computer device, and a storage medium, and aims to solve the problem of low data verification efficiency in a data synchronization process.
In order to achieve the above object, the present application provides a data verification method, including:
receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, and the data information to be verified comprises a table name to be verified and a table field;
acquiring a first data file of a first system, and extracting a first field in a first table from the first data file according to the table name and the table field to be verified;
mapping the first field into a standard field based on a preconfigured field mapping rule, and acquiring first data corresponding to the first field;
acquiring a second field mapped by the standard field in a second system and second data corresponding to the second field;
and calling a check rule to compare the first data with the second data, and if the first data and the second data meet the check rule, synchronizing the first data in the first system and the second data in the second system respectively.
Further, the obtaining a second field mapped by the standard field in the second system and second data corresponding to the second field includes:
acquiring a second data file of the second system;
acquiring a second table from the second data file;
acquiring a field to be selected mapped by the standard field;
and matching the field to be selected in the second table, acquiring a second field mapped with the standard field in the second table, and acquiring second data of the second field.
Further, the obtaining a second table from the second data file includes:
acquiring a target table name matched with the first table;
and acquiring a second table from the second data file according to the target table name.
Further, the invoking the verification rule to compare the first data with the second data includes:
acquiring third data associated with the first data in a check rule;
calculating the first data and the third data according to the check rule to obtain a first result;
calculating the second data and the third data according to the check rule to obtain a second result;
and comparing the first result with the second result so as to compare the first data with the second data.
Further, the mapping the first field to a standard field based on a preconfigured field mapping rule, comprising:
obtaining the classification type of the first field;
and according to the classification type, matching a field mapping rule of the classification type from the configuration file, and mapping the first field to a target field based on the field mapping rule.
Further, the obtaining the first data corresponding to the first field includes:
acquiring first data to be selected of the first field;
acquiring a data screening rule of the first field;
and screening the first data to be selected based on the data screening rule to obtain the screened first data corresponding to the first field.
Further, the first data and the second data satisfy the check rule, and the first data in the first system and the second data in the second system are synchronized respectively, further comprising:
respectively synchronizing first data in the first system and second data in a second system, and acquiring a second target file generated after data synchronization of the second system;
calculating an MD5 value for the first data file and an MD5 value for the second target file;
if the MD5 value of the first data file is the same as the MD5 value of the second target file, it is determined that the first data in the first system and the second data in the second system are completely synchronized.
The present application further provides a data verification apparatus, including:
the data configuration module is used for receiving a configuration instruction for data verification and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, and the data information to be verified comprises a table name to be verified and a table field;
the field acquisition module is used for acquiring a first data file of a first system and extracting a first field in a first table from the first data file according to the table name and the table field to be verified;
the data acquisition module is used for mapping the first field into a standard field based on a preconfigured field mapping rule and acquiring first data corresponding to the first field;
the synchronous system module is used for acquiring a second field mapped by the standard field in a second system and second data corresponding to the second field;
and the verification synchronization module is used for calling a verification rule to compare the first data with the second data, and if the first data and the second data meet the verification rule, synchronizing the first data in the first system with the second data in the second system respectively.
The application also provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the data verification methods when executing the computer program.
The present application further provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the data verification method of any one of the above.
The embodiment of the application provides a method for data synchronization and data verification among different systems, which comprises the steps of firstly receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, the data information to be verified comprises a table name and a table field to be verified, then a first data file of a first system is obtained, a first field in a first table is extracted from the first data file as data information to be verified according to the table name and the table field to be verified, the first field is mapped into a standard field based on a pre-configured field mapping rule, first data corresponding to the first field is obtained, a second field mapped by the standard field in a second system and second data corresponding to the second field are obtained, a verification rule is called to compare the first data with the second data, if the first data and the second data meet the verification rule, the first data in the first system and the second data in the second system are synchronized, through mapping of data fields among different systems and comparison and verification of data information of the mapped data fields, the first data and the second data meet the verification rule, the first data in the first system and the second data in the second system are synchronized respectively, namely the second data in the second system are written into a database of the second system, data verification efficiency during data synchronization process among different systems is achieved, and therefore efficiency and accuracy of data synchronization among different systems are improved.
Drawings
FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a data verification method according to the present application;
FIG. 2 is a schematic structural diagram of an embodiment of a data verification apparatus according to the present application;
FIG. 3 is a block diagram illustrating a computer device according to an embodiment of the present invention.
The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Referring to fig. 1, an embodiment of the present application provides a data verification method, including steps S10-S50, and details of each step of the data verification method are described as follows.
S10, receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, and the data information to be verified comprises a table name to be verified and a table field.
The embodiment is applied to data synchronization and verification scenes among different systems, wherein large-batch data synchronization is required to be performed among the different systems every day, and because the data of each system is independent, the data of an upstream system needs to be gradually synchronized to a downstream system. Firstly, a configuration file of data verification is formulated, namely a configuration instruction of the data verification is received, the configuration file of the data verification is generated according to the configuration instruction, the configuration instruction comprises configuration of data to be verified, verification rule configuration of each item of data and configuration of how the data is positioned and acquired, namely the generated configuration file comprises data information to be verified, and the data information to be verified comprises a table name and a table field to be verified.
S20, acquiring a first data file of a first system, and extracting a first field in a first table from the first data file according to the table name and the table field to be verified.
In the embodiment, a configuration instruction for data verification is received, and a configuration file for data verification is generated according to the configuration instruction; the configuration file comprises data information to be verified, the first data file of a first system is taken after the data information to be verified comprises a list name and a list field to be verified, and the first field in the first list is extracted from the first data file according to the list name and the list field to be verified. In one implementation, the type of a first data file is obtained, the first data file is converted into a preset standard type such as an xml type according to the type of the first data file, and a first field in a first table is extracted from the first data file as data information to be checked according to the name and the field of the table to be checked.
S30, mapping the first field into a standard field based on a pre-configured field mapping rule, and acquiring first data corresponding to the first field.
In this embodiment, after a first data file of a first system is obtained, a first field in a first table is extracted from the first data file according to the table name and the table field to be verified, the first field is mapped into a standard field based on a preconfigured field mapping rule, specifically, the preconfigured field mapping rule is also stored in the configuration file, since different fields may be used to represent the same information in different systems, for example, to represent an address, an address field is used in the system 1, and an address field is used in the system 2, but the two fields are fields representing the same information and have a mapping relationship, a field mapping rule is determined based on the mapping relationship of the different fields representing the same information, then the first field is mapped into the standard field based on the preconfigured field mapping rule, and first data corresponding to the first field is obtained, i.e. the data contained in said first field is obtained.
S40, acquiring a second field mapped by the standard field in a second system and second data corresponding to the second field.
In this embodiment, after mapping the first field to a standard field based on a preconfigured field mapping rule and acquiring first data corresponding to the first field, a system that needs to perform data synchronization is defined as a second system, and after data is migrated to the second system, it is necessary to check whether data in the migrated second system is consistent with data in the first system, that is, acquire a second field mapped to the standard field in the second system and second data corresponding to the second field.
S50, calling a check rule to compare the first data with the second data, and if the first data and the second data meet the check rule, synchronizing the first data in the first system and the second data in the second system respectively.
In this embodiment, after obtaining first data corresponding to a first field in a first system, and obtaining a second field mapped by the standard field in a second system and second data corresponding to the second field, because the first field and the second field are fields representing the same information, in the data migration synchronization process, it should be ensured that the data corresponding to the field are consistent or meet other requirements of the two fields, for example, after adding 1 to the first data of the first field, the first data is synchronized into the second data of the second field, a check rule is invoked to compare the first data with the second data, and if the first data and the second data meet the check rule, the first data in the first system and the second data in the second system are synchronized respectively. In an embodiment, the first data and the second data are respectively substituted into an operation logic of the check rule, then the operation logic is executed, if the operation logic is satisfied, it is determined that the first data and the second data satisfy the check rule, at this time, the first data in the first system and the second data in the second system are respectively synchronized, that is, the second data in the second system is written into a database of the second system, and the synchronization of data between the systems is completed, so that the efficiency and the accuracy of data synchronization between different systems are improved.
The embodiment provides a method for data synchronization and data verification among different systems, which includes the steps of firstly receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, the data information to be verified comprises a table name and a table field to be verified, then a first data file of a first system is obtained, a first field in a first table is extracted from the first data file as data information to be verified according to the table name and the table field to be verified, the first field is mapped into a standard field based on a pre-configured field mapping rule, first data corresponding to the first field is obtained, a second field mapped by the standard field in a second system and second data corresponding to the second field are obtained, a verification rule is called to compare the first data with the second data, if the first data and the second data meet the verification rule, the first data in the first system and the second data in the second system are synchronized, through mapping of data fields among different systems and comparison and verification of data information of the mapped data fields, the first data and the second data meet the verification rule, the first data in the first system and the second data in the second system are synchronized respectively, namely the second data in the second system are written into a database of the second system, data verification efficiency during data synchronization process among different systems is achieved, and therefore efficiency and accuracy of data synchronization among different systems are improved.
In one embodiment, the obtaining a second field of the standard field mapping and second data corresponding to the second field in a second system includes:
acquiring a second data file of the second system;
acquiring a second table from the second data file;
acquiring a field to be selected mapped by the standard field;
and matching the field to be selected in the second table, acquiring a second field mapped with the standard field in the second table, and acquiring second data of the second field.
In this embodiment, in the process of acquiring the second field mapped by the standard field in the second system and the second data corresponding to the second field, first, the second data file of the second system is acquired, then, the data table in the second data file is located, namely, a second table is obtained from the second data file, and then the field to be selected mapped by the standard field is obtained, i.e. which fields the standard fields of the first field mapping also contain, define these fields as candidate fields, then matching is carried out on the second table according to the field to be selected, a second field which is mapped with the standard field segment in the second table is obtained, and second data of the second field is obtained, the method can automatically perform synchronous check on the data of the fields with the encounter mapping in each system, thereby improving the accuracy of data synchronization.
In one embodiment, the obtaining the second table from the second data file includes:
acquiring a target table name matched with the first table;
and acquiring a second table from the second data file according to the target table name.
In this embodiment, in the process of acquiring the second table from the second data file, the names of the target tables matching with the first table, that is, the names of the tables storing the same data between different systems, are acquired, for example, the name of table 1 in system 1 is a user information table, the name of table 2 in system 2 is a user table, and tables 1 and 2 represent the same data, that is, the name of table 1 and the name of table 2 are mapped to each other, the name of table 2 mapped to each other may be determined by acquiring the name of table 1, that is, the name of the target table matching with the first table, and then the second table is acquired from the second data file according to the name of the target table, so that the tables in the second data file are accurately located, and the accuracy of data synchronization is improved.
In one embodiment, the invoking the verification rule to compare the first data with the second data includes:
acquiring third data associated with the first data in a check rule;
calculating the first data and the third data according to the check rule to obtain a first result;
calculating the second data and the third data according to the check rule to obtain a second result;
and comparing the first result with the second result so as to compare the first data with the second data.
In this embodiment, in the process of calling the check rule to compare the first data with the second data, because an association relationship exists between part of the data, the data having the association relationship needs to ensure that the data itself is correct, and it needs to ensure that each item of data is still correct after being associated, in the process of checking the first data with the second data, first, third data associated with the first data in the check rule is obtained, then, the first data and the third data are calculated according to the check rule to obtain a first result, then, the second data and the third data are calculated according to the check rule to obtain a second result, then, the first result and the second result are compared to compare the first data with the second data, so as to compare not only the first data with the second data, and comparing the first data and the second data after the third data are associated, and determining that the first data and the second data pass verification only if the first data and the second data are correct and the first data and the second data are also correct after the third data are associated respectively, so that the verification accuracy rate after data synchronization is improved.
In one embodiment, the mapping the first field to a standard field based on a preconfigured field mapping rule comprises:
obtaining the classification type of the first field;
and according to the classification type, matching a field mapping rule of the classification type from the configuration file, and mapping the first field to a target field based on the field mapping rule.
In this embodiment, in the process of mapping the first field to the standard field based on the preconfigured field mapping rule, the classification type of the first field is obtained, the mapping rules of different classification types are different, and the mapping rule is written in the configuration file, that is, the field mapping rule of the classification type is matched from the configuration file according to the classification type, and the first field is mapped to the target field based on the field mapping rule, so that the accuracy of field mapping is improved.
In one embodiment, the obtaining the first data corresponding to the first field includes:
acquiring first data to be selected of the first field;
acquiring a data screening rule of the first field;
and screening the first data to be selected based on the data screening rule to obtain the screened first data corresponding to the first field.
In this embodiment, in the process of obtaining the first data corresponding to the first field, the first data also needs to be screened, and specifically, the first data to be selected of the first field is obtained. Defining the data of the first field which is not screened as first data to be selected, then obtaining a data screening rule of the first field, and screening the first data to be selected based on the data screening rule to obtain the first data corresponding to the first field after screening, for example, removing the data of which the first data does not accord with a specified data range, or removing the data of which the character type does not accord with the specification in the first data to be selected. By screening the data which do not meet the requirements, the data volume and the data complexity during data comparison are reduced, and the data verification efficiency is improved.
In one embodiment, if the first data and the second data satisfy the check rule, synchronizing the first data in the first system and the second data in the second system respectively further includes:
respectively synchronizing first data in the first system and second data in a second system, and acquiring a second target file generated after data synchronization of the second system;
calculating an MD5 value for the first data file and an MD5 value for the second target file;
and if the MD5 value of the first data file is the same as the MD5 value of the second target file, determining that the first data in the first system and the second data in the second system are completely synchronized.
In this embodiment, in the process of determining that the first data and the second data satisfy the check rule, and synchronizing the first data in the first system and the second data in the second system, the synchronized data file is checked, and only when the data file passes the check, it is determined that the data is synchronized, specifically, the first data in the first system and the second data in the second system are synchronized, and a second target file generated after the data synchronization of the second system is obtained, and then an MD5 value of the first data file is calculated, and an MD5 value of the second target file is calculated, and if an MD5 value of the first data file is the same as an MD5 value of the second target file, it is determined that the first data in the first system and the second data in the second system are synchronized, first, each synchronized data is compared, and then, the files generated by the synchronized data are compared, so that the correctness of data synchronization is ensured.
Referring to fig. 2, the present application further provides a data verification apparatus, including:
the data configuration module 10 is configured to receive a configuration instruction for data verification, and generate a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, and the data information to be verified comprises a table name to be verified and a table field;
a field obtaining module 20, configured to obtain a first data file of a first system, and extract a first field in a first table from the first data file according to the table name and the table field to be verified;
the data obtaining module 30 is configured to map the first field into a standard field based on a preconfigured field mapping rule, and obtain first data corresponding to the first field;
a synchronization system module 40, configured to obtain a second field mapped by the standard field in a second system and second data corresponding to the second field;
and the verification synchronization module 50 is configured to invoke a verification rule to compare the first data with the second data, and synchronize the first data in the first system and the second data in the second system respectively if the first data and the second data meet the verification rule.
As described above, it can be understood that each component of the data verification apparatus proposed in the present application may implement the function of any one of the data verification methods described above.
In one embodiment, the obtaining a second field of the standard field mapping and second data corresponding to the second field in a second system includes:
acquiring a second data file of the second system;
acquiring a second table from the second data file;
acquiring a field to be selected mapped by the standard field;
and matching the field to be selected in the second table, acquiring a second field mapped with the standard field in the second table, and acquiring second data of the second field.
In one embodiment, the obtaining the second table from the second data file includes:
acquiring a target table name matched with the first table;
and acquiring a second table from the second data file according to the target table name.
In one embodiment, the invoking the verification rule to compare the first data with the second data includes:
acquiring third data associated with the first data in a check rule;
calculating the first data and the third data according to the check rule to obtain a first result;
calculating the second data and the third data according to the check rule to obtain a second result;
and comparing the first result with the second result so as to compare the first data with the second data.
In one embodiment, the mapping the first field to a standard field based on a preconfigured field mapping rule comprises:
obtaining the classification type of the first field;
and according to the classification type, matching a field mapping rule of the classification type from the configuration file, and mapping the first field to a target field based on the field mapping rule.
In one embodiment, the obtaining the first data corresponding to the first field includes:
acquiring first data to be selected of the first field;
acquiring a data screening rule of the first field;
and screening the first data to be selected based on the data screening rule to obtain the screened first data corresponding to the first field.
In one embodiment, if the first data and the second data satisfy the check rule, synchronizing the first data in the first system and the second data in the second system respectively further includes:
respectively synchronizing first data in the first system and second data in a second system, and acquiring a second target file generated after data synchronization of the second system;
calculating an MD5 value for the first data file and an MD5 value for the second target file;
if the MD5 value of the first data file is the same as the MD5 value of the second target file, it is determined that the first data in the first system and the second data in the second system are completely synchronized.
Referring to fig. 3, a computer device, which may be a mobile terminal and whose internal structure may be as shown in fig. 3, is also provided in the embodiment of the present application. The computer equipment comprises a processor, a memory, a network interface, a display device and an input device which are connected through a system bus. Wherein, the network interface of the computer equipment is used for communicating with an external terminal through network connection. The input means of the computer device is for receiving input from a user. The computer designed processor is used to provide computational and control capabilities. The memory of the computer device includes a storage medium. The storage medium stores an operating system, a computer program, and a database. The database of the computer device is used for storing data. The computer program is executed by a processor to implement a data verification method.
The processor executes the data verification method, and the method comprises the following steps: receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, and the data information to be verified comprises a table name to be verified and a table field; acquiring a first data file of a first system, and extracting a first field in a first table from the first data file according to the table name and the table field to be verified; mapping the first field into a standard field based on a preconfigured field mapping rule, and acquiring first data corresponding to the first field; acquiring a second field mapped by the standard field in a second system and second data corresponding to the second field; and calling a check rule to compare the first data with the second data, and if the first data and the second data meet the check rule, synchronizing the first data in the first system with the second data in the second system.
The computer equipment provides a method for data synchronization and data verification among different systems, and the method comprises the steps of firstly receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, the data information to be verified comprises a table name and a table field to be verified, then a first data file of a first system is obtained, a first field in a first table is extracted from the first data file as data information to be verified according to the table name and the table field to be verified, the first field is mapped into a standard field based on a pre-configured field mapping rule, first data corresponding to the first field is obtained, a second field mapped by the standard field in a second system and second data corresponding to the second field are obtained, a verification rule is called to compare the first data with the second data, if the first data and the second data meet the verification rule, the first data in the first system and the second data in the second system are synchronized, through mapping of data fields among different systems and comparison and verification of data information of the mapped data fields, the first data and the second data meet the verification rule, the first data in the first system and the second data in the second system are synchronized, namely the second data in the second system are written into a database of the second system, data verification efficiency during data synchronization process among different systems is achieved, and therefore efficiency and accuracy of data synchronization among different systems are improved.
An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by the processor, implements a data verification method, including the steps of: receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, and the data information to be verified comprises a table name to be verified and a table field; acquiring a first data file of a first system, and extracting a first field in a first table from the first data file according to the table name and the table field to be verified; mapping the first field into a standard field based on a preconfigured field mapping rule, and acquiring first data corresponding to the first field; acquiring a second field mapped by the standard field in a second system and second data corresponding to the second field; and calling a check rule to compare the first data with the second data, and if the first data and the second data meet the check rule, synchronizing the first data in the first system with the second data in the second system.
The computer readable storage medium provides a method for data synchronization and data verification among different systems, and the method comprises the steps of firstly receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, the data information to be verified comprises a table name and a table field to be verified, then a first data file of a first system is obtained, a first field in a first table is extracted from the first data file as data information to be verified according to the table name and the table field to be verified, the first field is mapped into a standard field based on a pre-configured field mapping rule, first data corresponding to the first field is obtained, a second field mapped by the standard field in a second system and second data corresponding to the second field are obtained, a verification rule is called to compare the first data with the second data, if the first data and the second data meet the verification rule, the first data in the first system and the second data in the second system are synchronized, through mapping of data fields among different systems and comparison and verification of data information of the mapped data fields, the first data and the second data meet the verification rule, the first data in the first system and the second data in the second system are synchronized, namely the second data in the second system are written into a database of the second system, data verification efficiency during data synchronization process among different systems is achieved, and therefore efficiency and accuracy of data synchronization among different systems are improved.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above.
Any reference to memory, storage, database, or other medium provided herein and used in the embodiments may include non-volatile and/or volatile memory.
Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.
The above description is only a preferred embodiment of the present application and is not intended to limit the scope of the present application.
All the equivalent structures or equivalent processes performed by using the contents of the specification and the drawings of the present application, or directly or indirectly applied to other related technical fields, are included in the scope of protection of the present application.
Claims (10)
1. A method for data verification, comprising:
receiving a configuration instruction for data verification, and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, and the data information to be verified comprises a table name to be verified and a table field;
acquiring a first data file of a first system, and extracting a first field in a first table from the first data file according to the table name and the table field to be verified;
mapping the first field into a standard field based on a preconfigured field mapping rule, and acquiring first data corresponding to the first field;
acquiring a second field mapped by the standard field in a second system and second data corresponding to the second field;
and calling a check rule to compare the first data with the second data, and if the first data and the second data meet the check rule, synchronizing the first data in the first system and the second data in the second system respectively.
2. The method of claim 1, wherein the obtaining a second field mapped by the standard field in the second system and second data corresponding to the second field comprises:
acquiring a second data file of the second system;
acquiring a second table from the second data file;
acquiring a field to be selected mapped by the standard field;
and matching the fields to be selected in the second table, acquiring a second field mapped with the standard field in the second table, and acquiring second data of the second field.
3. The data verification method of claim 2, wherein the retrieving the second table from the second data file comprises:
acquiring a target table name matched with the first table;
and acquiring a second table from the second data file according to the target table name.
4. The data verification method of claim 1, wherein the invoking the verification rule to compare the first data with the second data comprises:
acquiring third data associated with the first data in a check rule;
calculating the first data and the third data according to the check rule to obtain a first result;
calculating the second data and the third data according to the check rule to obtain a second result;
and comparing the first result with the second result so as to compare the first data with the second data.
5. The data verification method of claim 1, wherein mapping the first field to a standard field based on a preconfigured field mapping rule comprises:
obtaining the classification type of the first field;
and according to the classification type, matching a field mapping rule of the classification type from the configuration file, and mapping the first field to a target field based on the field mapping rule.
6. The data verification method of claim 1, wherein the obtaining first data corresponding to the first field comprises:
acquiring first data to be selected of the first field;
acquiring a data screening rule of the first field;
and screening the first data to be selected based on the data screening rule to obtain the screened first data corresponding to the first field.
7. The data verification method of claim 1, wherein if the first data and the second data satisfy the verification rule, synchronizing the first data in the first system and the second data in the second system respectively, further comprising:
respectively synchronizing first data in the first system and second data in a second system, and acquiring a second target file generated after data synchronization of the second system;
calculating an MD5 value for the first data file and an MD5 value for the second target file;
and if the MD5 value of the first data file is the same as the MD5 value of the second target file, determining that the first data in the first system and the second data in the second system are completely synchronized.
8. A data verification apparatus, comprising:
the data configuration module is used for receiving a configuration instruction for data verification and generating a configuration file for data verification according to the configuration instruction; the configuration file comprises data information to be verified, and the data information to be verified comprises a table name to be verified and a table field;
the field acquisition module is used for acquiring a first data file of a first system and extracting a first field in a first table from the first data file according to the table name and the table field to be verified;
the data acquisition module is used for mapping the first field into a standard field based on a preconfigured field mapping rule and acquiring first data corresponding to the first field;
the synchronous system module is used for acquiring a second field mapped by the standard field in a second system and second data corresponding to the second field;
and the verification synchronization module is used for calling a verification rule to compare the first data with the second data, and if the first data and the second data meet the verification rule, synchronizing the first data in the first system with the second data in the second system respectively.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor when executing the computer program performs the steps of the data verification method of any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the data verification method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210140993.7A CN114510536A (en) | 2022-02-16 | 2022-02-16 | Data verification method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210140993.7A CN114510536A (en) | 2022-02-16 | 2022-02-16 | Data verification method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114510536A true CN114510536A (en) | 2022-05-17 |
Family
ID=81551151
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210140993.7A Pending CN114510536A (en) | 2022-02-16 | 2022-02-16 | Data verification method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114510536A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150142A (en) * | 2023-04-14 | 2023-05-23 | 成都数联云算科技有限公司 | Method, device, medium, equipment and product for field verification and rule obtaining thereof |
| CN117009998A (en) * | 2023-08-29 | 2023-11-07 | 上海倍通医药科技咨询有限公司 | Data inspection method and system |
-
2022
- 2022-02-16 CN CN202210140993.7A patent/CN114510536A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150142A (en) * | 2023-04-14 | 2023-05-23 | 成都数联云算科技有限公司 | Method, device, medium, equipment and product for field verification and rule obtaining thereof |
| CN116150142B (en) * | 2023-04-14 | 2023-06-27 | 成都数联云算科技有限公司 | Method, device, medium, equipment and product for field verification and rule obtaining thereof |
| CN117009998A (en) * | 2023-08-29 | 2023-11-07 | 上海倍通医药科技咨询有限公司 | Data inspection method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108874644B (en) | Data monitoring method and device, computer equipment and storage medium | |
| CN114510536A (en) | Data verification method and device, computer equipment and storage medium | |
| CN111177005A (en) | Service application testing method, device, server and storage medium | |
| CN109800258B (en) | Data file deployment method, device, computer equipment and storage medium | |
| CN108874661B (en) | Test mapping relation library generation method and device, computer equipment and storage medium | |
| CN113282513B (en) | Interface test case generation method and device, computer equipment and storage medium | |
| CN111797026A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN112363937B (en) | Differential coverage rate testing method, differential coverage rate testing device, computer equipment and storage medium | |
| CN113656404A (en) | Data verification method and device, computer equipment and storage medium | |
| CN110750443A (en) | Webpage testing method and device, computer equipment and storage medium | |
| CN114610635A (en) | Interface test case generation method and device, computer equipment and storage medium | |
| CN112395202A (en) | Interface automation test method and device, computer equipment and storage medium | |
| CN112612706A (en) | Automated testing method, computer device and storage medium | |
| CN112527286A (en) | Service code generation method and device, computer equipment and storage medium | |
| CN113282515A (en) | System testing method, device, computer equipment and storage medium | |
| CN112363929A (en) | System online method and device, computer equipment and storage medium | |
| CN112463630A (en) | Version difference testing method and device, computer equipment and storage medium | |
| CN114218188A (en) | Data migration method, device, equipment and storage medium | |
| CN112541182B (en) | Kernel VFS layer system repairing method, device, equipment and storage medium | |
| CN113052463B (en) | Workflow verification method, workflow verification device, computer equipment and storage medium | |
| CN112347095B (en) | Data table processing method, device and server | |
| CN114048082A (en) | Configuration method and device for interface test, computer equipment and storage medium | |
| CN114090660A (en) | Configuration method, device, equipment and storage medium for insurance product application data | |
| CN112685309A (en) | Unit testing method, unit testing device, storage medium and computer equipment | |
| CN114237562A (en) | SQL statement generation method, device, equipment and storage medium of functional function |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |