CN115422180A - Data verification method and system - Google Patents
Data verification method and system Download PDFInfo
- Publication number
- CN115422180A CN115422180A CN202210865768.XA CN202210865768A CN115422180A CN 115422180 A CN115422180 A CN 115422180A CN 202210865768 A CN202210865768 A CN 202210865768A CN 115422180 A CN115422180 A CN 115422180A
- Authority
- CN
- China
- Prior art keywords
- data
- database table
- source database
- verification
- data verification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24558—Binary matching operations
- G06F16/2456—Join operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data checking method and a system, wherein the method comprises the following steps: segmenting the source database table according to the data verification task set in the preset configuration file and the data quantity of the source database table; sequentially caching first data corresponding to each segment in the source database table after segmentation and second data in the target database table; and comparing the first data with the second data to obtain a data verification result after data migration, and emptying the first data and the second data in the cache after data comparison is completed each time. The system performs the method. The invention is different from the traditional sampling data checking mode, and can realize high-efficiency, accurate and comprehensive data checking by processing the data to be checked in a segmented mode and emptying the data to be checked in the cache after data comparison is completed each time.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data verification method and system.
Background
Financial data is rigorous, data magnitude is large, data migration, data loading, data processing and the like are often involved, and in order to verify data correctness, data in a database and a data table in the database need to be compared.
The traditional method is to manually extract several pieces of data for verification (for example, several pieces of data in a table in one database are extracted, another database table is searched, and then data on two sides are compared), or a tool is used to extract thousands or ten thousands of pieces of data for comparison (for example, a tool is used to extract thousands or ten thousands of pieces of data to the local, and then data are obtained from a table to be compared according to sentences and are linked to the local for data splicing, and then comparison is performed after unique data on two sides are found).
Disclosure of Invention
The data verification method and the data verification system provided by the invention are used for solving the problems in the prior art, are different from the traditional sampling data verification mode, and can realize efficient, accurate and comprehensive data verification by performing segmented processing on data to be verified and emptying the data to be verified in a cache after data comparison is completed each time, so that little memory or storage resource is occupied.
The invention provides a data checking method, which comprises the following steps:
segmenting the source database table according to a data verification task and the data quantity of the source database table, wherein the data verification task comprises the maximum data quantity of each data comparison, and the maximum number of segments is determined according to the data quantity of the source database table and the maximum data quantity of each data comparison set in the data verification task;
sequentially caching first data corresponding to each segment in the source database table after the segmentation and second data in the target database table, wherein the second data are obtained after the first data are migrated to the target database table;
and comparing the first data with the second data to obtain a data verification result after data migration, wherein the first data and the second data in the cache are emptied after data comparison is completed each time.
According to a data verification method provided by the present invention, the source database table and the target database table are obtained by:
and acquiring the source database table and the target database table according to the identifier of the source database table and the identifier of the target database table in the preset configuration file.
According to a data verification method provided by the present invention, before acquiring the source database table and the target database table according to the preset configuration file, the identifier of the source database table, and the identifier of the target database table, the method further includes:
determining a first connection mode of the source database according to the type of the source database in the preset configuration file;
establishing connection with the source database according to the first connection mode;
determining a second connection mode of the target database according to the type of the target database in the preset configuration file;
and establishing connection with the target database according to the second connection mode.
According to the data verification method provided by the invention, the source database is a relational database or a non-relational database, and the target database is the relational database or the non-relational database.
According to a data verification method provided by the invention, the method further comprises the following steps:
and checking the source database table and the target database table according to the data checking type in the data checking task.
According to the data verification method provided by the invention, the data verification type comprises a structured verification type and a content verification type.
The present invention also provides a data verification system, comprising: the system comprises a data acquisition module, a data cache module and a data verification module;
the data acquisition module is used for segmenting the source database table according to a data verification task set in a preset configuration file and the data quantity of the source database table, wherein the data verification task comprises the maximum data quantity of each data comparison, and the maximum segment number is determined according to the data quantity of the source database table and the maximum data quantity of each data comparison set in the data verification task;
the data caching module is configured to sequentially cache first data corresponding to each segment in the source database table after the segment is segmented and second data in the target database table, where the second data is obtained by migrating the first data to the target database table;
the data verification module is configured to compare the first data with the second data, obtain a data verification result after data migration, and empty the first data and the second data in the cache after data comparison is completed each time.
The invention also provides an electronic device, which comprises a processor and a memory storing a computer program, wherein the processor executes the program to realize the data verification method.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data verification method as described in any of the above.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, implements a data verification method as described in any one of the above.
The data verification method and the data verification system provided by the invention are different from the traditional sampling data verification mode, and can realize efficient, accurate and comprehensive data verification by processing the data to be verified in a segmented mode and clearing the data to be verified in the cache after data comparison is completed each time, so that little memory or storage resource is occupied.
Drawings
In order to more clearly illustrate the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic flow chart of a data verification method provided by the present invention;
FIG. 2 is a schematic structural diagram of a data verification system provided in the present invention;
fig. 3 is a schematic physical structure diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flow chart of a data verification method provided by the present invention, and as shown in fig. 1, the method includes:
the data verification task comprises the maximum data volume of each data comparison, and the maximum segment number is determined according to the data volume of the source database table and the maximum data volume of each data comparison set in the data verification task;
the second data is obtained by migrating the first data to a target database table;
and clearing the first data and the second data in the cache after the data comparison is completed each time.
It should be noted that the execution subject of the method may be a computer device.
Optionally, the data verification method provided by the invention can be suitable for comparing the total amount of the financial data so as to solve the problem that the total amount of the financial data is compared in a traditional sampling verification mode and improve the test accuracy. In step 100, a preset configuration file (i.e., a preset configuration file) is obtained, where a data verification task is set in the preset configuration file, and the data verification task includes a maximum data amount for each data comparison configured.
And segmenting the source database table according to the data verification task in the acquired preset configuration file and the data table (hereinafter referred to as the source database table) in the source database.
In one embodiment, assume that the source database table A is as follows:
TABLE A
In one embodiment, the data verification task setting data verification type in the preset configuration file comprises content structure types such as identification number, name, age and gender. The identity card number is used as a key word, the uniqueness of the data can be marked, the name has the uniqueness that the data can not be marked repeatedly, and a plurality of data verification types can be used as the key words (such as name, age and sex).
In one embodiment, the table a is segmented by rows, the table a contains 4 rows of data, the data amount of the data is 4, and the maximum data amount per data comparison set by the data verification task is 2, the table a is divided into two segments, specifically, the data in the first row and the third row is first segmented data, and the data in the second row and the fourth row is second segmented data.
In step 200, each piece of segmented data (i.e., the first data) may be sequentially cached in a cursor setting manner, and corresponding second data is screened from a data table (hereinafter referred to as a target database table) in the target database according to the set data verification type, where the second data is obtained after the first data is migrated to the target database table.
In one embodiment, assuming that the set data check types include identification number, name, age, and gender (with the identification number as a key), the target database table B is as follows:
table B
The names, ages and sexes corresponding to the identification numbers 1 and 3 are respectively screened from the table B to be second data, and hundreds of fields are screened in actual use to be in one-to-one correspondence to be compared.
The data with the identification number 1 in the table A are respectively named one, age 4 and gender male according to the data verification types in the configured data verification task, the data with the identification number 1 in the table B are named one, age 4 and gender male according to the column name sequence of the configuration reading data, the comparison is consistent, the comparison result is consistent, if a certain field is different, the result age is wrong if the certain field is different, the age of the identification number 1 in the table A is age, and the age of the identification number 1 in the table B is age.
In one embodiment, after each data comparison is completed, the first data and the second data cached in the memory or the storage resource are cleared, so that no additional memory or storage resource is occupied during the next data comparison, a multi-thread service is developed, the comparison efficiency is improved by segmented inquiry, and the comparison and verification of the full data with unlimited data magnitude are realized.
In one embodiment, if the data caching is performed by dividing the data into 2 segments or 2 threads, marking is performed on a first line (identity card number is 1) and a third line (identity card number is 3) of a table A, a keyword (identity card number) is used for finding out a name, an age and a sex corresponding to the keyword in a table B and comparing the names, the age and the sex, the data verification result is displayed in real time, after the marking is finished, the released mark jumps to a second line (identity card number is 2) and the fourth line (identity card number is 4) to be compared at the same time, if the execution is finished in the middle or interrupted, the data verification results of the first line and the third line are still displayed, and meanwhile, the electronic form such as excel is also stored to screen the data to be verified, and the data are visually compared. It can be seen which fields find the number of problems.
In one embodiment, the data verification result is displayed on a screen in real time, and is compared at any time and output at any time. And the method supports multi-format text output such as excel and the like, and can screen the error position of a certain field through excel when the field content is required to be compared with more field contents. When a certain type of errors of a certain field reach the set threshold, the comparison is stopped in time, the logic problem of positioning migration or loading processing is found or positioned according to the displayed data verification result and the excel result, the time can be saved, and the error positioning and reason searching are not required to be carried out after all the data verification results are obtained.
Meanwhile, a threshold value can be set, when the comparison error frequency of a certain field is greater than a certain numerical value, the comparison can be stopped, the field is rechecked and corrected, and the logic problem of migration or loading processing is found or positioned.
Meanwhile, data loading verification, data migration verification and comparison of texts of different types and databases of different types are integrated, and iterative test verification is continuously realized.
In actual use, more than millions of data are compared, if the data are stored in a local or another library under a common condition, a large amount of memories or storage can be occupied, the data are dragged to a local 8G memory and can only be more than tens of thousands of data, the subsequent data are exponentially increased, the millions of data possibly occupy dozens of G of spaces according to the size of a table, and under the condition of resource shortage, the data can be comprehensively checked under the condition of occupying a small amount of memories and spaces by means of segmenting, branching and cursor setting through the method.
The data verification method provided by the invention is different from the traditional sampling data verification mode, and can realize efficient, accurate and comprehensive data verification by performing segmented processing on the data to be verified and emptying the data to be verified in the cache after data comparison is completed each time, so that little memory or storage resource is occupied.
Further, in one embodiment, the source database table and the target database table in step 100 are obtained by:
step 1001, acquiring a source database table and a target database table according to the identifier of the source database table and the identifier of the target database table in the preset configuration file.
Optionally, at the stage of presetting the configuration file, the data verification type is preset; if the check type relates to table structure and contents, configuring the table to be checked. And data verification is performed according to pre-configured information about the table, such as identification (which can be a table name), so that the accuracy and efficiency of verification are improved.
In an embodiment, the source database table and the target database table to be checked are obtained according to the identifier of the source database table (which may be the table name of the source database table, such as table a) and the identifier of the target database table (which may be the table name of the target database table, such as table B) set in the preset configuration file.
The data verification method provided by the invention is different from the traditional sampling data verification mode, the data to be verified is processed in a segmented mode, the data to be verified in the cache is cleared after the data comparison is completed each time, the data can be verified efficiently, accurately and comprehensively by only occupying little memory or storage resources, and meanwhile, the accuracy and the efficiency of data verification are further improved by pre-configuring the identifier of the table to be verified.
Further, in an embodiment, before step 1001, the method may further include:
step 1, determining a first connection mode of a source database according to the type of the source database in a preset configuration file;
step 2, establishing connection with a source database according to a first connection mode;
step 3, determining a second connection mode of the target database according to the type of the target database in the preset configuration file;
and 4, establishing connection with the target database according to the second connection mode.
Further, in one embodiment, the source database is a relational database or a non-relational database, and the target database is the relational database or the non-relational database.
Optionally, a connection manner (i.e., a first connection manner) with the source database may be determined according to a type of the source database configured in the preset configuration file, and a connection manner (i.e., a second connection manner) with the target database may be determined according to a type of the target database configured in the preset configuration file.
In one embodiment, the type of the source database may be a relational database (e.g., oracle, gbase, sql, mysql, etc.) or a non-relational database (e.g., nosql, etc.), and the type of the target database may also be a relational database (e.g., oracle, gbase, sql, mysql, etc.) or a non-relational database (e.g., nosql, etc.), where Nosql may be specifically MongoDB, redis, etc.
In an embodiment, it is assumed that the source database and the target database both use a relational database Oracle, and at this time, the connection modes of the source database and the target database are the same and are both sql plus connection modes.
Specifically, a field in the preset configuration file may be read to determine a connection string of the Oracle database, determine a database type corresponding to the connection string, then determine a connection mode according to the database type, and then call a corresponding connection tool to connect the databases, and SourceQueryTool and TargetQueryTool in the preset configuration file may be used to identify the connection modes of the source database and the target database, respectively.
In one embodiment, a connection tool of a source database and a target database is called according to a connection mode of the source database and the target database to obtain the source database table and the target database table, when the source database and the target database are both Oracle databases, the connection modes of the source database and the target database are both sqlplus connection modes, and the sqlplus connection tool is called to be respectively connected with the target database and the source database to obtain the source database table and the target database table.
The data verification method provided by the invention is based on the standard SQL language, supports the comparison of various database objects, and has a wide comparison range. For example, both the ORACLE database and the Gbase database support the standard SQL language, and query, statistics, analysis, and data verification operations can be performed on the databases as needed through the standard SQL language.
Further, in an embodiment, the method may further specifically include:
and checking the source database table and the target database table according to the data checking type in the data checking task.
Further, in one embodiment, the data check type includes a structured check type and a content check type.
Optionally, in addition to the above-mentioned verification of the data in the source database table and the target database table, the data verification method provided by the present invention may also perform verification on the source database table and the target database table, taking table a and table B as examples:
in one embodiment, the structure of Table A is as follows:
the structure of table B is as follows:
and checking the table A and the table B according to the data checking type (comprising a structured checking type and a content checking type) set in the data checking task.
In an embodiment, the structured check type may specifically include table data amount comparison and table structure comparison, and may also include: table name comparison, table field comparison, table index comparison, table partition comparison, and the like; the content verification type may specifically include field content comparison, keyword comparison, and the like.
For example, the table a structure and the table B structure are extracted, for example, if the table field of table a, such as the name field, is a character type and the maximum length is 10, and the table B name field is compared, as in the above example, table B is also a character type, then the name field in table B is consistent with the type of the name field in table a, but the length of the name field in table B is 8 and is not consistent with table a, at this time, the data verification result displayed on the screen in real time is: the name fields are of consistent type and inconsistent length, the maximum length of the name fields in table a is 10 and the maximum length of the name fields in table B is 8.
The data size of table a and table B may also be compared, where the data size of table a is 4, and the data size of table B is also 4, and at this time, the data verification result displayed on the screen in real time is: the data volumes in table a and table B are consistent.
It should be noted that, comparison of other data verification types in the data verification task is consistent with the comparison process of the table fields, which is not described in detail herein.
The data verification method provided by the invention realizes the comparison of data types of different databases and different forms of data, has high expandability, can reduce the occupancy rate of memory or storage resources, and realizes the data verification of unlimited data magnitude by using the lowest configuration resources.
The data verification system provided by the present invention is described below, and the data verification system described below and the data verification method described above may be referred to correspondingly.
Fig. 2 is a schematic structural diagram of a data verification system provided in the present invention, as shown in fig. 2, including:
a data acquisition module 210, a data cache module 211 and a data verification module 212;
the data obtaining module 210 is configured to segment the source database table according to a data verification task set in a preset configuration file and a data amount of the source database table, where the data verification task includes a maximum data amount of each data comparison, and the maximum segment number is determined according to the data amount of the source database table and the maximum data amount of each data comparison set in the data verification task;
the data caching module 211 is configured to sequentially cache first data corresponding to each segment in the source database table after the segment is segmented and second data in the target database table, where the second data is obtained by migrating the first data to the target database table;
the data checking module 212 is configured to compare the first data with the second data, and obtain a data checking result after data migration, where the first data and the second data in the cache are emptied after data comparison is completed each time.
The data verification system provided by the invention is different from the traditional sampling data verification mode, and can realize efficient, accurate and comprehensive data verification by performing segmented processing on the data to be verified and emptying the data to be verified in the cache after data comparison is completed each time, so that little memory or storage resource is occupied.
Fig. 3 is a schematic physical structure diagram of an electronic device provided in the present invention, and as shown in fig. 3, the electronic device may include: a processor (processor) 310, a communication interface (communication interface) 311, a memory (memory) 312 and a bus (bus) 313, wherein the processor 310, the communication interface 311 and the memory 312 complete communication with each other through the bus 313. The processor 310 may call logic instructions in the memory 312 to perform the following method:
segmenting the source database table according to a data verification task and the data quantity of the source database table, wherein the data verification task comprises the maximum data quantity of each data comparison, and the maximum segment number is determined according to the data quantity of the source database table and the maximum data quantity of each data comparison set in the data verification task;
sequentially caching first data corresponding to each segment in the source database table after segmentation and second data in the target database table, wherein the second data are obtained after the first data are migrated to the target database table;
and comparing the first data with the second data to obtain a data verification result after data migration, and emptying the first data and the second data in the cache after data comparison is completed each time.
In addition, the logic instructions in the memory may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention or a part thereof which substantially contributes to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer power screen (which may be a personal computer, a server, or a network power screen) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like.
Further, the present invention discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the data verification method provided by the above-mentioned method embodiments, for example, comprising:
segmenting the source database table according to a data verification task and the data quantity of the source database table, wherein the data verification task comprises the maximum data quantity of each data comparison, and the maximum segment number is determined according to the data quantity of the source database table and the maximum data quantity of each data comparison set in the data verification task;
sequentially caching first data corresponding to each segment in the source database table after segmentation and second data in the target database table, wherein the second data are obtained after the first data are migrated to the target database table;
and comparing the first data with the second data to obtain a data verification result after data migration, and emptying the first data and the second data in the cache after data comparison is completed each time.
In another aspect, the present invention also provides a non-transitory computer readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the data verification method provided in the foregoing embodiments when executed by a processor, for example, the method includes:
segmenting the source database table according to a data verification task and the data quantity of the source database table, wherein the data verification task comprises the maximum data quantity of each data comparison, and the maximum segment number is determined according to the data quantity of the source database table and the maximum data quantity of each data comparison set in the data verification task;
sequentially caching first data corresponding to each segment in the source database table after the segmentation and second data in the target database table, wherein the second data are obtained after the first data are migrated to the target database table;
and comparing the first data with the second data to obtain a data verification result after data migration, and emptying the first data and the second data in the cache after data comparison is completed each time.
The above-described system embodiments are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment may be implemented by software plus a necessary general hardware platform, and may also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., and includes instructions for causing a computer power supply screen (which may be a personal computer, a server, or a network power supply screen, etc.) to execute the methods described in the embodiments or some portions of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, and not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for data verification, comprising:
segmenting the source database table according to a data verification task and the data quantity of the source database table, wherein the data verification task comprises the maximum data quantity of each data comparison, and the maximum number of segments is determined according to the data quantity of the source database table and the maximum data quantity of each data comparison set in the data verification task;
sequentially caching first data corresponding to each segment in the source database table after segmentation and second data in the target database table, wherein the second data are obtained after the first data are migrated to the target database table;
and comparing the first data with the second data to obtain a data verification result after data migration, wherein the first data and the second data in the cache are emptied after data comparison is completed each time.
2. The data verification method of claim 1, wherein the source database table and the target database table are obtained by:
and acquiring the source database table and the target database table according to the identifier of the source database table and the identifier of the target database table in the preset configuration file.
3. The data verification method according to claim 1, wherein before acquiring the source database table and the target database table according to the preset configuration file, the identifier of the source database table, and the identifier of the target database table, the method further comprises:
determining a first connection mode of the source database according to the type of the source database in the preset configuration file;
establishing connection with the source database according to the first connection mode;
determining a second connection mode of the target database according to the type of the target database in the preset configuration file;
and establishing connection with the target database according to the second connection mode.
4. The method of claim 3, wherein the source database is a relational database or a non-relational database, and the target database is the relational database or the non-relational database.
5. A data verification method according to any one of claims 1 to 4, wherein the method further comprises:
and checking the source database table and the target database table according to the data checking type in the data checking task.
6. The data verification method of claim 5, wherein the data verification types include a structured verification type and a content verification type.
7. A data verification system, comprising: the system comprises a data acquisition module, a data cache module and a data verification module;
the data acquisition module is used for segmenting the source database table according to a data verification task set in a preset configuration file and the data quantity of the source database table, wherein the data verification task comprises the maximum data quantity of each data comparison, and the maximum segment number is determined according to the data quantity of the source database table and the maximum data quantity of each data comparison set in the data verification task;
the data caching module is configured to sequentially cache first data corresponding to each segment in the source database table after the segment is segmented and second data in the target database table, where the second data is obtained by migrating the first data to the target database table;
the data verification module is configured to compare the first data with the second data, obtain a data verification result after data migration, and empty the first data and the second data in the cache after data comparison is completed each time.
8. An electronic device comprising a processor and a memory storing a computer program, wherein the processor implements the data verification method of any one of claims 1 to 6 when executing the computer program.
9. A non-transitory computer-readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the data verification method of any one of claims 1 to 6.
10. A computer program product comprising a computer program, wherein the computer program when executed by a processor implements a data verification method as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210865768.XA CN115422180A (en) | 2022-07-21 | 2022-07-21 | Data verification method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210865768.XA CN115422180A (en) | 2022-07-21 | 2022-07-21 | Data verification method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115422180A true CN115422180A (en) | 2022-12-02 |
Family
ID=84196255
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210865768.XA Pending CN115422180A (en) | 2022-07-21 | 2022-07-21 | Data verification method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115422180A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150175A (en) * | 2023-04-18 | 2023-05-23 | 云账户技术(天津)有限公司 | Heterogeneous data source-oriented data consistency verification method and device |
-
2022
- 2022-07-21 CN CN202210865768.XA patent/CN115422180A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150175A (en) * | 2023-04-18 | 2023-05-23 | 云账户技术(天津)有限公司 | Heterogeneous data source-oriented data consistency verification method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107807982B (en) | Consistency checking method and device for heterogeneous database | |
| CN112199366A (en) | Data table processing method, device and equipment | |
| CN110292775B (en) | Method and device for acquiring difference data | |
| CN112559554A (en) | Query statement optimization method and device | |
| CN111259004B (en) | Method for indexing data in storage engine and related device | |
| EP2862101B1 (en) | Method and a consistency checker for finding data inconsistencies in a data repository | |
| CN110427375B (en) | Method and device for identifying field type | |
| CN110705226A (en) | Spreadsheet creating method and device and computer equipment | |
| CN112307124A (en) | Database synchronization verification method, device, equipment and storage medium | |
| CN115422180A (en) | Data verification method and system | |
| CN111177181A (en) | SQL text auditing method, system, storage medium and device | |
| CN112948429B (en) | Data reporting method, device and equipment | |
| CN113672653A (en) | Method and device for identifying private data in database | |
| CN114090722B (en) | Method and device for automatically completing query content | |
| CN115809228A (en) | Data comparison method and device, storage medium and electronic equipment | |
| CN114356454A (en) | Account checking data processing method, account checking data processing device, account checking data storage medium and program product | |
| CN110851437A (en) | Storage method, device and equipment | |
| CN115858487A (en) | Data migration method and device | |
| CN114564501A (en) | Database data storage and query methods, devices, equipment and medium | |
| CN110941601B (en) | Method and device for determining standard caliber of index, electronic equipment and storage medium | |
| CN115617773A (en) | Data migration method, device and system | |
| CN112540978A (en) | Wide table generation method and device and electronic equipment | |
| CN116401177B (en) | DDL correctness detection method, device and medium | |
| CN113138936B (en) | Data processing method, device, storage medium and processor | |
| CN108304430B (en) | Method for modifying database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |