CN107103077B - Method and system for determining integrity before and after data migration - Google Patents
Method and system for determining integrity before and after data migration Download PDFInfo
- Publication number
- CN107103077B CN107103077B CN201710277797.3A CN201710277797A CN107103077B CN 107103077 B CN107103077 B CN 107103077B CN 201710277797 A CN201710277797 A CN 201710277797A CN 107103077 B CN107103077 B CN 107103077B
- Authority
- CN
- China
- Prior art keywords
- data
- migrated
- determining
- migration process
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 84
- 230000005012 migration Effects 0.000 title claims abstract description 82
- 238000013508 migration Methods 0.000 title claims abstract description 82
- 238000004140 cleaning Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method and a system for determining integrity before and after data migration, wherein the method comprises the following steps: determining the data characteristics of the data to be migrated in the original database to obtain target characteristics; when the migration process for the data to be migrated is finished, judging whether data with the data characteristics consistent with the target characteristics exist in the target database, if so, judging that the migration process for the data to be migrated is a complete data migration process, and if not, judging that the migration process for the data to be migrated is an incomplete data migration process. The data characteristics of the data to be migrated are determined, so that the target characteristics are obtained, under the condition that the migration process of the data to be migrated is finished, whether data with the data characteristics consistent with the target characteristics exist in the target database or not is judged, and whether the data before and after the migration process keep integrity or not can be determined according to the judgment result.
Description
Technical Field
The invention relates to the technical field of data migration, in particular to a method and a system for determining integrity before and after data migration.
Background
As database systems continue to evolve, older systems, from being enabled to being replaced by newer systems, tend to accumulate a large amount of valuable historical data during their use, many of which are necessary for the new systems to be successfully enabled. In addition, the historical data is also an important basis for decision analysis. Data migration is the process of cleaning, converting, and loading these historical data into a new system. The data migration is mainly suitable for the situation that when one set of old system is switched to another set of new system, or a plurality of sets of old systems are switched to the same set of new system, historical data in the old system needs to be converted into the new system. Data migration at the present stage mainly focuses on the fact that data migration is not completed, and whether the integrity of the data is kept before and after the data migration is unknown, so that a user cannot know the reliability of the data migration process, and user experience is influenced.
From the above, it can be seen that how to determine whether data maintains integrity before and after migration is an issue that still remains to be solved.
Disclosure of Invention
In view of this, the present invention provides a method and a system for determining integrity before and after data migration, which can determine whether data maintains integrity before and after data migration. The specific scheme is as follows:
a method for determining integrity before and after data migration comprises the following steps:
determining the data characteristics of the data to be migrated in the original database to obtain target characteristics;
when the migration process for the data to be migrated is finished, judging whether data with data characteristics consistent with the target characteristics exist in a target database, if so, judging that the migration process for the data to be migrated is a complete data migration process, and if not, judging that the migration process for the data to be migrated is an incomplete data migration process.
Optionally, the process of determining the data characteristics of the data to be migrated in the original database includes:
determining the data type of the data to be migrated;
and determining the data characteristics of the data to be migrated according to the data type of the data to be migrated to obtain the target characteristics.
Optionally, the process of determining the data characteristics of the data to be migrated includes:
and if the data type of the data to be migrated is a primary key table, determining a primary key value of the primary key table as the target feature.
Optionally, the process of determining the data characteristics of the data to be migrated includes:
and if the data type of the data to be migrated is the unique key table, fusing the total row number of the unique key table on the basis of processing the corresponding primary key to obtain the target characteristic.
Optionally, the process of determining the data characteristics of the data to be migrated includes:
and if the data type of the data to be migrated is a common table, extracting row data corresponding to the common table, and determining an MD5 value corresponding to the extracted row data to obtain the target characteristic.
The invention also correspondingly discloses a system for determining the integrity before and after data migration, which comprises:
the characteristic determining module is used for determining the data characteristics of the data to be migrated in the original database to obtain target characteristics;
and the integrity judging module is used for judging whether data with data characteristics consistent with the target characteristics exist in a target database or not when the migration process aiming at the data to be migrated is finished, if so, judging that the migration process aiming at the data to be migrated is a complete data migration process, and if not, judging that the migration process aiming at the data to be migrated is an incomplete data migration process.
Optionally, the feature determining module includes:
the type determining unit is used for determining the data type of the data to be migrated;
and the characteristic determining unit is used for determining the data characteristics of the data to be migrated according to the data type of the data to be migrated to obtain the target characteristics.
Optionally, the feature determining unit is specifically configured to determine, when the data type of the data to be migrated is a primary key table, a primary key value of the primary key table as the target feature.
Optionally, the feature determining unit is specifically configured to, when the data type of the data to be migrated is a unique key table, fuse the total number of rows of the unique key table on the basis of processing the corresponding primary key to obtain the target feature.
Optionally, the feature determining unit is specifically configured to, when the data type of the data to be migrated is a common table, extract line data corresponding to the common table, and determine an MD5 value corresponding to the extracted line data, to obtain the target feature.
In the invention, the integrity determination method before and after data migration comprises the following steps: determining the data characteristics of the data to be migrated in the original database to obtain target characteristics; when the migration process for the data to be migrated is finished, judging whether data with the data characteristics consistent with the target characteristics exist in the target database, if so, judging that the migration process for the data to be migrated is a complete data migration process, and if not, judging that the migration process for the data to be migrated is an incomplete data migration process.
Therefore, the data characteristics of the data to be migrated are determined, so that the target characteristics are obtained, under the condition that the migration process of the data to be migrated is finished, whether the data with the data characteristics consistent with the target characteristics exist in the target database or not is judged, and whether the data before and after the migration process keep integrity or not can be determined according to the judgment result, so that a user can know the reliability of the data migration process, and the user experience is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a method for determining integrity before and after data migration according to an embodiment of the present invention;
fig. 2 is a flowchart of a specific method for determining integrity before and after data migration according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a system for determining integrity before and after data migration according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention discloses a method for determining integrity before and after data migration, which is shown in figure 1 and comprises the following steps:
step S11: determining the data characteristics of the data to be migrated in the original database to obtain target characteristics;
step S12: when the migration process for the data to be migrated is finished, whether data with the data characteristics consistent with the target characteristics exist in the target database is judged, if yes, the step S13 is executed, and if not, the step S14 is executed.
Step S13: and judging that the migration process aiming at the data to be migrated is a complete data migration process.
Step S14: and judging that the migration process aiming at the data to be migrated is an incomplete data migration process.
Therefore, the data characteristics of the data to be migrated are determined, so that the target characteristics are obtained, under the condition that the migration process of the data to be migrated is finished, whether the data with the data characteristics consistent with the target characteristics exist in the target database or not is judged, and whether the data before and after the migration process keep integrity or not can be determined according to the judgment result, so that a user can know the reliability of the data migration process, and the user experience is improved.
Referring to fig. 2, an embodiment of the present invention discloses a specific method for determining integrity before and after data migration, including the following steps:
step S21: determining the data type of the data to be migrated.
It should be noted that, in the database system, the main types of data include a primary key table, a unique key table, and a general table, where the general table refers to a data table having neither a primary key nor a unique key.
Step S22: and determining the data characteristics of the data to be migrated according to the data type of the data to be migrated to obtain the target characteristics.
In a specific embodiment, the determining the data characteristics of the data to be migrated includes: and if the data type of the data to be migrated is the primary key table, determining the primary key value of the primary key table as the target characteristic.
In another specific embodiment, the determining the data characteristics of the data to be migrated includes: and if the data type of the data to be migrated is the unique key table, fusing the total row number of the unique key table on the basis of processing the corresponding primary key to obtain the target characteristic.
In another specific embodiment, the determining the data characteristics of the data to be migrated includes: if the data type of the data to be migrated is the common table, extracting the row data corresponding to the common table, and determining an MD5 value (MD5, namely Message Digest Algorithm5, fifth version of Message Digest Algorithm) corresponding to the extracted row data to obtain the target characteristics. That is, when the data type of the data to be migrated is the normal table, the row data corresponding to the normal table is marked in an MD5 manner, and a corresponding MD5 value is obtained.
As can be seen from the above, the data characteristics of different types of data in a database system may vary. It is understood that if multiple types of data are included in the data to be migrated, the data characteristics corresponding to the data to be migrated will correspondingly include multiple characteristics, for example, if the data types of the data to be migrated include a primary key value and a common table, the data characteristics corresponding to the data to be migrated include corresponding primary key values and MD5 values.
Step S23: when the migration process for the data to be migrated is finished, whether data with the data characteristics consistent with the target characteristics exist in the target database is judged, if yes, the step S24 is executed, and if not, the step S25 is executed.
Step S24: and judging that the migration process aiming at the data to be migrated is a complete data migration process.
Step S25: and judging that the migration process aiming at the data to be migrated is an incomplete data migration process.
Correspondingly, the embodiment of the present invention further discloses a system for determining integrity before and after data migration, as shown in fig. 3, the system includes:
the characteristic determining module 11 is configured to determine a data characteristic of data to be migrated in an original database to obtain a target characteristic;
the integrity judgment module 12 is configured to, when the migration process for the data to be migrated is completed, judge whether data with data characteristics consistent with the target characteristics exists in the target database, if yes, judge that the migration process for the data to be migrated is an integral data migration process, and if not, judge that the migration process for the data to be migrated is an incomplete data migration process.
In this embodiment, the characteristic determining module 11 may specifically include a type determining unit and a characteristic determining unit; wherein the content of the first and second substances,
the type determining unit is used for determining the data type of the data to be migrated;
and the characteristic determining unit is used for determining the data characteristics of the data to be migrated according to the data type of the data to be migrated to obtain the target characteristics.
In a specific embodiment, the characteristic determining unit is specifically configured to determine, when the data type of the data to be migrated is a primary key table, a primary key value of the primary key table as the target characteristic.
In another specific embodiment, the feature determining unit is specifically configured to, when the data type of the data to be migrated is the unique key table, fuse the total number of rows of the unique key table on the basis of processing the corresponding primary key to obtain the target feature.
In another embodiment, the characteristic determining unit is specifically configured to, when the data type of the data to be migrated is an ordinary table, extract row data corresponding to the ordinary table, and determine an MD5 value corresponding to the extracted row data, so as to obtain the target characteristic.
Therefore, the data characteristics of the data to be migrated are determined, so that the target characteristics are obtained, under the condition that the migration process of the data to be migrated is finished, whether the data with the data characteristics consistent with the target characteristics exist in the target database or not is judged, and whether the data before and after the migration process keep integrity or not can be determined according to the judgment result, so that a user can know the reliability of the data migration process, and the user experience is improved.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The integrity determination method and system before and after data migration provided by the present invention are described in detail above, and a specific example is applied in the text to explain the principle and the implementation of the present invention, and the description of the above embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (2)
1. A method for determining integrity before and after data migration is characterized by comprising the following steps:
determining the data characteristics of the data to be migrated in the original database to obtain target characteristics;
when the migration process for the data to be migrated is finished, judging whether data with data characteristics consistent with the target characteristics exist in a target database, if so, judging that the migration process for the data to be migrated is a complete data migration process, and if not, judging that the migration process for the data to be migrated is an incomplete data migration process;
the process of determining the data characteristics of the data to be migrated in the original database includes: determining the data type of the data to be migrated; determining the data characteristics of the data to be migrated according to the data type of the data to be migrated to obtain the target characteristics;
and the data to be migrated comprises multiple types of data, and correspondingly, the data characteristics corresponding to the data to be migrated comprise multiple characteristics, wherein the types of the data to be migrated comprise a primary key table, a unique key table and a common table, and the data characteristics corresponding to the primary key table, the unique key table and the common table are respectively a primary key value, a total row number and an MD5 value corresponding to row data.
2. A system for determining integrity before and after data migration, comprising:
the characteristic determining module is used for determining the data characteristics of the data to be migrated in the original database to obtain target characteristics;
the integrity judgment module is used for judging whether data with data characteristics consistent with the target characteristics exist in a target database or not when the migration process aiming at the data to be migrated is finished, if so, judging that the migration process aiming at the data to be migrated is a complete data migration process, and if not, judging that the migration process aiming at the data to be migrated is an incomplete data migration process;
wherein the feature determination module comprises:
the type determining unit is used for determining the data type of the data to be migrated;
the characteristic determining unit is used for determining the data characteristics of the data to be migrated according to the data type of the data to be migrated to obtain the target characteristics;
and the data to be migrated comprises multiple types of data, and correspondingly, the data characteristics corresponding to the data to be migrated comprise multiple characteristics, wherein the types of the data to be migrated comprise a primary key table, a unique key table and a common table, and the data characteristics corresponding to the primary key table, the unique key table and the common table are respectively a primary key value, a total row number and an MD5 value corresponding to row data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710277797.3A CN107103077B (en) | 2017-04-25 | 2017-04-25 | Method and system for determining integrity before and after data migration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710277797.3A CN107103077B (en) | 2017-04-25 | 2017-04-25 | Method and system for determining integrity before and after data migration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107103077A CN107103077A (en) | 2017-08-29 |
| CN107103077B true CN107103077B (en) | 2021-05-18 |
Family
ID=59656363
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710277797.3A Active CN107103077B (en) | 2017-04-25 | 2017-04-25 | Method and system for determining integrity before and after data migration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107103077B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112835869B (en) * | 2021-01-15 | 2023-05-02 | 中国船舶重工集团公司七五0试验场 | Fuzzy data cleaning method based on reverse data irrigation |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104504122A (en) * | 2014-12-29 | 2015-04-08 | 浪潮(北京)电子信息产业有限公司 | Database migration data verification method and system |
| US20150358323A1 (en) * | 2014-06-05 | 2015-12-10 | Owl Computing Technologies, Inc. | System and method for modular and continuous data assurance |
| CN105487816A (en) * | 2015-11-26 | 2016-04-13 | 布比（北京）网络技术有限公司 | Data transfer method and apparatus |
| CN105989044A (en) * | 2015-02-04 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Database verification method and system |
-
2017
- 2017-04-25 CN CN201710277797.3A patent/CN107103077B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150358323A1 (en) * | 2014-06-05 | 2015-12-10 | Owl Computing Technologies, Inc. | System and method for modular and continuous data assurance |
| CN104504122A (en) * | 2014-12-29 | 2015-04-08 | 浪潮(北京)电子信息产业有限公司 | Database migration data verification method and system |
| CN105989044A (en) * | 2015-02-04 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Database verification method and system |
| CN105487816A (en) * | 2015-11-26 | 2016-04-13 | 布比（北京）网络技术有限公司 | Data transfer method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107103077A (en) | 2017-08-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9612898B2 (en) | Fault analysis apparatus, fault analysis method, and recording medium | |
| JP6919569B2 (en) | Log analysis systems, methods, and recording media | |
| US8205195B2 (en) | Method and system for automatically classifying and installing patches on systems | |
| US20120159625A1 (en) | Malicious code detection and classification system using string comparison and method thereof | |
| CN106202166B (en) | File cleaning method and device and corresponding client | |
| JP6295857B2 (en) | Extraction method, apparatus, and program | |
| JP5521807B2 (en) | Failure cause estimation apparatus, failure cause estimation program, and failure cause estimation method | |
| CN103761483A (en) | Method and device for detecting malicious codes | |
| CN108063725A (en) | A kind of information push method | |
| CN104424356A (en) | Non-transitory computer-readable storage medium for storing acronym-management program, acronym-management device | |
| EP2897401B1 (en) | Method and device for guaranteeing consistency of planning data | |
| US11704302B2 (en) | Method, system and apparatus for processing database updates | |
| CN107103077B (en) | Method and system for determining integrity before and after data migration | |
| CN111061681A (en) | Method and device for partitioning directory based on case insensitivity and storage medium | |
| CN113722134A (en) | Cluster fault processing method, device and equipment and readable storage medium | |
| JP6256606B2 (en) | Analyzer management system | |
| CN107977258A (en) | Computer-readable medium, system and information processing method | |
| WO2016031681A1 (en) | Log analysis device, log analysis system, log analysis method, and computer program | |
| CN107168788A (en) | The dispatching method and device of resource in distributed system | |
| CN109344118B (en) | Snapshot rollback recovery method, system, device and computer readable storage medium | |
| JP5375281B2 (en) | Failure analysis information collection device, failure analysis information collection method, failure analysis information collection program | |
| US20150278006A1 (en) | Information processing apparatus and information processing method | |
| CN109670337B (en) | Detection method and device | |
| JP6543990B2 (en) | Information processing device | |
| CN109784053B (en) | Method and device for generating filter rule, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |