CN113641653A - Historical data migration method and system based on domestic dream database - Google Patents
Historical data migration method and system based on domestic dream database Download PDFInfo
- Publication number
- CN113641653A CN113641653A CN202110910904.8A CN202110910904A CN113641653A CN 113641653 A CN113641653 A CN 113641653A CN 202110910904 A CN202110910904 A CN 202110910904A CN 113641653 A CN113641653 A CN 113641653A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- mapping
- new system
- migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 95
- 230000005012 migration Effects 0.000 title claims abstract description 95
- 238000000034 method Methods 0.000 title claims abstract description 79
- 238000013507 mapping Methods 0.000 claims abstract description 95
- 238000013506 data mapping Methods 0.000 claims abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 24
- 238000012360 testing method Methods 0.000 claims abstract description 24
- 230000001502 supplementing effect Effects 0.000 claims abstract description 8
- 238000004140 cleaning Methods 0.000 claims description 20
- 230000008569 process Effects 0.000 claims description 7
- 238000012795 verification Methods 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000009795 derivation Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 4
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000013515 script Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 description 7
- 238000011161 development Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000006978 adaptation Effects 0.000 description 4
- 239000000047 product Substances 0.000 description 3
- 238000010200 validation analysis Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000013524 data verification Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011990 functional testing Methods 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 235000006508 Nelumbo nucifera Nutrition 0.000 description 1
- 240000002853 Nelumbo nucifera Species 0.000 description 1
- 235000006510 Nelumbo pentapetala Nutrition 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013502 data validation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 235000019580 granularity Nutrition 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a historical data migration method and a system based on a domestic dream database, wherein A, a data mapping model is established, the data mapping model comprises a service layer mapping module and a data layer mapping module, and the service layer mapping module compares key words and/or characteristic items and establishes corresponding mapping of a new system A' and an old system A on service functions; B. realizing field sets and field mapping relations of the databases of the old system A and the new system A' through a data layer mapping module according to the keywords and/or the characteristic items; C. the data mapping model establishes a corresponding mapping relation from a service layer to a data layer of the old system A and the new system A 'and realizes the migration of the data of the old system A to the new system A'. According to the invention, data migration and data testing are realized through the data mapping model, the data processing strategy and the data supplementing strategy, so that historical data migration of airworthiness approval operation can be completed, the migration efficiency of the historical data is obviously improved, and the migration quality of the historical data is improved.
Description
Technical Field
The invention relates to the field of data migration, in particular to a historical data migration method and system based on a domestic Dameng database.
Background
The localization of software is an important means for protecting national information security, and the database is taken as a basic software reason and is first of all one of the main fields for the promotion of localization. Under the current situation of rapid development of big data environment and information technology, data becomes a core asset of government departments and public institutions, and data storage and migration become a problem concerned by informatization construction of enterprises and government departments. In the updating and updating of the implementation system, the existing data needs to be migrated to a new system, but due to different database structures of the new system and the old system and how to implement data migration rapidly, correctly and completely under the service constraint condition, the integrity, consistency and inheritance of the data are guaranteed, and the like, the industry lacks a support of a data migration theoretical method.
At present, dreams data of domestic databases are widely applied to e-government systems, the dreams database only supports the migration operation of mainstream databases in the field of data migration, and lacks support for small databases such as Lotus domino databases, and migration is only the migration of data tables, and lacks functions such as data completion and mapping of service layers.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a historical data migration method and system based on a domestic dream database.
The purpose of the invention is realized by the following technical scheme:
a historical data migration method based on a domestic Dameng database comprises the following steps:
A. establishing a data mapping model, wherein the data mapping model comprises a service layer mapping module and a data layer mapping module, the service layer mapping module respectively extracts respective corresponding service functions from a new system A ' and an old system A, the service function M of the old system A comprises M1 and M2 … Mn, the service function M ' of the new system A ' comprises M '1 and M '2 … M ', the service layer mapping module compares and establishes corresponding mapping of the new system A ' and the old system A on the service functions through keywords and/or characteristic items, and the corresponding mapping is carried out according to the following logic formula:
equation (1) corresponds to the following four cases of business function mapping:
the first method comprises the following steps: mapping the service functions of the old system A to the new system A' one by one;
and the second method comprises the following steps: a plurality of service functions in the old system A are correspondingly mapped to one service function of the new system A';
and the third is that: a plurality of service functions in the old system A are mapped to a plurality of service functions of the new system A';
and fourthly: the new system A' adds a brand new service function;
B. the field set and the field mapping relation of the database of the old system A and the database of the new system A' are realized through a data layer mapping module according to key words and/or characteristic items, and the mapping logic formula is as follows:
wherein T represents the field set of the old system A, T 'represents the field set of the old system A', C represents the field of the old system A, and C 'represents the field of the old system A';
C. the data mapping model establishes a corresponding mapping relation from a service layer to a data layer of the old system A and the new system A ', the data of the old system A is migrated to the new system A', and the data is supplemented by the supplementing module in a method comprising a default value mode and a correlation value derivation mode. The data migration in step C preferably employs a DMS tool.
The historical data migration method also comprises the following steps:
D. and (3) carrying out data test on the new system A' after the migration by adopting the following method:
d1, data monitoring test: carrying out integrity check, consistency check, total score balance check, record number check and special sample data check on the new system A';
d2, inquiring the data with the same index of the old system A and the new system A' through a data comparison inquiry operation, and performing comparison inquiry verification.
In the historical data migration method, the data migration from an old system A to a new system A' respectively and correspondingly realizes data migration by adopting four modules including a data pipeline, a data processing strategy, a database drive set and an intermediate database, wherein the data pipeline is a channel for data from a source database to a target database; the intermediate library is responsible for temporary storage of intermediate data, the data pipeline integrates data of a plurality of tables, and the intermediate library temporarily stores intermediate result data and processes the data; the database driver set stores database drivers, the database drivers include database connection information, and the database connection information includes host names or IP addresses of the servers, database product names, port numbers, user names and passwords.
The further technical scheme is as follows: the data processing strategy in the historical data migration method comprises a cleaning strategy, a completion strategy and a field mapping dictionary, wherein the cleaning strategy works out a detailed data cleaning strategy to perform traversal analysis on data, find out repeated data and discarded garbage data and process the repeated data and the discarded garbage data; the missing fields are sorted out by the completion strategy, and the missing fields are reasonably supplemented into the data through unified updating of database scripts and multi-table association query; and the field mapping dictionary establishes a field mapping relation between the source database and the target data, and stores the data into the target database after format processing.
The historical data migration method further comprises the following data emergency method:
E. the emergency treatment method is established from three aspects of a service layer, a database and a network platform:
e1, reserving the original service application system when the service of the new system A' is processed, and ensuring that the client configuration environment of the original service application system can be recovered to the former configuration in the shortest time;
e2, database emergency method is to keep backup in the new system A' database, and according to the old system A data set situation different users store backup data;
e3, network platform emergency method is to ensure the smoothness of the whole network link under the condition of large data concentration.
A historical data migration system based on a domestic Dameng database comprises a data mapping module, a data migration module and a data testing module, wherein the data mapping module comprises a service layer mapping module and a data layer mapping module, the data migration module comprises a data assembly line, a data processing strategy, a database drive set and a middle base, and the data processing strategy comprises a cleaning strategy, a completion strategy and a field mapping dictionary.
Compared with the prior art, the invention has the following advantages and beneficial effects:
according to the invention, data migration and data testing are realized through the data mapping model, the data processing strategy and the data completion strategy, so that historical data migration of the airworthiness approval operation management system can be completed, a data migration platform from a document database to a relational dreams database is realized, the migration efficiency of historical data is obviously improved, the migration quality of the historical data is improved, and the migration progress of the historical data of a project is accelerated.
Drawings
FIG. 1 is a schematic diagram illustrating a MMU data migration method model according to a second embodiment of the present invention;
FIG. 2 is a schematic diagram of a data mapping model according to a second embodiment;
FIG. 3 is a schematic diagram of field mapping of a data mapping model according to a second embodiment;
FIG. 4 is a data migration platform architecture according to the second embodiment;
FIG. 5 is a data migration platform operation page according to the second embodiment;
FIG. 6 is a policy editing page of the data migration platform according to the second embodiment;
FIG. 7 is a block diagram of a schematic configuration of a historical data migration system of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the following examples:
example one
A historical data migration method based on a domestic Dameng database comprises the following steps:
A. establishing a data mapping model, wherein the data mapping model comprises a service layer mapping module and a data layer mapping module, the service layer mapping module respectively extracts respective corresponding service functions from a new system A ' and an old system A, the service function M of the old system A comprises M1 and M2 … Mn, the service function M ' of the new system A ' comprises M '1 and M '2 … M ', the service layer mapping module compares and establishes corresponding mapping of the new system A ' and the old system A on the service functions through keywords and/or characteristic items, and the corresponding mapping is carried out according to the following logic formula:
equation (1) corresponds to the following four cases of business function mapping:
the first method comprises the following steps: mapping the service functions of the old system A to the new system A' one by one;
and the second method comprises the following steps: a plurality of service functions in the old system A are correspondingly mapped to one service function of the new system A';
and the third is that: a plurality of service functions in the old system A are mapped to a plurality of service functions of the new system A';
and fourthly: the new system A' adds a brand new service function;
B. the field set and the field mapping relation of the database of the old system A and the database of the new system A' are realized through a data layer mapping module according to key words and/or characteristic items, and the mapping logic formula is as follows:
wherein T represents the field set of the old system A, T 'represents the field set of the old system A', C represents the field of the old system A, and C 'represents the field of the old system A';
C. the data mapping model establishes a corresponding mapping relation from a service layer to a data layer of the old system A and the new system A ', the data of the old system A is migrated to the new system A', and the data is supplemented by the supplementing module in a method comprising a default value mode and a correlation value derivation mode. The data migration in step C preferably employs a DMS tool.
Example two
A historical data migration method and a system based on a domestic Dameng database are disclosed, wherein the method comprises the following steps:
A. establishing a data mapping model, wherein the data mapping model comprises a service layer mapping module and a data layer mapping module, the service layer mapping module respectively extracts respective corresponding service functions from a new system A ' and an old system A, the service function M of the old system A comprises M1 and M2 … Mn, the service function M ' of the new system A ' comprises M '1 and M '2 … M ', and the service layer mapping module compares and establishes corresponding mapping of the new system A ' and the old system A on the service functions through service layer keywords and/or characteristic items and performs corresponding mapping according to the following logic formula:
equation (1) corresponds to the following four cases of business function mapping:
the first method comprises the following steps: mapping the service functions of the old system A to the new system A' one by one;
and the second method comprises the following steps: a plurality of service functions in the old system A are correspondingly mapped to one service function of the new system A';
and the third is that: a plurality of service functions in the old system A are mapped to a plurality of service functions of the new system A';
and fourthly: the new system A' adds a brand new service function;
the data migration of the embodiment adopts an MMU data migration method model, which aims at data migration in a new and old system replacement scenario, and the method solves the problem of how to smoothly migrate historical data of an old system to a new system, evolve and form service data of the new system, and enable the historical data to be used as native data of the new system, and the operation principle of the MMU data migration method model is shown in fig. 1.
Map (Map): and completing the mapping of the new system and the old system in a service layer and a data layer. The design of a new system can be improved in an upgrading way on the basis of an old system, functional points, service logics and the like of a service layer are changed, and further, the data layer has larger difference. In the mapping stage, the contents of data migration mapping model, data completion and the like are researched. Migration (Migrate): the data migration from the source database to the target database is accomplished through the existing dms (datamigration system) tool or self-built tool. At this stage, we complete the conversion of data types, implement cleaning, and complete data. Use (Use): data can be really used, and adaptation in program aspects is often needed, such as adding historical data identification, supporting specific data of historical data and the like, data testing is fully considered in the stage, and users also need to participate in feedback problems if necessary to complete necessary data.
The MMU data migration method is more emphasized in iteration, and data of an old system is gradually migrated to a new system according to actual conditions, service modules, time dimension of historical data and other iterative evolutions. The key to the MMU data migration method is the mapping (Map) of the old and new systems. How to effectively migrate data of an old system to a new system is the first problem to be solved, namely the mapping between the new system and the old system. It is not an easy matter to establish a mapping between old and new systems.
The schematic diagram of the data mapping model of this embodiment is as shown in fig. 2, where a module function mapping from an old system to a new system is established in a service layer, and mappings from a service layer module to a table field of a data layer are respectively established in the dimensions of the old system and the new system, and the mapping from the old table field to the new table field of the data layer is introduced by the mapping of the first two layers.
B. The field set and the field mapping relation of the database of the old system A and the database of the new system A' are realized through a data layer mapping module according to key words and/or characteristic items, and the mapping logic formula is as follows:
wherein T represents the field set of the old system A, T 'represents the field set of the old system A', C represents the field of the old system A, and C 'represents the field of the old system A';
as shown in FIG. 3, the data mapping model can establish a mapping relationship between fields of the old system and fields of the new system and fields of the underlying database. The development of data completion work is not away from establishing perfect field mapping rules, the fields of the new system and the old system are not in one-to-one correspondence, and data may have larger loss. The present embodiment may adopt the following solutions:
1. the record with the missing data is discarded.
2. And filling up missing data. Common modes include default value mode, associated value derivation mode, etc.
3. And predicting the missing data by adopting a model. The method is usually directed at the prediction completion work of continuous numerical values, and the common methods comprise a completion strategy based on a decision table and a missing data filling method based on incomplete data clustering. The three methods are not mutually exclusive, and there may be close relation between different methods in specific implementation algorithms.
C. The data mapping model establishes a corresponding mapping relation from a service layer to a data layer of the old system A and the new system A ', the data of the old system A is migrated to the new system A', and the data is supplemented by the supplementing module in a method comprising a default value mode and a correlation value derivation mode.
Migration (M) in the MMU data method according to this embodiment can be performed in three main methods, including manual logging before system switching (manual logging is possible), migration by a migration tool before system switching, and automatic generation by a new system after system switching.
1. Before the system switching, relevant data in the old system is extracted by a DMS (data migration System) tool, converted into a corresponding data type, and injected into the new system. This method is implemented on the premise that historical data is available and can be mapped into new systems. This is currently the most common and compact method of data migration.
2. Prior to system switchover, the organization's more familiar personnel manually enter relevant, valuable data into the new system. This method is labor and material consuming and has a high error rate. This approach is often used for tables with relatively small amounts of data, such as data dictionaries, underlying data, etc.
3. After the system is switched, required data is generated through a specially developed corollary program or related functions of a new system. This method is carried out on the premise that these data can be generated by other data. The required information, such as table associated foreign key values, is typically generated from data that has been migrated into the new system, other systems already having data.
The architecture of the data migration platform of the present embodiment is shown in fig. 4, the data migration platform replaced by the old system is very different from the tool of the ELT, and the field mapping and data completion policy are emphasized more. The data migration platform comprises four big modules, namely a data pipeline, a data processing strategy, a database drive set and a middle library.
A data pipeline is the passage of data from a source database to a target database. The loading module is responsible for loading data from a source database, the cleaning module cleans the data according to a configurable cleaning strategy, the supplementing module supplements the data according to a supplementing strategy, and the writing module is responsible for type conversion and writing into a specified target database according to the field mapping dictionary.
The data processing strategy is a core brain of historical data migration evolution, and comprises a cleaning strategy, a completion strategy and a field mapping dictionary:
1. and (3) cleaning strategy: and a detailed data cleaning strategy is made, each module performs traversal analysis on the data of the module according to the cleaning strategy, finds out repeated data and discarded garbage data, analyzes reasons, files the data and reasonably eliminates redundant data.
2. And (3) a completion strategy: and sorting out a completion strategy of each module missing field, and combing out the association relation of the foreign keys for the modules which cannot be updated through the single table. According to the completion scheme and the table foreign key association relation, missing fields are reasonably supplemented into the module data through unified updating of database scripts and multi-table association query, the integrity of the data is guaranteed, and the data can be adapted to a new system.
3. A field mapping dictionary: and (4) formulating a field mapping relation of each module of the source database and the target data, and storing the data into the target database after format processing. For unmatched dictionary entries, the matched rules need to be unified, and source data are stored in a target database after being updated uniformly.
The intermediate library is responsible for temporary storage of intermediate data. The data of a plurality of tables are integrated in the data pipeline, the intermediate database can temporarily store the intermediate result data, the processed data of the intermediate database is directly used, the access of a source database is reduced, and the data integration efficiency is improved. The database driver set stores JDBC drivers for the database. When a corresponding database is to be connected, the corresponding database driver needs to be correctly selected, and various connection related information of the database is filled, which generally includes a host name or an IP address of a server, a database product name, a port number, a user name and a password. Only when the relevant connection information is correctly filled, the corresponding database can be connected.
The data migration platform is used for realizing the data processing process, two completely heterogeneous databases can be migrated through a series of strategies and rules appointed by the platform, and the platform completes the data migration work with huge workload in a time-saving and labor-saving manner through an abstraction method.
Data migration from an old system A to a new system A' is realized by adopting four modules including a data pipeline, a data processing strategy, a database driving set and an intermediate database respectively and correspondingly, wherein the data pipeline is a channel for data from a source database to a target database; the intermediate library is responsible for temporary storage of intermediate data, the data pipeline integrates data of a plurality of tables, and the intermediate library temporarily stores intermediate result data and processes the data; the database driver set stores database drivers, the database drivers include database connection information, and the database connection information includes host names or IP addresses of the servers, database product names, port numbers, user names and passwords.
The data processing strategy comprises a cleaning strategy, a completion strategy and a field mapping dictionary, wherein the cleaning strategy is used for making a detailed data cleaning strategy to perform traversal analysis on data, finding out repeated data and discarded junk data and processing the repeated data and the discarded junk data; the missing fields are sorted out by the completion strategy, and the missing fields are reasonably supplemented into the data through unified updating of database scripts and multi-table association query; and the field mapping dictionary establishes a field mapping relation between the source database and the target data, and stores the data into the target database after format processing.
D. And (3) carrying out data test on the new system A' after the migration by adopting the following method:
d1, data monitoring test: carrying out integrity check, consistency check, total score balance check, record number check and special sample data check on the new system A';
d2, inquiring the data with the same index of the old system A and the new system A' through a data comparison inquiry operation, and performing comparison inquiry verification.
The data test is divided into two levels of tests, one is a data monitoring test, namely a data conversion correctness test after the data conversion is finished; the second is a validation test that verifies the correctness of the data conversion by running the actual business using the new system that has passed the functional test. The translation correctness test and the validation test are at the migration (M) and use (U) stages, respectively, of the MMU data migration method.
After the data migration is completed, a conversion correctness test needs to be performed on the migrated data. The verification after the data migration is the check on the migration quality, and meanwhile, the result of the data verification is also an important basis for judging whether the new system can be formally started. The migrated data may be verified in two ways. The quality analysis of the migrated data can be performed by a data quality inspection tool or writing a targeted inspection program. The verification of the data after the migration is different from the quality analysis of the historical data before the migration, and mainly aims to check different indexes. The indexes of the data verification after the migration mainly comprise five aspects: integrity check, whether the referenced foreign key exists; checking consistency, namely whether the values of the data with the same meaning at different positions are consistent; the total score balance check, such as the comparison of the total of the tax return indexes with the total of different granularities of the departments and the households; checking the number of records, and checking whether the number of records corresponding to the new database is consistent with the number of records corresponding to the old database; and checking the special sample data to check whether the same sample is consistent in the old database and the new database. The new and old systems query data comparison and check, query the data with the same index through respective query tools of the new and old systems, and compare the final query result; the data of the new system is restored to the state of the old system in the day before the old system is migrated, then all the services on the old system in the last day are added to the new system, whether the abnormal conditions exist is checked, and the finally generated result is compared with the old system.
Issues to be considered for use (U) in MMU data migration methods include data validation testing, program adaptation, and online contingency plans. In the use of historical data, a validation test is performed to verify the correctness of the data migration by running the actual business using the new system that has passed the functional test. The verification test is a key link, solves the problem of adaptation of historical data on a service level and is related to the success of system switching. In the development and construction of the new system, although the migration and use of the historical data are considered in the early stage of design, the development of the functions of the new system is usually focused, and some detailed problems of the historical data are ignored. This presents new challenges for the old system and for the secondary use of archived data. The application program needs to be adapted on the program according to a verification test result, a support field of common program adaptation is Null, a generation strategy of a main key is modified, historical data identification is added, and the like.
The test may not cover all historical migration data, in which case an emergency protocol must be initiated to resolve. The specific emergency plan is as follows: the emergency plan needs to consider emergency treatment measures from three aspects of a service system, a database and a network platform, and only if the three aspects are simultaneously restored to the state before system switching, the normal operation of the original system service handling can be ensured.
1. The emergency measure of the service application system is mainly to keep the original service application system when the service is processed and ensure that the client configuration environment of the original service application system can be recovered to the former configuration in the shortest time.
2. The database emergency measures are processed by keeping the original data consistent with the original system, namely, backup is kept in a new system database, backup data are stored for different users under the condition of data concentration of the original system, but the user name still needs to adopt the user name of the original database system. Once an emergency occurs, the new system database is immediately switched into the original backup database.
3. The network platform emergency plan is to ensure the smoothness of the whole social security network link under the condition of large data concentration.
E. The emergency treatment method is established from three aspects of a service layer, a database and a network platform:
e1, reserving the original service application system when the service of the new system A' is processed, and ensuring that the client configuration environment of the original service application system can be recovered to the former configuration in the shortest time;
e2, database emergency method is to keep backup in the new system A' database, and according to the old system A data set situation different users store backup data;
e3, network platform emergency method is to ensure the smoothness of the whole network link under the condition of large data concentration.
The conversion of the data types in the embodiment is the most important and critical problem in the data migration platform. The method is realized in two steps: firstly, a corresponding relation of data types from a source database to a destination database is given, namely a data type corresponding to a certain data type in the source database in the destination database is determined; and secondly, converting the data type of the source data into a corresponding JAVA type by the background, and converting the JAVA type into the data type required by the target database. The data types between different databases may not all be in a one-to-one correspondence, and there are generally one-to-many or many-to-one situations. The data migration system for DM therefore uses a method of bridging between source and destination database data types through the data types of JDBC. The data type of the source database corresponds to the specified JDBC data type, and the JDBC type corresponds to the data type of the destination database, so that the mapping of the data types of the source database and the destination database is completed.
As shown in fig. 5, the migrated module name is first filled out to be used as a label for the migrated flow, and then the table name of the target database is filled out. And finally, filling data in the table, and checking columns corresponding to the strategy IDs for certain fields with strategy requirements, so that field special strategy processing can be realized. It should be noted that there is only one pair of "source database and view" and "source column name" by default in the table, and if there is a policy requirement for a certain field, the "add source database column" below can be clicked to implement the policy requirement. The "add" and "delete" buttons at the bottom right are used to add multiple mapping fields. Clicking to begin execution may begin data migration.
The method realizes the one-by-one mapping from the source database to the target database, and simply and efficiently controls the data type and precision of the field of the target database. In addition, through the many-to-many corresponding relation between the fields and the strategies, special data of the source database can be processed in a multi-faced and simplified mode, and service correspondence and development time are greatly saved.
As shown in fig. 6, the selection is performed on the selected row of the policy to be selected (or multiple selections are possible), each policy has different policy ID and policy type, and after the action of the policy is known according to the policy description, multiple policies are selected. In order to ensure the accuracy and the uniformity of migration data, a plurality of migration strategies are established at a business level, and the strategy categories comprise a cleaning strategy, a supplementing strategy and a mapping strategy. The strategies can help us to improve migration quality, so that the migration process becomes more elegant, and the migration result is more accurate. For example, the cleansing policy Strategy02, if the value of this field of the source database is found to be null during the migration process, we start a padding policy to pad the preset value into the target database; and (4) mapping the Strategy Strategy03, and if the Strategy is selected, automatically splicing two or more fields in the source database and assigning the two or more fields to the fields of the target database.
As shown in fig. 7, a historical data migration system based on a domestic dreams database includes a data mapping module, a data migration module, and a data testing module, where the data mapping module includes a service layer mapping module and a data layer mapping module, the data migration module includes four modules, namely a data pipeline, a data processing policy, a database driver set, and an intermediate repository, and the data processing policy includes a cleaning policy, a completion policy, and a field mapping dictionary.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (7)
1. A historical data migration method based on a domestic Dameng database is characterized in that: the method comprises the following steps:
A. establishing a data mapping model, wherein the data mapping model comprises a service layer mapping module and a data layer mapping module, the service layer mapping module respectively extracts respective corresponding service functions from a new system A ' and an old system A, the service function M of the old system A comprises M1 and M2 … Mn, the service function M ' of the new system A ' comprises M '1 and M '2 … M ', the service layer mapping module compares and establishes corresponding mapping of the new system A ' and the old system A on the service functions through keywords and/or characteristic items, and the corresponding mapping is carried out according to the following logic formula:
equation (1) corresponds to the following four cases of business function mapping:
the first method comprises the following steps: mapping the service functions of the old system A to the new system A' one by one;
and the second method comprises the following steps: a plurality of service functions in the old system A are correspondingly mapped to one service function of the new system A';
and the third is that: a plurality of service functions in the old system A are mapped to a plurality of service functions of the new system A';
and fourthly: the new system A' adds a brand new service function;
B. the field set and the field mapping relation of the database of the old system A and the database of the new system A' are realized through a data layer mapping module according to key words and/or characteristic items, and the mapping logic formula is as follows:
wherein T represents the field set of the old system A, T 'represents the field set of the old system A', C represents the field of the old system A, and C 'represents the field of the old system A';
C. the data mapping model establishes a corresponding mapping relation from a service layer to a data layer of the old system A and the new system A ', the data of the old system A is migrated to the new system A', and the data is supplemented by the supplementing module in a method comprising a default value mode and a correlation value derivation mode.
2. The method for migrating historical data based on the domestic dreams database as claimed in claim 1, wherein: the method also comprises the following steps:
D. and (3) carrying out data test on the new system A' after the migration by adopting the following method:
d1, data monitoring test: carrying out integrity check, consistency check, total score balance check, record number check and special sample data check on the new system A';
d2, inquiring the data with the same index of the old system A and the new system A' through a data comparison inquiry operation, and performing comparison inquiry verification.
3. The method for migrating historical data based on the domestic dreams database as claimed in claim 1, wherein: data migration from an old system A to a new system A' is realized by adopting four modules including a data pipeline, a data processing strategy, a database driving set and an intermediate database respectively and correspondingly, wherein the data pipeline is a channel for data from a source database to a target database; the intermediate library is responsible for temporary storage of intermediate data, the data pipeline integrates data of a plurality of tables, and the intermediate library temporarily stores intermediate result data and processes the data; the database driver set stores database drivers, the database drivers include database connection information, and the database connection information includes host names or IP addresses of the servers, database product names, port numbers, user names and passwords.
4. The method for migrating historical data based on the domestic dreams database as claimed in claim 3, wherein: the data processing strategy comprises a cleaning strategy, a completion strategy and a field mapping dictionary, wherein the cleaning strategy is used for making a detailed data cleaning strategy to perform traversal analysis on data, finding out repeated data and discarded junk data and processing the repeated data and the discarded junk data; the missing fields are sorted out by the completion strategy, and the missing fields are reasonably supplemented into the data through unified updating of database scripts and multi-table association query; and the field mapping dictionary establishes a field mapping relation between the source database and the target data, and stores the data into the target database after format processing.
5. The method for migrating historical data based on the domestic dreams database as claimed in claim 1, wherein: the method also comprises the following data emergency method:
E. the emergency treatment method is established from three aspects of a service layer, a database and a network platform:
e1, reserving the original service application system when the service of the new system A' is processed, and ensuring that the client configuration environment of the original service application system can be recovered to the former configuration in the shortest time;
e2, database emergency method is to keep backup in the new system A' database, and according to the old system A data set situation different users store backup data;
e3, network platform emergency method is to ensure the smoothness of the whole network link under the condition of large data concentration.
6. The method for migrating historical data based on the domestic dreams database as claimed in claim 1, wherein: and D, migrating the data in the step C by using a DMS tool.
7. A historical data migration system based on a domestic Dameng database is characterized in that: the data migration module comprises a data pipeline, a data processing strategy, a database drive set and a middle base, and the data processing strategy comprises a cleaning strategy, a completion strategy and a field mapping dictionary.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110910904.8A CN113641653A (en) | 2021-08-10 | 2021-08-10 | Historical data migration method and system based on domestic dream database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110910904.8A CN113641653A (en) | 2021-08-10 | 2021-08-10 | Historical data migration method and system based on domestic dream database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113641653A true CN113641653A (en) | 2021-11-12 |
Family
ID=78420274
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110910904.8A Pending CN113641653A (en) | 2021-08-10 | 2021-08-10 | Historical data migration method and system based on domestic dream database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113641653A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114168606A (en) * | 2021-12-10 | 2022-03-11 | 中国建设银行股份有限公司 | Primary key ID generation method and device, electronic equipment and storage medium |
| CN114936199A (en) * | 2022-07-21 | 2022-08-23 | 平安银行股份有限公司 | Data processing method for system reconstruction, computer equipment and storage medium |
| CN117494045A (en) * | 2023-11-06 | 2024-02-02 | 南京海汇装备科技有限公司 | Data integration intelligent management and control system and method based on data fusion |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090319544A1 (en) * | 2008-06-20 | 2009-12-24 | Griffin James R | Facilitating integration of different computer data systems |
| CN111061817A (en) * | 2019-12-16 | 2020-04-24 | 华云数据有限公司 | Adaptive service construction system, method and computer readable medium |
-
2021
- 2021-08-10 CN CN202110910904.8A patent/CN113641653A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090319544A1 (en) * | 2008-06-20 | 2009-12-24 | Griffin James R | Facilitating integration of different computer data systems |
| CN111061817A (en) * | 2019-12-16 | 2020-04-24 | 华云数据有限公司 | Adaptive service construction system, method and computer readable medium |
Non-Patent Citations (1)
| Title |
|---|
| 陈晓等: "基于国产数据库历史数据迁移的研究与实践", 《民航学报》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114168606A (en) * | 2021-12-10 | 2022-03-11 | 中国建设银行股份有限公司 | Primary key ID generation method and device, electronic equipment and storage medium |
| CN114936199A (en) * | 2022-07-21 | 2022-08-23 | 平安银行股份有限公司 | Data processing method for system reconstruction, computer equipment and storage medium |
| CN117494045A (en) * | 2023-11-06 | 2024-02-02 | 南京海汇装备科技有限公司 | Data integration intelligent management and control system and method based on data fusion |
| CN117494045B (en) * | 2023-11-06 | 2024-04-26 | 南京海汇装备科技有限公司 | Data integration intelligent management and control system and method based on data fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113641653A (en) | Historical data migration method and system based on domestic dream database | |
| US7401085B2 (en) | System and method for controlling the release of updates to a database configuration | |
| CN110209650B (en) | Data normalization and migration method and device, computer equipment and storage medium | |
| US8694472B2 (en) | System and method for rebuilding indices for partitioned databases | |
| US8103704B2 (en) | Method for database consolidation and database separation | |
| CN109977110A (en) | Data cleaning method, device and equipment | |
| US8065323B2 (en) | Offline validation of data in a database system for foreign key constraints | |
| Aboutorabiª et al. | Performance evaluation of SQL and MongoDB databases for big e-commerce data | |
| US8019795B2 (en) | Data warehouse test automation framework | |
| US20050154740A1 (en) | Method and system for a self-healing query access plan | |
| EP2365447A1 (en) | Data integration system | |
| CN107003935A (en) | Optimize database duplicate removal | |
| CN111651431A (en) | Database service oriented management flow standardization method | |
| KR102639897B1 (en) | Data architecture management system | |
| CN113792025A (en) | MMT data migration method based on domestic CPU environment | |
| JPH09212353A (en) | Method and device for supporting reused design | |
| US8478791B2 (en) | Interoperability across heterogeneous taxonomies | |
| CN116719799A (en) | Environment-friendly data management method, device, computer equipment and storage medium | |
| CN114861229A (en) | Hive dynamic desensitization method and system | |
| CN107402920A (en) | The method and apparatus for determining relation database table connection complexity factor | |
| US8321480B2 (en) | Full-function to high availability large database conversion | |
| CN113626423B (en) | Log management method, device and system of business database | |
| KR20190127066A (en) | Real-time DDL generation method for standard dictionary-based metadata change management | |
| EP2365448A1 (en) | Data integration system | |
| Day | Record linkage i: evaluation of commercially available record linkage software for use in NASS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211112 |
|
| RJ01 | Rejection of invention patent application after publication |