CN115408333A

CN115408333A - Data migration method, device and equipment

Info

Publication number: CN115408333A
Application number: CN202211079112.1A
Authority: CN
Inventors: 熊茜
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2022-09-05
Filing date: 2022-09-05
Publication date: 2022-11-29

Abstract

The application discloses a data migration method, a data migration device and data migration equipment, which can be applied to the field of big data and the field of finance, can be compatible with system characteristics and data characteristics of different migration systems, and can realize data migration between systems in a configurable mode so as to improve the effect of data migration. The method comprises the following steps: firstly, a migration rule is parameterized and configured through a migration system, then, data acquisition processing is carried out on source system data according to the migration rule to obtain a data set to be migrated, then, the data set to be migrated is exported and a text to be migrated meeting a preset text format is generated, and then the text to be migrated can be transmitted to a preset directory in the migration system.

Description

Data migration method, device and equipment

Technical Field

The present application relates to the field of big data technologies, and in particular, to a data migration method, apparatus, and device.

Background

With the rapid development of social economy, each big bank serves as a center of financial services, and more customers transact various financial businesses through the bank, so that more and more data need to be processed by the bank.

At present, when part or all of functions of an old system of a bank are migrated to a new system, existing data of the old system is inevitably migrated to the new system. The original migration mode is to export the dump file of the whole database and migrate the dump file to a new system; or the database content is exported to a text format by using a tool and then migrated to a new system. The data fetching migration method only aims at specific migration requirements of a specific system, cannot make a universal solution for the characteristics of various different service systems, and has the defects of repeated workload, high error rate and the like, so that the data migration effect is poor.

Disclosure of Invention

The embodiment of the application mainly aims to provide a data migration method, a device and equipment, which can be compatible with system characteristics and data characteristics of different migration-in and migration-out systems, and can realize data migration among the systems in a configurable manner, so as to improve the effect of data migration.

In a first aspect, an embodiment of the present application provides a data migration method, where the method includes:

carrying out parameterized configuration on the migration rule through the migration system;

according to the migration rule, data acquisition processing is carried out on source system data to obtain a data set to be migrated;

exporting the data set to be migrated and generating a text to be migrated which meets a preset text format;

and transmitting the text to be migrated to a preset directory in a migration system.

Optionally, the performing data acquisition processing on the source system data according to the migration rule to obtain a data set to be migrated includes:

limiting a migration range of source data, cleaning the source data, and converting the source data to obtain a set of data to be migrated;

and according to a preset configuration rule, extracting data from the set of the data to be migrated, and generating a data set to be migrated.

Optionally, the method further includes:

and recording the intermediate state of the data migration so as to analyze the data migration process by using the recorded result and realize breakpoint retransmission.

Optionally, after the text to be migrated is transmitted to a preset directory in the migration system, the method further includes:

and calling a script program to clear the intermediate file generated in the data migration process.

In a second aspect, an embodiment of the present application further provides a data migration apparatus, where the apparatus includes:

the configuration unit is used for carrying out parameterized configuration on the migration rule through the migration system;

the acquisition unit is used for acquiring and processing data of the source system according to the migration rule to obtain a data set to be migrated;

the generating unit is used for exporting the data set to be migrated and generating a text to be migrated which meets a preset text format;

and the transmission unit is used for transmitting the text to be migrated to a preset directory in the migration system.

Optionally, the collecting unit includes:

the conversion subunit is used for limiting the migration range of the source data, cleaning the source data and converting the source data to obtain a set of data to be migrated;

and the generating subunit is used for extracting data from the set of the data to be migrated according to a preset configuration rule, and generating a data set to be migrated.

Optionally, the apparatus further comprises:

and the recording unit is used for recording the intermediate state of the data migration so as to analyze the data migration process by using the recorded result and realize breakpoint retransmission.

Optionally, the apparatus further comprises:

and the clearing unit is used for calling the script program to clear the intermediate file generated in the data migration process.

An embodiment of the present application further provides a data migration apparatus, including: a processor, a memory, a system bus;

the processor and the memory are connected through the system bus;

the memory is configured to store one or more programs, the one or more programs including instructions, which when executed by the processor, cause the processor to perform any one of the implementations of the data migration method described above.

An embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a terminal device, the terminal device is caused to execute any implementation manner of the data migration method.

According to the data migration method, the data migration device and the data migration equipment, firstly, a migration rule is parameterized and configured through a migration system, then, data acquisition and processing are carried out on source system data according to the migration rule to obtain a data set to be migrated, then, the data set to be migrated is exported and generated into a text to be migrated which meets a preset text format, and the text to be migrated can be transmitted to a preset directory in the migration system, so that system characteristics and data characteristics of different migration systems can be compatible, data migration between systems can be achieved through a configurable mode, and the effect of data migration is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic flowchart of a data migration method according to an embodiment of the present application;

fig. 2 is a schematic block diagram of data migration according to an embodiment of the present disclosure;

fig. 3 is a schematic composition diagram of a data migration apparatus according to an embodiment of the present application.

Detailed Description

At present, when part or all of functions of an old system of a bank are migrated to a new system, existing data of the old system is inevitably migrated to the new system. However, in the existing data migration scheme, the whole migration process is regarded as an atomic operation, and if an error occurs in the migration process, the whole migration process can only be executed again, so that the work is repeated, and the requirement of fine-grained management cannot be met. Moreover, when the migration rule is modified, the original data migration method needs to modify the whole migration program, and the modification is relatively large, so that the modification of the migration rule cannot be compatible in a configuration mode. In addition, the original data migration method may further need to manually execute a plurality of migration steps, and cannot automatically implement the flows of data acquisition, data cleaning, data migration and the like, so that the existing silver-row data migration effect is poor.

In order to solve the above defects, an embodiment of the present application provides a data migration method, where firstly, a migration rule is parameterized and configured by a migration system, then, according to the migration rule, data acquisition and processing are performed on source system data to obtain a data set to be migrated, then, the data set to be migrated is exported and a text to be migrated meeting a preset text format is generated, and then the text to be migrated can be transmitted to a preset directory in the migration system, so that system characteristics and data characteristics of different migration systems can be compatible, data migration between systems can be realized in a configurable manner, and further, an effect of data migration is improved.

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

First embodiment

Referring to fig. 1, a schematic flow chart of a data migration method provided in this embodiment is shown, where the method includes the following steps:

s101: and carrying out parameterized configuration on the migration rule through the migration system.

In this embodiment, in order to improve the effect of data migration, the migration rule is first parameterized and configured by the migration system to execute the subsequent step S102.

Wherein, the migration system refers to a source system in the data migration process. The migration rule refers to an interface which is specified by the migration system for the migration data in the data migration process. The interface comprises a table name, a field length, a field value range, an inter-table constraint, an inter-field constraint and the like. And the migration system migrates the data to the migration system according to the interface requirement. The migrated system refers to a target system in the data migration process. Data migration refers to the process of migrating historical data stored in an old system to a new system in need of synchronization when some or all of the functionality of the old system is replaced by the new system.

Specifically, step S101 is a preparation work of the data migration process, and provides parameters and rule support for the subsequent migration process. The method specifically comprises three parts of content, namely rule configuration, parameter configuration and dynamic configuration.

Wherein, the rule configuration refers to providing mapping for the tables and fields of the migrated system and migrated system, and writing the mapping into the configuration table. If the field a of the table a of the migration system is traced according to the business rule, and the mapping result is the field y of the table X of the migration system 1, and the field n of the table M of the migration system 2, the similar (x.y- > a.a) and (m.n- > a.a) rules are written into the configuration table. And reading the configured rule content by a subsequent migration program, processing the source data, and processing the source data into a data format required by the migration rule.

The parameter configuration refers to configuring the behavior of the migration process, configuring data source information of the migration system, receiving directories of the migration system, generating directories of intermediate files, execution parallelism of the migration program and the like.

The dynamic configuration refers to that after parameters are configured, the parameters can be dynamically adjusted in an operation stage, and if the migration execution efficiency is detected to be low, the migration parallelism can be dynamically adjusted according to the system operation condition.

S102: and according to the migration rule, carrying out data acquisition processing on the source system data to obtain a data set to be migrated.

In this embodiment, after the migration rule is parameterized and configured in step S101, data acquisition processing may be further performed on source system data according to the migration rule to obtain a data set to be migrated, so as to execute subsequent step S103.

An optional implementation manner is that, according to a migration rule, a migration range of source data is defined, the source data is cleaned, and the source data is converted to obtain a set of data to be migrated, and then, according to a preset configuration rule, data is extracted from the set of data to be migrated to generate a data set to be migrated.

Specifically, the source system data may be collected and processed according to the migration rule, and the collection and processing may be subdivided into defining a data set, generating a data set, converting data, processing exception data, and the like.

Defining a data set means that when data is migrated out, the whole database data of a source system may be split into a plurality of migration tasks, and each migration task is responsible for migrating a part of data. Therefore, before migration execution, the data needs to be cut into different data sets through different dimensions. If the migration system requires data migration with modification time as a requirement, data before 2022.01.01 is migrated for the first time, and data after 2022.01.01 is migrated for the second time, the whole database data can be split into different data sets according to the modification time, and subsequent migration is only performed on data in a special data set. By defining the data set, the migration task is more granular, and the data is kept static during migration.

The data set generation refers to simply processing the source table data according to the configured migration rule (namely, the preset configuration rule), and screening out the normal data set and the exceptional data set. The method specifically comprises the following steps: and predefining a normal data storage table and an exceptional data storage table. The normal data table stores table data which is to be migrated and conforms to the migration rule constraint, the exception data table stores data which does not conform to the requirement, and the part of data is migrated to the migration system in a report form; if there is a migration rule (x.y- > a.a) (x.z- > a.b) where the constraints of the X fields y, z of the source system table are varchar (6), number (3), respectively, and the constraints of the fields a, b of the migrated system table a are varchar (5), number (2), respectively, then both the normal data table and the exceptional data table will contain two fields a, b, but the constraints of the normal data table are varchar (5), number (2), and the constraints of the exceptional data table are consistent with the source table system, i.e., varchar (6), number (3). Data acquisition: the source system data is written to the normal data table or the exceptional data table defined above. The writing method can be divided into: fully normal, partially normal, and completely exceptional. After the two tables are defined as above, it is assumed that record 1 ('1234', 99), record 2 ('123456', '999'), and record 3 ('1234', '99') exist in the source table, wherein record 1 conforms to the immigration rule constraint and can be completely written into the normal data table; record 2, both fields do not conform to the constraint, and the data is completely written into the exception data table; record 3 field a is written to the normal data table and field b is written to the exception data table.

Data transformation refers to data transformation of the aforementioned generated data set. If the value of the migration system definition field A (the field is the value of the code value) is 1,2,3, and the code value range defined by the source system is a, b and c, the data of the type needs to be converted during migration, and if the conversion cannot be performed, the data is written into an exception data table. It should be noted that multiple transformations may be involved to generate the final dataset to be migrated.

S103: and exporting the data set to be migrated and generating a text to be migrated which meets a preset text format.

In this embodiment, after the data set to be migrated is obtained in step S102, the data set to be migrated may be further exported, and control information, splitting, compressing, and the like may be added to the data set to be migrated to generate a text to be migrated that meets a preset text format, so as to execute subsequent step S104.

Specifically, the data in the data set to be migrated can be exported and generated into a text format required by the migration system based on a final normal data table and an exceptional data table generated by data acquisition, wherein field separators and file terminators are written into the text in a configurable manner; and the text is split, compressed and the like, if the text is exported in a full table of 10G, the text can be split into 10 small files, and the number is sequentially increased.

S104: and transmitting the text to be migrated to a preset directory in a migration system.

In this embodiment, after the text to be migrated meeting the preset text format is generated in step S103, the text to be migrated may be further transmitted to a preset directory in the migration system, so as to complete migration of data.

An optional implementation manner is that, in order to provide traceability of data, an intermediate state of data migration may be recorded, so that a recorded result is used to analyze a data migration process, thereby implementing breakpoint retransmission.

In an implementation manner, when the data migration process of the above step S101-a104 is executed, the intermediate state of the data migration is recorded, where the state of each step may be subdivided into PREEXECUTE, execute, FAILED, and COMPLETE. If data acquisition starts, the state of PREEXECUTE is written, and if execution is successful, COMPLETE is written. The state record also provides support for breakpoint retransmission, and if a certain step in the migration process fails, when the program is restarted, the program can be re-executed from the failed position by judging the migration state, and the whole process is not required to be re-executed. Moreover, the recording of the state is beneficial to monitoring and analyzing the migration process.

In addition, another optional implementation manner is that after the text to be migrated is transmitted to a preset directory in the migration system, a script program is called to perform cleaning processing on an intermediate file (such as a plurality of intermediate tables defined in the data acquisition stage, the generated migrated text, and the like) generated in the data migration process.

In order to facilitate understanding of the data migration method provided by the present application, the present application also provides an overall modularization schematic diagram of data migration, as shown in fig. 2, during the data migration process, firstly, the data migration process needs to be analyzed, the migrated long link needs to be split into a plurality of sub-modules, functions and constraints of different sub-modules are defined, and boundaries and connection conditions between modules are defined; carrying out parameterization configuration on the emigration rule, wherein the configured content comprises the mapping of the relation between the field to be emigration and the field of the source system, the range of the emigration field, emigration data screening conditions and the like; and (4) breakpoint retransmission, wherein when a certain module has an error in the emigration process, the program can be called again to continue processing from the error position.

As shown in fig. 2, the configuration management module is mainly used for parameter configuration, migration rule configuration, and the like in the whole migration process, and provides support for subsequent modules, and supports dynamic modification, where different migration systems can implement data migration to the same migration system by configuring different options; the data acquisition module is used for defining a data set to be migrated, intercepting, cleaning, converting and the like the data of the source system, and generating a final data set for data migration; the data generation module is used for converting the data in the data set into a format required by the migration system; the data migration module is used for migrating the content generated by the data generation module to a migration system, wherein the migration system specifies a specific migration form, such as FTP transmission; the state recording module is used for recording the execution states of the processing modules.

In this way, by executing the steps S101 to S104, the data migration process is modularized, the common module is abstracted for unified processing, and the system characteristic module is configured, which is beneficial to system reuse. And when data are migrated, operations such as acquisition, cleaning, conversion and the like are carried out on the data in a filter screen mode, wherein different filter screens can be customized according to system characteristics, and different filtering requirements on the data are formed. The method has strong expansibility and can greatly improve the development efficiency. Meanwhile, the migration rule can be mapped and the parameters can be configured. Since migration rules are more text or list format, it is more difficult to maintain. Also, if there are changes, the procedure needs to be modified appropriately. By mapping, the management of the migration rules is more visual and visual, and in addition, the dynamic configuration of the migration rules can avoid frequent modification of programs and improve the development efficiency

In summary, according to the data migration method provided in this embodiment, a migration rule is first configured parametrically by the migration system, then data acquisition is performed on source system data according to the migration rule to obtain a data set to be migrated, then the data set to be migrated is exported and a text to be migrated is generated, where the text to be migrated meets a preset text format, and then the text to be migrated can be transmitted to a preset directory in the migration system, so that system characteristics and data characteristics of different migration and migration systems can be compatible, data migration between systems can be realized in a configurable manner, and an effect of data migration is further improved.

Second embodiment

In this embodiment, a data migration apparatus will be described, and please refer to the above method embodiment for related contents.

Referring to fig. 3, a schematic composition diagram of a data migration apparatus provided in this embodiment specifically includes:

a configuration unit 301, configured to perform parameterized configuration on the migration rule through the migration system;

the acquisition unit 302 is configured to perform data acquisition processing on source system data according to the migration rule to obtain a data set to be migrated;

a generating unit 303, configured to export the to-be-migrated data set and generate a to-be-migrated text meeting a preset text format;

the transmission unit 304 is configured to transmit the text to be migrated to a preset directory in the migration system.

In an implementation manner of this embodiment, the acquisition unit 302 includes:

and the generation subunit is used for extracting data from the set of the data to be migrated according to a preset configuration rule and generating a data set to be migrated.

In an implementation manner of this embodiment, the apparatus further includes:

and the clearing unit is used for calling a script program to clear the intermediate file generated in the data migration process.

In summary, according to the data migration device provided in this embodiment, firstly, a migration rule is configured parametrically by the migration system, then, according to the migration rule, data acquisition and processing are performed on source system data, so as to obtain a data set to be migrated, then, the data set to be migrated is exported and generated into a text to be migrated that meets a preset text format, and then the text to be migrated can be transmitted to a preset directory in the migration system, so that system characteristics and data characteristics of different migration and migration systems can be compatible, data migration between systems can be realized in a configurable manner, and further, an effect of data migration is improved.

Further, an embodiment of the present application further provides a data migration apparatus, including: a processor, a memory, a system bus;

the processor and the memory are connected through the system bus;

the memory is used for storing one or more programs, and the one or more programs comprise instructions which, when executed by the processor, cause the processor to execute any implementation of the data migration method.

Further, an embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a terminal device, the terminal device is caused to execute any implementation method of the data migration method.

As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by software plus a necessary general hardware platform. Based on such understanding, the technical solutions of the present application or portions contributing to the prior art may be embodied in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the method described in the embodiments or some portions of the embodiments of the present application.

It should be noted that, in the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.

It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.

In addition, it should be noted that the data migration method, the device and the equipment provided by the invention can be used in the field of big data and the field of finance. The foregoing is merely an example, and does not limit the application fields of the data migration method, apparatus, and device provided by the present invention.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of data migration, the method comprising:

2. The method according to claim 1, wherein the acquiring data of the source system according to the migration rule to obtain the data set to be migrated includes:

and extracting data from the set of the data to be migrated according to a preset configuration rule to generate a data set to be migrated.

3. The method of claim 1, further comprising:

4. The method as claimed in claim 1, wherein after the text to be migrated is transmitted to a preset directory in the migration system, the method further comprises:

5. A data migration apparatus, comprising:

6. The apparatus of claim 5, wherein the acquisition unit comprises:

7. The apparatus of claim 5, further comprising:

8. The apparatus of claim 5, further comprising:

9. A data emigration device, comprising: a processor, a memory, a system bus;

the processor and the memory are connected through the system bus;

the memory is to store one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform the method of any of claims 1-4.

10. A computer-readable storage medium having stored therein instructions that, when executed on a terminal device, cause the terminal device to perform the method of any one of claims 1-4.