CN114490580A

CN114490580A - Configurable data migration method and apparatus executed by automation script

Info

Publication number: CN114490580A
Application number: CN202210098240.4A
Authority: CN
Inventors: 袁静佳; 刘永波; 郑智超
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2022-01-27
Filing date: 2022-01-27
Publication date: 2022-05-13

Abstract

The disclosure provides a configurable data migration method executed by an automation script, which can be applied to the technical field of data migration. The method comprises the following steps: creating an intermediate table according to the data structure of the target table; transferring source data into the intermediate table; converting the source data of the intermediate table according to a service data mapping parameter table to generate a converted intermediate table; performing data check on the converted intermediate table according to the data check configuration table; generating a target table according to the service data mapping parameter table and the converted intermediate table; performing data check on the target table according to the data check configuration table; and performing static inspection on the target table according to a static authentication configuration table. The present disclosure also provides a configurable data migration apparatus, device, storage medium and program product for execution by an automation script.

Description

Configurable data migration method and apparatus executed by automation script

Technical Field

The present disclosure relates to the field of data migration technologies, and more particularly, to a configurable data migration method, apparatus, device, medium, and program product executed by an automation script.

Background

With the rapid development of banking business, part of existing systems of banks may be difficult to meet the business requirements of the banks, and new systems need to be introduced. Data migration of a new system and an old system is particularly important, and due to the problems of large data volume, different data structures and the like of the old system, the data migration needs customized development programs; when data comes from an upstream component, the rule change of the upstream component causes frequent modification of the check script.

At present, most of data migration adopts a manual intervention method, data processing is carried out manually, a checking script is modified manually, the efficiency of data migration is low due to large data volume and complex data processing rules, and workload of later static inspection is increased due to manual intervention and misoperation.

Disclosure of Invention

In view of the foregoing, the present disclosure provides configurable data migration methods, apparatus, devices, media and program products executed by automation scripts that improve the efficiency of data migration.

According to a first aspect of the present disclosure, there is provided a configurable data migration method performed by an automation script, the automation script including a service data mapping parameter table, a data checking configuration table, and a static verification configuration table, comprising:

creating an intermediate table according to the data structure of the target table;

transferring source data into the intermediate table;

converting the source data of the intermediate table according to the service data mapping parameter table to generate a converted intermediate table;

performing data check on the converted intermediate table according to the data check configuration table;

generating a target table according to the service data mapping parameter table and the converted intermediate table;

performing data check on the target table according to the data check configuration table; and

performing static check 0 on the target table according to a static verification configuration table

According to an embodiment of the present disclosure, the service data mapping parameter table includes a field mapping parameter table, the field mapping parameter table includes a first mapping rule and a second mapping rule, the first mapping rule is used for converting source data to generate intermediate data, the second mapping rule is used for converting the intermediate data to generate target data,

the converting the source data of the intermediate table according to the service data mapping parameter table to generate a converted intermediate table includes:

converting the source data of the intermediate table according to the first mapping rule to generate intermediate data;

and generating a converted intermediate table according to the intermediate data.

According to the embodiment of the disclosure, the data checking configuration table includes a first checking rule and a second checking rule corresponding to each field in the database table, the first checking rule is used for checking the data of the converted intermediate table, the second checking rule is used for checking the data of the target table,

the data checking the converted intermediate table according to the data checking configuration table comprises the following steps:

performing data checking on the converted intermediate table according to the first checking rule;

a first data review report is generated.

According to an embodiment of the present disclosure, the generating a target table according to a service data mapping parameter table and the converted intermediate table includes:

performing data conversion on the intermediate data according to the second mapping rule to generate target data;

and generating a target table according to the target data.

According to an embodiment of the present disclosure, the performing data check on the target table according to the data check configuration table includes:

performing data checking on the target table according to the second checking rule;

and generating a second data checking report.

According to the embodiment of the disclosure, the static verification configuration table comprises static verification rules corresponding to each field in the database table,

the static checking of the target table according to the static verification configuration table comprises:

performing static inspection on the target table according to the static verification rule;

a static verification report is generated.

According to the embodiment of the present disclosure, before transferring the source data into the intermediate table, the method further includes:

acquiring a dependent upstream component table;

and determining whether to start executing the data transfer-in operation according to the dependent upstream component table.

According to an embodiment of the present disclosure, further comprising:

if the data quantity of the source data table is larger than a preset threshold value, generating an index according to the source data table in the data conversion process;

the index is deleted after the data conversion is completed.

A second aspect of the present disclosure provides a configurable data migration apparatus executed by an automation script, the automation script including a service data mapping parameter table, a data checking configuration table, and a static verification configuration table, including:

the data transfer module is used for creating an intermediate table according to the data structure of the target table and transferring the source data into the intermediate table;

the first data processing module is used for converting the source data of the intermediate table according to the service data mapping parameter table to generate a converted intermediate table;

the first data checking module is used for checking the data of the converted intermediate table according to the data checking configuration table;

the second data processing module is used for generating a target table according to the service data mapping parameter table and the converted intermediate table;

the second data checking module is used for checking the data of the target table according to the data checking configuration table; and

and the static verification module is used for performing static verification on the target table according to the static verification configuration table.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the configurable data migration method performed by the automation script described above.

A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described configurable data migration method performed by an automation script.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above configurable data migration method performed by an automation script.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be apparent from the following description of embodiments of the disclosure, which proceeds with reference to the accompanying drawings, in which:

FIG. 1 schematically shows a flow chart of data migration steps according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow diagram of a configurable data migration method performed by an automation script, in accordance with an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow diagram for data processing according to an embodiment of the disclosure;

FIG. 4 schematically illustrates a flow diagram for data checking according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a block diagram of a configurable data migration apparatus executed by an automation script, in accordance with an embodiment of the present disclosure; and

FIG. 6 schematically illustrates a block diagram of an electronic device adapted to implement a configurable data migration method performed by an automation script in accordance with an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

With the rapid development of banking business, part of existing systems of banks may be difficult to meet self business requirements, and after a new system needs to be introduced, data migration of a new system and an old system is very important, and the problems of large data volume, different data structures and the like of the old system cause the following problems in data migration: (1) the migration data volume is large, different bank data processing rules are different, and a customized development program is needed, so that the migration efficiency is low; (2) the method comprises the following steps that (1) components with upstream dependence exist, and the rules of the upstream components change to cause frequent modification of a check script; (3) the impact of pipelining on the data migration component.

Aiming at the technical problems, the conventional data migration adopts a manual intervention method, partial data is manually processed, the check script is manually modified, the data size is large, different components correspond to different data processing rules, the data processing rules are complex, the data migration efficiency is low, and the workload of later-stage static inspection is increased due to manual intervention and misoperation.

Based on the above problems, an embodiment of the present disclosure provides a configurable data migration method executed by an automation script, where the automation script includes a service data mapping parameter table, a data checking configuration table, and a static verification configuration table, and the method includes: creating an intermediate table according to the data structure of the target table; transferring source data into the intermediate table; converting the source data of the intermediate table according to the service data mapping parameter table to generate a converted intermediate table; performing data check on the converted intermediate table according to the data check configuration table; generating a target table according to the service data mapping parameter table and the converted intermediate table; performing data checking on the target table according to the data checking configuration table; and performing static inspection on the target table according to the static authentication configuration table.

FIG. 1 schematically shows a flow chart of data migration steps according to an embodiment of the present disclosure.

As shown in fig. 1, the data migration step according to this embodiment mainly includes data transfer-in, data processing, data checking, and static validation. The data of the upstream component is preprocessed according to a preset rule depending on the upstream component table, then an intermediate table is introduced, the intermediate table is established according to a data structure of a target table, and the target table comprises a necessary field of a new system, so that a conversion rule between the intermediate table and the target table is generally unchanged, when the data of an old system is transferred into the new system, only a program for developing the old system to the intermediate table needs to be customized, the problem that the data processing rule strongly depends on the old system is solved, and the development workload is reduced; processing the source data of the intermediate table according to the service data mapping parameter table to generate a converted intermediate table, and checking the intermediate data table to generate a data checking report; and then, carrying out data processing on the converted intermediate table according to the service data mapping parameter table to obtain a target table, carrying out data checking and static verification on the target table, and generating a data checking report and a static verification report.

The configurable data migration method performed by the automation script of the disclosed embodiments will be described in detail below with reference to fig. 2-4.

FIG. 2 schematically illustrates a flow diagram of a configurable data migration method performed by an automation script, according to an embodiment of the disclosure.

As shown in FIG. 2, the configurable data migration method executed by the automation script of the embodiment includes operations S210 to S270, and the data migration method may be executed by the automated scheduling platform.

In operation S210, an intermediate table is created according to the data structure of the target table.

In one example, because the data structures of the old and new systems are different and the data processing rules are different, when data migration is performed, different programs are often customized and developed to process data of the old table. In order to solve the problem that the data processing rule strongly depends on the old system, in the embodiment of the disclosure, the intermediate table is created according to the data structure of the target table, the intermediate table contains necessary fields of the new system, and the structure is unchanged, so that the conversion rule between the intermediate table and the target table is unchanged, and when the data of the old system is transferred to the new system, only a program for developing the old system to the intermediate table needs to be customized, so that the development workload is reduced, and the problem that the data processing rule strongly depends on the old system is solved.

In operation S220, source data is transferred to the intermediate table.

In operation S230, the source data of the intermediate table is converted according to the service data mapping parameter table to generate a converted intermediate table.

In one example, the service data mapping parameter table records the mapping relationship between the old system and the new system, so that the development of a program for converting the intermediate table into the target table is facilitated. And after the old system source data is transferred into the intermediate table, performing data processing operation on the source data according to the mapping relation in the service data mapping parameter table, and replacing the source data with the data processed to generate the converted intermediate table.

In operation S240, the converted intermediate table is subjected to data checking according to the data checking configuration table.

After operation S230, the converted intermediate table needs to be checked to determine the integrity of the data. Specifically, each component combs a database table field check rule list baseline, a development toolkit converts a data check rule into a data check rule configuration table, automatically generates a check rule, forms a fixed analysis program for components with upstream dependence, and automatically generates a data check report in a data check stage of executing data migration after checking that upstream component files are ready.

In operation S250, a target table is generated according to the service data mapping parameter table and the converted intermediate table.

In one example, the service data mapping parameter table further includes a mapping relationship of fields between the converted intermediate table and the target table, and the data in the converted intermediate table is further processed according to the service data mapping parameter table to generate the target table.

In operation S260, data checking is performed on the target table according to the data checking configuration table.

The operation is the same as the data checking process in operation S240, and is not described again.

In operation S270, a static check is performed on the target table according to the static verification configuration table.

After operation S260, a target table is obtained, static inspection needs to be performed on data of the target table, consistency of the data in a data migration process is guaranteed, and it is ensured that the data is not tampered, and specifically, each component combs a "service migration static verification rule list"; the development toolkit converts the static verification rule list into a static verification configuration table and automatically generates static verification rules; and after the data check of the data migration is finished, automatically generating a static verification report. The problem that after each migration is completed, the static verification sql needs to be manually executed, and the static verification report needs to be manually filled in is solved, and the data migration efficiency is further improved.

In the embodiment, by combing the database table field check rule list baseline of each component baseline document, the service migration static verification rule list, the full database table and the field mapping rule list, the data migration efficiency is improved by standardizing the data migration process, an automatic data check program is formed, the data check and static verification rules are automatically generated by a data check and static check report automatic generation tool, the data check report and the static verification report are automatically generated, manual intervention is reduced, misoperation is avoided, and the efficiency of data migration is improved.

FIG. 3 schematically shows a flow diagram of data processing according to an embodiment of the disclosure.

As shown in fig. 3, the data processing step of this embodiment includes operations S310 to S340, and the data processing step may generate an automated script tool to be executed by the automated scheduling platform.

In operation S310, a dependent upstream component table is acquired; a determination is made as to whether to begin performing a data transfer operation based on the dependent upstream component table.

In one example, data transfer operation needs to be performed on data before data processing, since different upstream components cause different data processing rules, data in the upstream components needs to be preprocessed first, the data of the upstream components are processed according to a preset dependent upstream component table, the upstream components provide data according to a template provided by the dependent upstream component table, before data is imported, whether a table in the dependent upstream component table is in place needs to be checked, and the situation that some necessary data is absent after data conversion is completed is avoided. And forming a fixed analysis program during the first round of migration so as to solve the problem that the checking script is frequently modified due to the rule change of the upstream component.

In operation S320, an intermediate table is created according to the data structure of the target table, and the source data is transferred to the intermediate table.

This operation can be referred to as operations S210 and S220 shown in fig. 2, and is not described again.

In operation S330, source data of the intermediate table is converted according to the service data mapping parameter table to generate a converted intermediate table.

According to the embodiment of the disclosure, the service data mapping parameter table comprises a field mapping parameter table, the field mapping parameter table comprises a first mapping rule and a second mapping rule, the first mapping rule is used for converting source data to generate intermediate data, the second mapping rule is used for converting the intermediate data to generate target data, and the source data of the intermediate table is converted to generate intermediate data according to the first mapping rule; and generating a converted intermediate table according to the intermediate data.

In one example, the service data mapping parameter table includes intermediate table information, existing system data table information, and data migration description, where the intermediate table information includes an application component/logic subsystem name, an english name in an entity, an english name in an attribute, a data category, a field length, whether a primary key is available, an effective value range, a default value, and a data dictionary data item number. The existing system data table information comprises a system name, a subsystem name, an English name in a data table, an English name in a database field, the field group, a data type, a value range, whether null exists or not, a default value and the like. The data migration description comprises a data source mode, a mapping relation description, a service contact person and remark information.

In operation S340, a target table is generated according to the service data mapping parameter table and the converted intermediate table.

According to the embodiment of the disclosure, data conversion is performed on the intermediate data according to a second mapping rule to generate target data; and generating a target table according to the target data.

When the data volume of the source data table is large, the data conversion time is long, in order to improve the efficiency of data conversion, indexes are temporarily increased during data processing, and the indexes are deleted after the data processing is finished. Specifically, if the data volume of the source data table is greater than a preset threshold, an index is generated according to the source data table in the data conversion process; the index is deleted after the data conversion is completed. The preset threshold is determined according to the actual data migration working time, for example, if the predicted processing time of a certain table is longer than two hours, the index is added during data processing, and the data conversion efficiency is improved.

After two times of data processing, in order to ensure the integrity and consistency of data, data checking and static verification are required to be performed on the data, and a data checking report and a static verification report are formed.

FIG. 4 schematically shows a flow diagram of data checking according to an embodiment of the disclosure. As shown in fig. 4, the data checking process includes operation S410 and operation S420.

In operation S410, data checking is performed on the converted intermediate table according to the data checking configuration table. In operation S420, data checking is performed on the target table according to the data checking configuration table.

According to the embodiment of the present disclosure, the data checking configuration table includes a first checking rule and a second checking rule corresponding to each field in the database table, the first checking rule is used for checking the data of the converted intermediate table, the second checking rule is used for checking the data of the target table,

performing data check on the converted intermediate table according to the first check rule; a first data review report is generated. Performing data checking on the target table according to the second checking rule; a second data review report is generated.

In one example, a baseline document of a database table field check rule list is combed to generate a data check rule to form an automatic check script, wherein the baseline document comprises check refinement rules and execution conditions, and the check refinement rules comprise physical subsystem numbers, component names, English names in a file table, English names in fields, check rule numbers and rule descriptions. The execution condition comprises the record number, the record passing check number, the failing number, the record passing rate and the failing reason.

The process of data static validation is described below.

The static verification process comprises the steps of performing static verification on the target table according to a static verification configuration table and performing static verification on the target table according to the static verification rule; a static verification report is generated.

In one example, the static verification configuration table includes static verification rules corresponding to each field in the database table, and specifically includes english name in the table, english name in the field, check rule type, check rule number, rule condition, SQL statement, rule design description, and the like. According to the combing 'business migration static verification rule list'; the development toolkit converts the static verification rule list into a static verification configuration table and automatically generates static verification rules; and after the data check of the data migration is finished, automatically generating a static verification report.

Based on the configurable data migration method executed by the automation script, the disclosure also provides a configurable data migration device executed by the automation script. The apparatus will be described in detail below with reference to fig. 5.

FIG. 5 schematically illustrates a block diagram of a configurable data migration apparatus executed by an automation script, according to an embodiment of the disclosure.

As shown in fig. 5, the configurable data migration apparatus 500 executed by the automation script of this embodiment includes a data transfer module 510, a first data processing module 520, a first data checking module 530, a second data processing module 540, a second data checking module 550, and a static verification module 560.

The data transfer module 510 is configured to create an intermediate table according to the data structure of the target table, and transfer the source data into the intermediate table. In an embodiment, the data import module 510 may be configured to perform the operations S210 and S220 described above, which are not described herein again.

The first data processing module 520 is configured to convert the source data of the intermediate table according to the service data mapping parameter table to generate a converted intermediate table. In an embodiment, the first data processing module 520 may be configured to perform the operation S230 described above, which is not described herein again.

The first data checking module 530 is configured to perform data checking on the converted intermediate table according to the data checking configuration table. In an embodiment, the first data checking module 530 may be configured to perform the operation S240 described above, which is not described herein again.

The second data processing module 540 is configured to generate a target table according to the service data mapping parameter table and the converted intermediate table. In an embodiment, the second data processing module 540 may be configured to perform the operation S250 described above, and is not described herein again.

And a second data checking module 550, configured to perform data checking on the target table according to the data checking configuration table. In an embodiment, the second data checking module 550 may be configured to perform the operation S260 described above, which is not described herein again.

And a static verification module 560, configured to perform static verification on the target table according to a static verification configuration table. In an embodiment, the static verification module 560 may be configured to perform the operation S270 described above, which is not described herein again.

According to an embodiment of the present disclosure, any plurality of the data import module 510, the first data processing module 520, the first data checking module 530, the second data processing module 540, the second data checking module 550, and the static verification module 560 may be combined into one module to be implemented, or any one of them may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the data import module 510, the first data processing module 520, the first data checking module 530, the second data processing module 540, the second data checking module 550, and the static verification module 560 may be at least partially implemented as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementations of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the data import module 510, the first data manipulation module 520, the first data checking module 530, the second data manipulation module 540, the second data checking module 550 and the static verification module 560 may be at least partially implemented as a computer program module that, when executed, may perform a corresponding function.

As shown in fig. 6, an electronic apparatus 900 according to an embodiment of the present disclosure includes a processor 901 which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage portion 908 into a Random Access Memory (RAM) 903. Processor 901 may comprise, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 901 may also include on-board memory for caching purposes. The processor 901 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 903, various programs and data necessary for the operation of the electronic apparatus 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. The processor 901 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 900 may also include input/output (I/O) interface 905, input/output (I/O) interface 905 also connected to bus 904, according to an embodiment of the present disclosure. The electronic device 900 may also include one or more of the following components connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output portion 907 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 902 and/or the RAM 903 described above and/or one or more memories other than the ROM 902 and the RAM 903.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated in the flow chart. When the computer program product is run in a computer system, the program code is used for causing the computer system to implement the configurable data migration method executed by the automation script provided by the embodiment of the disclosure.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 901. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of a signal on a network medium, and downloaded and installed through the communication section 909 and/or installed from the removable medium 911. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The computer program, when executed by the processor 901, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A configurable data migration method performed by an automation script, the automation script comprising a business data mapping parameter table, a data checking configuration table, and a static validation configuration table, the method comprising:

transferring source data into the intermediate table;

converting the source data of the intermediate table according to a service data mapping parameter table to generate a converted intermediate table;

performing data checking on the converted intermediate table according to the data checking configuration table;

and performing static inspection on the target table according to a static authentication configuration table.

2. The data migration method of claim 1, wherein the service data mapping parameter table comprises a field mapping parameter table, the field mapping parameter table comprises a first mapping rule and a second mapping rule, the first mapping rule is used for converting source data to generate intermediate data, the second mapping rule is used for converting the intermediate data to generate target data,

3. The data migration method according to claim 1, wherein the data checking configuration table includes a first checking rule and a second checking rule corresponding to each field in the database table, the first checking rule is used for checking the converted intermediate table, the second checking rule is used for checking the target table,

performing data check on the converted intermediate table according to the first check rule;

a first data review report is generated.

4. The data migration method according to claim 2, wherein the generating a target table according to the service data mapping parameter table and the converted intermediate table comprises:

and generating a target table according to the target data.

5. The data migration method according to claim 3, wherein the performing data check on the target table according to the data check configuration table comprises:

a second data review report is generated.

6. The data migration method of claim 5, wherein the static validation configuration table includes static validation rules corresponding to fields in a database table,

a static verification report is generated.

7. The data migration method of claim 1, further comprising, prior to transferring source data to the intermediate table:

acquiring a dependent upstream component table;

8. The data migration method according to claim 1, further comprising:

the index is deleted after the data conversion is completed.

9. A configurable data migration apparatus to be executed by an automation script, the automation script including a business data mapping parameter table, a data checking configuration table, and a static validation configuration table, comprising:

10. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data migration method of any of claims 1-8.

11. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform a method of data migration according to any one of claims 1 to 8.

12. A computer program product comprising a computer program which, when executed by a processor, implements a method of data migration according to any one of claims 1 to 8.