CN114490583A

CN114490583A - Data migration method and device, electronic equipment and storage medium

Info

Publication number: CN114490583A
Application number: CN202210111812.8A
Authority: CN
Inventors: 彭建恩; 陈悌宇; 赵晓龙; 张扬
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2022-01-29
Filing date: 2022-01-29
Publication date: 2022-05-13

Abstract

The invention discloses a data migration method, a data migration device, electronic equipment and a storage medium, and relates to the technical field of data storage. One embodiment of the method comprises: responding to the data migration instruction, and acquiring a corresponding migration type, a source data table identifier and a target data table identifier; inquiring a source data table corresponding to the source data table identification, determining data to be migrated from the source data table based on the migration type, and splitting the data to be migrated into a plurality of first data fragments; writing the first data fragments into an intermediate data table with the same structure as the target data table based on the splitting sequence of the first data fragments; and querying a target data table corresponding to the target data table identifier, splitting the data in the intermediate data table into a plurality of second data fragments, and migrating the data in the intermediate data table to the target data table based on the splitting sequence of the second data fragments. The embodiment can solve the problem that the data migration efficiency is low due to the manual migration mode when the data volume is too large.

Description

Data migration method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data storage technologies, and in particular, to a data migration method and apparatus, an electronic device, and a storage medium.

Background

In the field of financial hosting, due to the fact that business of the industry relates to the characteristics of wide trading market, wide trading variety, wide time span and the like, a large amount of trading data and result data are accumulated every year, and due to business requirements, the data need to be stored for a long time, and therefore in order to achieve the purposes that historical data can be searched and a production system can run stably, mass data are generally required to be migrated and stored. In the prior art, data migration is usually realized through manual migration of a user, but when the data volume is too large, the data migration efficiency is low due to the manual migration mode.

Disclosure of Invention

In view of this, embodiments of the present invention provide a data migration method, an apparatus, an electronic device, and a storage medium, which can solve the problem that when the data size is too large, the data migration efficiency is low due to a manual migration manner.

To achieve the above object, according to an aspect of an embodiment of the present invention, a data migration method is provided.

The data migration method of the embodiment of the invention comprises the following steps: responding to the data migration instruction, and acquiring a corresponding migration type, a source data table identifier and a target data table identifier; querying a source data table corresponding to the source data table identifier, determining data to be migrated from the source data table based on the migration type, and splitting the data to be migrated into a plurality of first data fragments; writing the first data fragment into an intermediate data table with the same structure as a target data table based on the splitting sequence of the first data fragment; and querying a target data table corresponding to the target data table identifier, splitting the data in the intermediate data table into a plurality of second data fragments, and migrating the data in the intermediate data table to the target data table based on the splitting sequence of the second data fragments.

In one embodiment, the splitting the data to be migrated into a plurality of first data fragments includes:

acquiring a first fragmentation threshold value to split the data to be migrated based on the primary key data of the source data table to obtain a plurality of first data fragments;

splitting the data in the intermediate data table into a plurality of second data fragments, including:

and acquiring a second fragmentation threshold value to split the data in the data table to be intermediate based on the primary key data of the source data table to obtain a plurality of second data fragments.

In another embodiment, splitting the data to be migrated based on the primary key data of the source data table to obtain a plurality of first data slices includes:

and splitting the data to be migrated based on the primary key data of the source data table, and adding separators between each piece of data in the split data to obtain a plurality of first data fragments.

judging whether the data volume of the data to be migrated is larger than the first fragmentation threshold value or not;

if yes, splitting the data to be migrated based on the primary key data of the source data table to obtain a plurality of first data fragments; if not, determining the data to be migrated as first fragment data.

In yet another embodiment, querying the target data table to identify a corresponding target data table includes:

judging whether the data volume of the data to be migrated is the same as the data volume of the target data table;

if yes, inquiring a target data table corresponding to the target data table identification; if not, deleting the data in the target data table, re-writing the first data fragment into an intermediate data table with the same structure as the target data table, and judging whether the data volume of the data to be migrated is the same as the data volume re-written into the target data table again.

In yet another embodiment, determining data to be migrated from the source data table based on the migration type includes:

judging whether the migration type is full migration or not;

if yes, determining all data in the source data table as data to be migrated; if not, acquiring a migration condition, and determining the data to be migrated from the source data table based on the migration condition.

In another embodiment, after migrating the data in the intermediate data table to the target data table, the method further includes:

and calling a preset data cleaning strategy to clean the data to be migrated in the source data table based on the migration type.

In another embodiment, the splitting the data to be migrated into a plurality of first data fragments includes:

splitting the data to be migrated into a plurality of first data fragments to generate text files corresponding to the first data fragments;

writing the first data fragment into an intermediate data table with the same structure as a target data table based on the splitting order of the first data fragment, including:

and calling a preset file loading model to write the text file into an intermediate data table with the same structure as a target data table based on the splitting sequence of the first data fragment.

To achieve the above object, according to another aspect of the embodiments of the present invention, there is provided a data migration apparatus.

A data migration apparatus according to an embodiment of the present invention includes: the acquisition unit is used for responding to the data migration instruction and acquiring the corresponding migration type, the source data table identifier and the target data table identifier; the splitting unit is used for inquiring a source data table corresponding to the source data table identifier, determining data to be migrated from the source data table based on the migration type, and splitting the data to be migrated into a plurality of first data fragments; a writing unit, configured to write the first data fragment into an intermediate data table having a structure identical to that of a target data table based on a splitting order of the first data fragment; and the migration unit is used for inquiring the target data table corresponding to the target data table identifier, splitting the data in the intermediate data table into a plurality of second data fragments, and migrating the data in the intermediate data table to the target data table based on the splitting sequence of the second data fragments.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided an electronic apparatus.

An electronic device according to an embodiment of the present invention includes: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data migration method provided by the embodiment of the invention.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer-readable medium.

A computer-readable medium of an embodiment of the present invention stores thereon a computer program, which, when executed by a processor, implements the data migration method provided by an embodiment of the present invention.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer program product.

A computer program product according to an embodiment of the present invention includes a computer program, and when the computer program is executed by a processor, the computer program implements the data migration method according to an embodiment of the present invention.

One embodiment of the above invention has the following advantages or benefits: in the embodiment of the invention, after responding to the data migration instruction, the migration type, the source data table identifier and the target data table identifier can be obtained first, and the data to be migrated can be determined from the source data table corresponding to the source data table identifier through the migration type; in order to improve the system performance, the data to be migrated can be split to obtain a first data fragment, and the data to be migrated is written into the intermediate data table based on the first data fragment; the structure of the intermediate data table is the same as that of the target data table, so that the data in the intermediate data table can be divided into second data fragments, and then the data to be migrated is migrated to the target data table through the second data fragments. In the embodiment of the invention, the data to be migrated can be migrated into the target data table through the intermediate data table, so that the automatic migration of the data is realized, the data migration can be realized without manual operation for mass data, and the efficiency of the data migration is improved. In addition, the intermediate data table is set in the embodiment of the invention, the data to be migrated is firstly written into the intermediate data table, and then the data migration process is divided into two parts based on the intermediate data table, so that under the condition that the data migration is unsuccessful, the whole data migration process does not need to be traced back, the failure reset of the data migration is simplified, and the efficiency of the data migration is improved.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of a system architecture of a data migration system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a main flow of a data migration method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of yet another major flow of a data migration method according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of the primary elements of a data migration apparatus according to an embodiment of the present invention;

FIG. 5 is a diagram of yet another exemplary system architecture to which embodiments of the present invention may be applied;

FIG. 6 is a schematic block diagram of a computer system suitable for use in implementing embodiments of the present invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. According to the technical scheme, the data acquisition, storage, use, processing and the like meet relevant regulations of national laws and regulations.

The embodiment of the invention provides a data migration system which can be used for a scene of migrating mass data.

Specifically, as shown in fig. 1, the data migration system may include a data unloading module, a data loading module, a data migration module, and a data cleaning module. The data unloading module can take out the data to be migrated from the source data table, namely unloading; the data loading module can write the unloaded data into a preset intermediate data table, and the structure of the intermediate data table is the same as that of the target data table; the data migration module can migrate data in the intermediate data table to the target data table to realize data migration; the data cleaning module can clean the data migrated in the source data table after the data migration is completed.

An embodiment of the present invention provides a data migration method, which may be executed by the data migration system shown in fig. 1, and as shown in fig. 2, the method includes:

s201: and responding to the data migration instruction, and acquiring the corresponding migration type, the source data table identification and the target data table identification.

The data migration instruction may be sent by an external device, or may be automatically triggered by a preset program. The data migration instruction comprises a preconfigured migration type, a source data table identifier and a target data table identifier. The migration type may include full migration and incremental migration, where the full migration represents total data migration in the source data table, and the incremental migration represents incremental data migration in the source data table, and if the migration type is incremental migration, a migration condition is further required in the migration type, so that the incremental data is determined based on the migration condition to perform data migration. The data table corresponding to the source data table identification stores data to be migrated, and the data table corresponding to the target data table identification is a data table to which the data is migrated and is used for storing the migrated data.

S202: and inquiring a source data table corresponding to the source data table identification, determining data to be migrated from the source data table based on the migration type, and splitting the data to be migrated into a plurality of first data fragments.

If the migration type is full data, all data in the source data table can be determined as data to be migrated; and if the migration type is incremental data, acquiring the migration condition again, and determining the data to be migrated from the source data table based on the migration condition. For example, the migration condition may be the time of data storage, and the data in the source data table whose storage time meets the migration condition may be determined as the data to be migrated.

Data migration is usually performed after a certain amount of data is accumulated, especially mass data migration, so that the data volume of data to be migrated is large, and in order to avoid system performance degradation caused by processing a large amount of data at one time, in this step, the data to be migrated may be split first to obtain a first data fragment, and then the first data fragment with a small data volume is processed.

Specifically, in this step, splitting the data to be migrated may be specifically performed as: and acquiring a first fragmentation threshold value to split the data to be migrated based on the primary key data of the source data table to obtain a plurality of first data fragments. The first fragmentation threshold is preconfigured and may represent a maximum data size of the first data fragmentation.

Because the data is stored in the source data table based on the table format, when the data to be migrated is split, the data can be split based on the primary key data in the source data table in order to ensure the corresponding relationship between the data, and preset spacers can be added between the data corresponding to different fields.

It should be noted that, sometimes the data amount of the data to be migrated cannot reach the first fragmentation threshold, and at this time, data splitting is not needed, so that it may be determined whether the data amount of the data to be migrated is greater than the first fragmentation threshold first when the data to be migrated is split in this step; if so, indicating that the data volume of the data to be migrated is larger, splitting the data to be migrated based on the primary key data of the source data table to obtain a plurality of first data fragments; if not, the data volume of the data to be migrated is less, and the data to be migrated can be determined as the first fragmented data.

S203: and writing the first data fragment into an intermediate data table with the same structure as the target data table based on the splitting sequence of the first data fragment.

In order to make the data structure of the migrated data the same as that of the source data table, the first data slice may be written into an intermediate data table having the same structure as that of the target data table based on the splitting order of the first data slice. The intermediate data table is preset.

Specifically, in order to facilitate data migration, after the first data fragment is obtained in step S202, a corresponding text file may be generated, so as to facilitate data storage. The file loading model may be embodied as Oracle's SQL Loader.

S204: and querying a target data table corresponding to the target data table identifier, splitting the data in the intermediate data table into a plurality of second data fragments, and migrating the data in the intermediate data table to the target data table based on the splitting sequence of the second data fragments.

In order to improve the performance of the data migration system, in this step, the intermediate data table is re-split, and the specific implementation may be: and acquiring a second fragmentation threshold value to split data in the data table to be intermediate based on the primary key data of the source data table to obtain a plurality of second data fragments. The second fragmentation threshold is pre-configured and represents a maximum value of the data size of the second data fragmentation, and may be 20000 pieces, for example. After the second data fragment is obtained, the data in the intermediate data table may be migrated to the target data table based on the splitting order of the second data fragment, and the index relationship of the target data table is established in the migration process.

In order to ensure the accuracy of data migration, before step S201, data verification may be performed, specifically, the following steps may be performed: judging whether the data volume of the data to be migrated is the same as the data volume of the target data table or not; if yes, inquiring a target data table corresponding to the target data table identification; if not, deleting the data in the target data table, re-writing the first data fragment into an intermediate data table with the same structure as the target data table, and judging whether the data volume of the data to be migrated is the same as the data volume re-written into the target data table again until the judgment result is the same.

And verifying whether the data loaded in the intermediate data table is accurate or not by comparing whether the data volume of the data to be migrated is the same as that of the target data table or not. If so, the step can be executed; if not, backtracking is needed, namely data in the target data table is deleted, the first data is stored to an intermediate data table with the same structure as the target data table in a fragmentation mode, and whether the data volume of the data to be migrated is the same as the data volume of the target data table is judged. Similarly, in the step, after the data in the intermediate data table is migrated to the target data table, data verification can be performed again, that is, whether the data amount in the target data table is the same as the data amount in the intermediate data table is judged; if yes, storing the target data table; if not, backtracking the data migration step, namely deleting the data in the target data table, migrating the data in the intermediate data table to the target data table based on the splitting sequence of the second data fragment again, and judging whether the data quantity in the target data table is the same as the data quantity in the intermediate data table again until the judgment result is the same.

In this way, in the embodiment of the present invention, the data migration process is divided into two parts based on the intermediate data table, the data to be migrated is written into the intermediate data table, the loading of the data to be migrated is realized, and the data in the intermediate data table is subsequently migrated into the target data table, so that the data migration is realized. Under the condition that data loading and data migration are unsuccessful, the backtracking of all data migration processes is not needed, the failure backtracking of data migration is simplified, and the efficiency of data migration is improved.

In the embodiment of the present invention, after step S204 is completed, in some cases, it is further necessary to clean up the migration to be determined in the source data table. Specifically, a data cleaning policy may be configured based on the migration type, and after step S204 is completed, a preset data cleaning policy may be called to clean the data to be migrated in the source data table based on the migration type.

In the embodiment of the invention, the data to be migrated can be migrated into the target data table through the intermediate data table, so that the automatic migration of the data is realized, the data migration can be realized without manual operation for mass data, and the efficiency of the data migration is improved. In addition, the intermediate data table is set in the embodiment of the invention, the data to be migrated is firstly written into the intermediate data table, and then the data migration process is divided into two parts based on the intermediate data table, so that under the condition that the data migration is unsuccessful, the whole data migration process does not need to be traced back, the failure reset of the data migration is simplified, and the efficiency of the data migration is improved.

It should be noted that, in order to facilitate execution of data migration, in the embodiment of the present invention, parameters required for data migration, such as a source data table identifier, a target data identifier, a first fragmentation threshold, a second fragmentation threshold, an intermediate data table identifier, a migration type, a migration condition delimiter, and the like, may be configured in advance. After the configuration is completed, corresponding tasks, that is, a data unloading task, a data loading task, a data migration task, and a data cleaning task, may be generated for each module based on the system architecture shown in fig. 1, so that the tasks may be sequentially executed in order after responding to a data migration instruction to implement data migration.

In the following, a data migration method in an embodiment of the present invention is specifically described with reference to the above production task, as shown in fig. 3, the method includes:

s301: and responding to the data migration instruction, triggering a data unloading task and executing.

The data unloading task execution process may be specifically the data processing process of step S201 and step S202, and is not described herein again. After the data unloading task is executed, the first data segment may be stored to a preset address, so as to facilitate reading when the data loading task is executed.

S302: and triggering a data loading task and executing.

The data loading task execution process may specifically be the data processing process in step S203, and is not described herein again. After the data loading task is executed, the intermediate data table may be stored to a preset address, so as to facilitate reading when the data migration task is executed.

S303: and triggering a data migration task and executing.

The data migration task execution process may specifically be the data processing process in step S204, and is not described herein again.

S304: and triggering and executing a data cleaning task.

Specifically, a data cleaning policy may be configured based on the migration type, and after step S303 is completed, a preset data cleaning policy may be called to clean the data to be migrated in the source data table based on the migration type.

It should be noted that, before the steps S302 to S304 are executed, data verification may be executed first to ensure the accuracy of data migration. In the embodiment of the invention, allowable running time can be configured for each task to control the time of data migration, and the execution of the data loading task, the data migration task and the data cleaning task needs to depend on the processing result of the previous task, so that whether the previous task is executed or not needs to be judged when triggering; if yes, task triggering can be carried out; if not, the triggering of the memorable task after the execution of the previous task is finished can be waited.

In order to solve the problems in the prior art, an embodiment of the present invention provides a data migration apparatus 400, as shown in fig. 4, where the apparatus 400 includes:

an obtaining unit 401, configured to obtain, in response to the data migration instruction, a corresponding migration type, a source data table identifier, and a target data table identifier;

a splitting unit 402, configured to query a source data table corresponding to the source data table identifier, determine data to be migrated from the source data table based on the migration type, and split the data to be migrated into multiple first data segments;

a writing unit 403, configured to write the first data fragment into an intermediate data table having the same structure as a target data table based on a splitting order of the first data fragment;

a migration unit 404, configured to query a target data table corresponding to the target data table identifier, split the data in the intermediate data table into a plurality of second data fragments, and migrate the data in the intermediate data table to the target data table based on a splitting order of the second data fragments.

It should be understood that the manner of implementing the embodiment of the present invention is the same as the manner of implementing the embodiment shown in fig. 2, and the description thereof is omitted.

In an implementation manner of the embodiment of the present invention, the splitting unit 402 is specifically configured to:

the migration unit 404 is specifically configured to:

In another implementation manner of the embodiment of the present invention, the splitting unit 402 is specifically configured to:

In another implementation manner of the embodiment of the present invention, the migration unit 404 is specifically configured to:

judging whether the data volume of the data to be migrated is the same as the data volume of the target data table or not;

judging whether the migration type is full migration or not;

In another implementation manner of the embodiment of the present invention, the apparatus 400 further includes:

and the cleaning unit is used for calling a preset data cleaning strategy so as to clean the data to be migrated in the source data table based on the migration type.

In another implementation manner of the embodiment of the present invention, the splitting unit 402 is specifically configured to split the data to be migrated into a plurality of first data fragments, and generate a text file corresponding to each first data fragment;

the writing unit 403 is specifically configured to invoke a preset file loading model, so as to write the text file into an intermediate data table having the same structure as the target data table based on the splitting order of the first data fragment.

It should be understood that the embodiment of the present invention is implemented in the same manner as the embodiment shown in fig. 2 or fig. 3, and is not repeated herein.

According to an embodiment of the present invention, an electronic device and a readable storage medium are also provided.

The electronic device of the embodiment of the invention comprises: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the processor, and the instructions are executed by the at least one processor to cause the at least one processor to execute the data migration method provided by the embodiment of the invention.

Fig. 5 illustrates an exemplary system architecture 500 to which the data migration method or the data migration apparatus of the embodiments of the present invention may be applied.

As shown in fig. 5, the system architecture 500 may include

terminal devices

501, 502, 503, a network 504, and a server 505. The network 504 serves to provide a medium for communication links between the

terminal devices

501, 502, 503 and the server 505. Network 504 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. Various client applications may be installed on the

terminal devices

501, 502, 503.

The

terminal devices

501, 502, 503 may be, but are not limited to, smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 505 may be a server providing various services, and the server may analyze and process data such as a received product information query request, and feed back a processing result (for example, product information — just an example) to the terminal device.

It should be noted that the data migration method provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the data migration apparatus is generally disposed in the server 505.

It should be understood that the number of terminal devices, networks, and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 6, a block diagram of a computer system 600 suitable for use in implementing embodiments of the present invention is shown. The computer system illustrated in FIG. 6 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the invention.

As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 601.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a unit, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present invention may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes an acquisition unit, a splitting unit, a writing unit, and a migration unit. Where the names of these units do not in some cases constitute a limitation on the unit itself, for example, a migration unit may also be described as a "unit of data migration functionality".

As another aspect, the present invention also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to perform the data migration method provided by the present invention.

As another aspect, the present invention further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program implements the data migration method provided by the embodiment of the present invention.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of data migration, comprising:

responding to the data migration instruction, and acquiring a corresponding migration type, a source data table identifier and a target data table identifier;

querying a source data table corresponding to the source data table identifier, determining data to be migrated from the source data table based on the migration type, and splitting the data to be migrated into a plurality of first data fragments;

writing the first data fragment into an intermediate data table with the same structure as a target data table based on the splitting sequence of the first data fragment;

and querying a target data table corresponding to the target data table identifier, splitting the data in the intermediate data table into a plurality of second data fragments, and migrating the data in the intermediate data table to the target data table based on the splitting sequence of the second data fragments.

2. The method of claim 1, wherein splitting the data to be migrated into a plurality of first data segments comprises:

3. The method of claim 2, wherein splitting the data to be migrated based on the primary key data of the source data table results in a plurality of first data slices, comprising:

4. The method of claim 2, wherein splitting the data to be migrated based on the primary key data of the source data table results in a plurality of first data slices, comprising:

5. The method of claim 1, wherein querying the target data table to identify a corresponding target data table comprises:

6. The method of claim 1, wherein determining data to be migrated from the source data table based on the migration type comprises:

judging whether the migration type is full migration or not;

if yes, determining all data in the source data table as data to be migrated; if not, acquiring a migration condition to determine the data to be migrated from the source data table based on the migration condition.

7. The method of claim 1, wherein after migrating the data in the intermediate data table to the target data table, further comprising:

8. The method of claim 1, wherein splitting the data to be migrated into a plurality of first data segments comprises:

9. A data migration apparatus, comprising:

the acquisition unit is used for responding to the data migration instruction and acquiring the corresponding migration type, the source data table identifier and the target data table identifier;

the splitting unit is used for inquiring a source data table corresponding to the source data table identifier, determining data to be migrated from the source data table based on the migration type, and splitting the data to be migrated into a plurality of first data fragments;

a writing unit, configured to write the first data fragment into an intermediate data table having a structure identical to that of a target data table based on a splitting order of the first data fragment;

and the migration unit is used for inquiring the target data table corresponding to the target data table identifier, splitting the data in the intermediate data table into a plurality of second data fragments, and migrating the data in the intermediate data table to the target data table based on the splitting sequence of the second data fragments.

10. The apparatus according to claim 9, wherein the splitting unit is specifically configured to:

the migration unit is specifically configured to:

11. The apparatus according to claim 10, wherein the splitting unit is specifically configured to:

12. The apparatus according to claim 10, wherein the splitting unit is specifically configured to:

13. An electronic device, comprising:

one or more processors;

a storage device to store one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.

14. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.

15. A computer program product comprising a computer program, characterized in that the program, when executed by a processor, implements the method of any one of claims 1-8.