CN113836114B

CN113836114B - Data migration method, system, equipment and storage medium

Info

Publication number: CN113836114B
Application number: CN202111138193.3A
Authority: CN
Inventors: 杨智
Original assignee: Beijing Hujin Xinrong Technology Co ltd
Current assignee: Beijing Hujin Xinrong Technology Co ltd
Priority date: 2021-09-27
Filing date: 2021-09-27
Publication date: 2024-04-26
Anticipated expiration: 2041-09-27
Also published as: CN113836114A

Abstract

The invention provides a data migration method, a system, equipment and a storage medium, wherein the data migration method comprises the following steps: acquiring a data migration configuration table, wherein the migration configuration table comprises data tables to be migrated and corresponding migration order mapping relation tables, and the data tables comprise order information of a plurality of orders; determining a data table to be migrated in the service database according to the migration configuration table; determining an order to be migrated in a data table to be migrated and generating a migration instruction; migrating the order to be migrated from the service database to the first database according to the migration instruction; checking the order information of each order migrated to the data table of the first database with the order information of the corresponding order in the data table of the service database; if an order passes the verification, the order is deleted from the data table of the business database. The data migration method can effectively improve the reading efficiency and ensure the high availability of the system.

Description

Data migration method, system, equipment and storage medium

Technical Field

The present invention relates to the field of data processing, and in particular, to a data migration method, system, device, and storage medium.

Background

The online database A is a database for order processing, the online backup database A_Bak is an online backup database, and the online database A and the online backup database A_Bak are respectively deployed on different servers. The database A and the backup database A_Bak have service access, and the rear end of the database A is hung with four read libraries with services, and the service has the requirement of the delay of reading the rear end of the database A and cannot exceed 5 seconds.

In particular, the online database A relates to hundreds of data tables, stores data of the last 3 months, is generally a solid state disk (SSD disk), and has operations of INSERT (INSERT), DELETE (DELETE), UPDATE (UPDATE) and the like; the online backup database a_bak relates to hundreds of data tables, stores data of the last 2 years, and is generally an SAS (SERIAL ATTACHED SCSI ) disk. When there is an order to process, the order is restored from backup database A_Bak to database A in real time.

On-line business is required, the database is migrated from A to the backup database A_Bak according to Order ID related data before 3 months in the hundreds of data tables, the on-line business cannot be influenced, and the invisible time length of a single Order in the database A and the backup database A_Bak is controlled to be about 1 minute.

Under the conventional migration flow, multiple data tables are processed concurrently, namely, how many data tables are involved and how many threads are started.

Each data table is backed up according to a single Order ID, the backup database A_Bak is written first, and then the backup database A is deleted from the database A, and the process is simple and rough, and has the following risks for online business.

1. Firstly writing the backup database A_Bak, and then ensuring that the same row of data is written and deleted in the process of deleting from the database A;

2. how many data sheets and how many threads are opened, the database reading delay at the rear end of the database A is easy to be caused to exceed 5 seconds, all pressures are transferred to the database A, and finally the database A is likely to be down;

3. the migration progress of a plurality of data tables cannot be controlled, the data of the same Order number (Order ID) cannot be guaranteed, the data are deleted from the database A, the data are written into the backup database A_Bak, and the invisible time length is less than 1 minute;

4. The backup of the data table has sequential requirements, such as Tb1 and Tb2, and it cannot be guaranteed that the data table Tb2 is backed up first and then the data table Tb1 is backed up.

It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the invention and thus may include information that does not form the prior art that is already known to those of ordinary skill in the art.

Disclosure of Invention

Aiming at the problems in the prior art, the invention aims to provide a data migration method, a system, equipment and a storage medium.

The embodiment of the invention provides a data migration method, which comprises the following steps:

s100: acquiring a data migration configuration table, wherein the migration configuration table comprises data tables to be migrated and corresponding migration order mapping relation tables, and the data tables comprise order information of a plurality of orders;

S200: determining a data table to be migrated in the service database according to the migration configuration table;

s300: determining an order to be migrated in a data table to be migrated and generating a migration instruction;

s400: migrating the order to be migrated from the service database to the first database according to the migration instruction;

s500: checking the order information of each order migrated to the data table of the first database with the order information of the corresponding order in the data table of the service database;

If an order passes the verification, S600: deleting the order from the data table of the business database.

According to some examples of the present invention, after verifying the data table migrated to the first database with the data table of the service database, the method further comprises the steps of:

If the order passes the verification, marking the state of the order migrated to the first database in the migration state table as successful;

If an order does not pass the verification, the state of the order migrated to the first database in the migration state table is marked as failure.

According to some examples of the invention, a single thread is employed to migrate orders to be migrated from a business database into a first database.

According to some examples of the invention, after determining the order to be migrated of the data table to be migrated, the method further comprises the steps of:

acquiring performance parameters of service database equipment and/or first database equipment;

Judging whether the performance parameters meet migration conditions or not;

If not, the migration instruction is not executed.

According to some examples of the invention, the service database and the first database belong to the same server cluster, the service database is responsible for service processing, and the first database is an intermediate database. .

According to some examples of the invention, the performance parameter is one or more of a number of transactions performed per second of the server cluster, CPU usage, or a delay between the traffic database and the first database.

According to some examples of the invention, the data migration method further comprises the steps of:

obtaining a migration state table;

And migrating each order in the first database to a second database according to the state of each order in the migration state table.

The embodiment of the invention also provides a data migration system for realizing the data migration method, which comprises a configuration module, an instruction module, a migration module, an inspection module and a deletion module, wherein:

the configuration module is used for acquiring a data migration configuration table, the migration configuration table comprises data tables to be migrated and corresponding migration order mapping relation tables, and the data tables comprise order information of a plurality of orders;

the instruction module is used for determining a data table to be migrated in the service database at this time according to the migration configuration table; and

The method comprises the steps of determining orders to be migrated according to order sequences in a business database and generating migration instructions;

migrating the order to be migrated from the service database to the first database according to the migration instruction;

the checking module is used for checking the order information of each order migrated to the data table of the first database and the order information of the corresponding order in the data table of the service database;

If an order passes the verification, the deleting module is used for deleting the order from the data table of the service database.

The embodiment of the invention also provides electronic equipment, which comprises:

A processor;

a memory having stored therein executable instructions of the processor;

Wherein the processor is configured to perform the steps of the data migration method via execution of the executable instructions.

An embodiment of the present invention also provides a computer-readable storage medium storing a program, wherein the program is executed to implement the steps of the data migration method.

The data migration method can rapidly archive a huge amount of order data in a data table with a sequence in an online transaction processing system (On-Line Transaction Processin, OLTP), and can execute a migration instruction according to system parameters during migration, thereby ensuring that the influence On the service is minimum during migration.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application and, together with the description, further features, objects and advantages of the application, will become apparent from a reading of the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. It is evident that the drawings in the following description are only some embodiments of the present application and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.

FIG. 1 is a flow chart of a data migration method according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating data migration according to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating a data migration system according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of a computer readable storage medium according to an embodiment of the present invention.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.

Fig. 1 is a flowchart of a data migration method according to an embodiment of the present invention, specifically, the data migration method includes the following steps:

s400: migrating the order to be migrated from the business database A to the first database A_Tmp according to the migration instruction;

s500: checking the order information of each order migrated to the data table of the first database with the order information of the corresponding order in the data table of the service database A;

If an order passes the verification, S600: the order is deleted from the data table of the business database a.

The migration configuration table in the Step S100 may be shown in table 1, where, for each table Tb1, tb2 to TbN in the service database a, different orders may be set according to the importance degree of each data table in the actual scene, the important database sets a large order value, the data tables with the order of 1 have no order, the Step S200 places the data table with the order value greater than 1 in one logic according to the migration configuration table, the data table with the order of 1 step=1 in another logic, and may migrate according to the name order of the data table after migrating the data table with the order value greater than 1. Through the setting of the migration configuration table, the data migration method supports migration according to the sequence defined by the data table.

Table 1 is a schematic diagram of a migration configuration table according to an embodiment

Data table name	Cis-position
		Tb1	1
Tb2	1
		Tb3	2
......	......
		Tb50	3
......	......
		Tb65	2
......	......
		Tb105	1
......	......
		TbN	1

In the data migration method of the present invention, the service database a and the first database a_tmp belong to the same server cluster (for example, server cluster 1), the service database is responsible for service processing, and the first database is an intermediate database. I.e. the traffic database a is an OLTP type database, whereas the first database a_tmp may be used as a middleware for data access services, independently of the data management and sharing platform, to facilitate deployment and offloading. The service database A and the first database A_Tmp belong to the same server cluster, and the capability of the service database A for processing the service is greatly affected by directly reading the data memory from the service database A. In general internet application, the reading operation of other servers is concentrated in the first database A_Tmp of the intermediate library, so that the reading efficiency can be effectively improved, and the high availability of the system is ensured.

After determining the data table to be migrated, executing the step S300 to determine the order to be migrated in the data table to be migrated and generating a migration instruction; in this step, the order numbers may be migrated according to the order of the order numbers in the data table, or the migration order of the order numbers may be set.

Then the system executes step S400 to migrate the order to be migrated from the business database A to the first database A_Tmp according to the migration instruction; in one embodiment, a single thread is used to migrate orders to be migrated from business database A to first database A_Tmp. The single thread when migrating to the first database A_Tmp in the data service database A can minimize the influence of the migration process on the database. Of course, it is checked in real time whether the table structures of the data tables in the service database a and the first database a_tmp are identical or not at the time of data migration, and if not, the table structure in the first database a_tmp is corrected with reference to the table structure in the service database a at the time of migration. The steps S300 and S400 implement stripping of the service generated data from the service database into the first database a_tmp as soon as possible.

S500: checking the order information of each order migrated to the data table of the first database with the order information of the corresponding order in the data table of the service database; the check here is a consistency check. If an order passes the verification, that is, each order migrated to the data table of the first database is consistent with the order information of the corresponding order in the data table of the service database, S600 is executed to delete the order from the data table of the service database. In other alternative embodiments, after step S500, the method further includes the steps of:

Table 3 is a schematic diagram of a migration status table in an embodiment, where the migration status table may record migration status of each order in the data table to be migrated, and may further include migration time, order numbers in a new database after migration, and the like, if migration is completed.

Table 3 is a schematic diagram of a migration status table in one embodiment

In some other embodiments, the data migration method further comprises the steps of:

S700: obtaining a migration state table;

s800: and migrating each order in the first database to a second database according to the state of each order in the migration state table. The second database does not belong to the same server cluster as the service database a and the first database a_tmp, e.g. to the server cluster 2. In the process of migrating data from the first database to the second database, the step of data consistency verification can be set to obtain the same data. After the data migration process is finished, the migration state table can be emptied, and meanwhile, a migration report can be sent. In general, there is no delay in providing the service database for service processing to the first database of the intermediate library, and if the delay exceeds a certain threshold, the service pressure of the service database will be increased, the service database will be dragged across, and even the entire server cluster will be down. In some embodiments, after determining the order to be migrated of the data table to be migrated in step S300, the method further comprises the steps of:

S310: acquiring performance parameters of service database equipment and/or first database equipment; the performance parameter is one or more of a number of transactions per second (transaction per second, TPS) performed by the server cluster, CPU usage, or a delay between the traffic database and the first database.

S320: judging whether the performance parameters meet migration conditions or not; for example, the performance parameter in step S310 is the CPU utilization of the server cluster, which is the CPU resource occupied by the running program, and represents the situation of running the program at a certain point in time by the service database device. Step S320 may be to determine whether the CPU utilization is below a set threshold, where a higher CPU utilization indicates that the device has run more programs at this time and vice versa. The usage rate is directly related to the CPU strength of the device itself. If the CPU usage is not lower than the set threshold, S330: not executing the migration instruction; if the CPU usage is below the set threshold, step S400 is performed.

For another example, the performance parameter in step S310 is a delay between the service database and the first database, such as by monitoring the value of the second_ Behind _master parameter outputted by show slave status/G command to determine whether there is a delay between the two. Step S320 may be determining whether the delay between the service database and the first database is less than a set threshold, if the delay is not less than the set threshold, then S330: not executing the migration instruction; if the delay is less than the set threshold, step S400 is performed. In the embodiment of the invention, according to different services, different thresholds can be preset for the performance parameters, and more or less than certain thresholds can be specified as conditions for meeting the migration instruction of the system.

Fig. 2 is a diagram illustrating data migration according to an embodiment of the present invention, in which a data table migrates a first database a_tmp from a service database a, and then from the first database a_tmp to a second database a_bak. For example, the service database a includes a plurality of data tables to be migrated, and when the data tables to be migrated include 100 tens of thousands of orders, whether to execute the migration instruction is determined according to the performance parameter, and the data migration process of the specific data tables to be migrated is shown in table 3.

Table 3 shows a data migration process of a data table to be migrated

The embodiment of the invention also provides a data migration system for implementing the data migration method, and fig. 3 is a schematic structural diagram of the data migration system according to an embodiment of the invention, where the data migration system includes a configuration module M100, an instruction module M200, a migration module M300, an inspection module M400, and a deletion module M500, where:

The configuration module M100 is configured to obtain a data migration configuration table, where the migration configuration table includes data tables to be migrated and a mapping relationship table corresponding to the data tables, and the data tables include order information of a plurality of orders;

The instruction module M200 is used for determining a data table to be migrated in the service database at this time according to the migration configuration table; and

migrating the order to be migrated from the service database to a first database according to the migration instruction M300;

The checking module M400 is configured to check order information of each order migrated to the data table of the first database with order information of a corresponding order in the data table of the service database;

if an order passes the verification, the deleting module M500 is configured to delete the order from the data table of the service database.

The functional implementation manner of each functional module in the data migration system of the embodiment may be implemented by adopting a specific implementation manner of each step in the data migration method. For example, the configuration module M100, the instruction module M200, the migration module M300, the inspection module M400, and the deletion module M500 may implement the functions of the specific embodiments of the steps S100 to S600, respectively, which are not described herein. The data migration system of the invention realizes that the data generated by the service is stripped from the service database A to the first database A_tmp as soon as possible, so that other servers can acquire the data table at any time without affecting the service processing capacity of the service database A,

An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 4. The electronic device 600 shown in fig. 4 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.

As shown in fig. 4, the electronic device 600 is embodied in the form of a general purpose computing device. Components of electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including memory unit 620 and processing unit 610), a display unit 640, etc.

Wherein the storage unit stores program code that is executable by the processing unit 610 such that the processing unit 610 performs the steps according to various exemplary embodiments of the invention described in the above method section of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.

The storage unit 620 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 6201 and/or cache memory unit 6202, and may further include Read Only Memory (ROM) 6203.

The storage unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Bus 630 may be a local bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or using any of a variety of bus architectures.

The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and/or any device (e.g., router, modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 650. Also, electronic device 600 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 600, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage platforms, and the like.

The embodiment of the invention also provides a computer readable storage medium for storing a program, and the program is executed to implement the steps of the data migration method. In some possible embodiments, the aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the method portions of this specification, when the program product is run on the terminal device.

Referring to fig. 5, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

In summary, the present invention provides a data migration method, system, device and storage medium, where the data migration method has the following advantages:

the service requirement can be met, and the online service is not influenced during the data migration;

The single thread is used during migration, so that the influence on the database is reduced to the minimum;

migration may be controlled according to device performance parameters and may be supported in a tabular order.

The foregoing is a further detailed description of the application in connection with the preferred embodiments, and it is not intended that the application be limited to the specific embodiments described. It will be evident to those skilled in the art that the application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the apparatus claims can also be implemented by means of one unit or means in software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.

Claims

1. A method of data migration, comprising the steps of:

S400: migrating orders to be migrated from a business database to a first database according to the migration instruction, wherein the business database and the first database belong to the same server cluster, the business database is responsible for business execution, and the first database is an intermediate database;

If an order passes the verification, S600: deleting the order from the data table of the business database;

the data migration method further comprises the following steps: obtaining a migration state table;

And migrating each order in the first database to a second database according to the state of each order in the migration state table, wherein the second database and the service database do not belong to the same server cluster.

2. The data migration method of claim 1, further comprising the steps of, after verifying the data table migrated to the first database with the data table of the service database:

3. The data migration method of claim 1, wherein the order to be migrated is migrated from the service database to the first database using a single thread.

4. The data migration method of claim 1, further comprising the steps of, after determining an order to be migrated for the data table to be migrated:

Judging whether the performance parameters meet migration conditions or not;

If not, the migration instruction is not executed.

5. The data migration method of claim 4, wherein the performance parameter is one or more of a number of transactions performed per second, a CPU utilization, or a delay between the traffic database and the first database for the server cluster.

6. A data migration system for implementing the data migration method according to any one of claims 1 to 5, comprising a configuration module, an instruction module, a migration module, a check module, and a deletion module, wherein:

Migrating orders to be migrated from a business database to a first database according to the migration instruction, wherein the business database and the first database belong to the same server cluster, the business database is responsible for business execution, and the first database is an intermediate database;

if an order passes the verification, the deleting module is used for deleting the order from the data table of the business database;

The data migration system is further configured to: obtaining a migration state table;

7. An electronic device, comprising:

A processor;

a memory having stored therein executable instructions of the processor;

Wherein the processor is configured to perform the steps of the data migration method of any one of claims 1 to 5 via execution of the executable instructions.

8. A computer readable storage medium storing a program, wherein the program when executed by a processor implements the steps of the data migration method of any one of claims 1 to 5.