CN113836114A - Data migration method, system, device and storage medium - Google Patents
Data migration method, system, device and storage medium Download PDFInfo
- Publication number
- CN113836114A CN113836114A CN202111138193.3A CN202111138193A CN113836114A CN 113836114 A CN113836114 A CN 113836114A CN 202111138193 A CN202111138193 A CN 202111138193A CN 113836114 A CN113836114 A CN 113836114A
- Authority
- CN
- China
- Prior art keywords
- database
- order
- migration
- data
- migrated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013508 migration Methods 0.000 title claims abstract description 141
- 230000005012 migration Effects 0.000 title claims abstract description 141
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012795 verification Methods 0.000 claims abstract description 14
- 238000013507 mapping Methods 0.000 claims abstract description 7
- 238000012545 processing Methods 0.000 claims description 16
- 238000012217 deletion Methods 0.000 claims description 4
- 230000037430 deletion Effects 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data migration method, a system, equipment and a storage medium, wherein the data migration method comprises the following steps: acquiring a data migration configuration table, wherein the migration configuration table comprises each data table to be migrated and a mapping relation table of a migration order corresponding to the data table, and the data table comprises order information of a plurality of orders; determining a data table to be migrated from the business database at this time according to the migration configuration table; determining an order to be migrated in a data table to be migrated and generating a migration instruction; according to the migration instruction, migrating the order to be migrated from the service database to the first database; the order information of each order transferred to the data table of the first database is verified with the order information of the corresponding order in the data table of the business database; and if the order passes the verification, deleting the order from the data table of the service database. The data migration method can effectively improve the reading efficiency and ensure the high availability of the system.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data migration method, system, device, and storage medium.
Background
The online database A is a database for order processing, the online backup database A _ Bak is a database for online backup, and the online database A and the online backup database A _ Bak are respectively deployed on different servers. The database A and the backup database A _ Bak have service access, four read databases with services are hung at the rear end of the database A, and the service has a requirement on the read database delay at the rear end of the database A, which cannot exceed 5 seconds.
Specifically, the online database a relates to hundreds of data tables, stores data of the last 3 months, and is generally a solid state disk (SSD disk) with operations such as INSERT (INSERT), DELETE (DELETE), UPDATE (UPDATE), and the like; the online backup database a _ Bak refers to hundreds of data tables, and stores data of the last 2 years, and is generally a Serial Attached SCSI (SAS) disk. When an order is to be processed, the order is restored from the backup database a _ Bak to the database a in real time.
The business requires that the Order ID related data before 3 months in the hundreds of data tables are periodically migrated from the database A to the backup database A _ Bak, the online business cannot be influenced, and the invisible duration of a single Order in the database A and the backup database A _ Bak is controlled to be about 1 minute.
Under the conventional migration flow, a plurality of data tables are processed concurrently, namely, how many data tables are involved and how many threads are opened.
And each data table is backed up according to a single Order ID, and is written into a backup database A _ Bak firstly and then deleted from the database A, so that the process is simple and rough, and the following risks are caused to the online service.
1. Writing a backup database A _ Bak, and ensuring that the same row of data is written and deleted in the process of deleting the backup database A;
2. the number of data tables and the number of threads are opened, so that the reading delay at the rear end of the database A is easily caused to exceed 5 seconds, all the pressure is transferred to the database A, and finally the database A can be in a downtime state;
3. migration progress of a plurality of data tables cannot be controlled, data of the same Order number (Order ID) cannot be guaranteed, the data are deleted in the database A, meanwhile, the data are written in the backup database A _ Bak, and invisible duration is less than 1 minute;
4. the backup of the data table has sequential requirements, such as Tb1 and Tb2, which cannot guarantee that the data table Tb2 is backed up first and then the data table Tb1 is backed up.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present invention and therefore may include information that does not constitute prior art known to a person of ordinary skill in the art.
Disclosure of Invention
The present invention provides a data migration method, system, device and storage medium, which can effectively improve the reading efficiency and ensure the high availability of the system.
The embodiment of the invention provides a data migration method, which comprises the following steps:
s100: acquiring a data migration configuration table, wherein the migration configuration table comprises each data table to be migrated and a mapping relation table of a migration order corresponding to the data table, and the data table comprises order information of a plurality of orders;
s200: determining a data table to be migrated from the business database at this time according to the migration configuration table;
s300: determining an order to be migrated in a data table to be migrated and generating a migration instruction;
s400: according to the migration instruction, migrating the order to be migrated from the service database to the first database;
s500: the order information of each order transferred to the data table of the first database is verified with the order information of the corresponding order in the data table of the business database;
if an order passes verification, S600: and deleting the order from the data table of the service database.
According to some examples of the present invention, after checking the data table migrated to the first database with the data table of the service database, the method further includes the following steps:
if an order passes the verification, marking the state of the order in the migration state table migrated to the first database as successful;
and if the order does not pass the verification, marking the state of the order migrated to the first database in the migration state table as failure.
According to some examples of the invention, the order to be migrated is migrated from the business database to the first database using a single thread.
According to some examples of the invention, after determining the order to be migrated of the data table to be migrated, further comprising the steps of:
acquiring performance parameters of the service database equipment and/or the first database equipment;
judging whether the performance parameters meet migration conditions;
if not, the migration instruction is not executed.
According to some examples of the invention, the service database and the first database belong to the same server cluster, the service database is responsible for service processing, and the first database is an intermediate database. .
According to some examples of the invention, the performance parameter is one or more of a number of transactions executed per second for the cluster of servers, a CPU usage rate, or a latency between the traffic database and the first database.
According to some examples of this invention, the data migration method further comprises the steps of:
acquiring a migration state table;
and migrating each order in the first database to a second database according to the state of each order in the migration state table.
The embodiment of the invention also provides a data migration system, which is used for realizing the data migration method and comprises a configuration module, an instruction module, a migration module, a checking module and a deletion module, wherein:
the configuration module is used for acquiring a data migration configuration table, the migration configuration table comprises each data table to be migrated and a mapping relation table of a migration order corresponding to the data table, and the data table comprises order information of a plurality of orders;
the instruction module is used for determining a data table to be migrated from the business database at this time according to the migration configuration table; and
the order generation module is used for determining orders to be migrated according to the order sequence in the service database and generating migration instructions;
according to the migration instruction, migrating the order to be migrated from the service database to the first database;
the checking module is used for verifying the order information of each order transferred to the data table of the first database with the order information of the corresponding order in the data table of the service database;
and if the order passes the verification, the deleting module is used for deleting the order from the data table of the service database.
An embodiment of the present invention further provides an electronic device, including:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the data migration method via execution of the executable instructions.
An embodiment of the present invention also provides a computer-readable storage medium storing a program, wherein the program is configured to implement the steps of the data migration method when executed.
The data migration method can rapidly archive the large amount of order data in the data table with the sequence in an On-Line Transaction processing system (OLTP), and simultaneously can execute the migration instruction according to the system parameters during migration, thereby ensuring the minimum influence On the service during migration.
Drawings
Other features, objects, and advantages of the invention will be apparent from the following detailed description of non-limiting embodiments, which proceeds with reference to the accompanying drawings and which is incorporated in and constitutes a part of this specification, illustrating embodiments consistent with the present application and together with the description serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a flow chart of a data migration method according to an embodiment of the present invention;
FIG. 2 is an exemplary diagram of data migration according to one embodiment of the invention;
FIG. 3 is a block diagram of a data migration system according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the invention;
fig. 5 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
Fig. 1 is a flowchart of a data migration method according to an embodiment of the present invention, specifically, the data migration method includes the following steps:
s100: acquiring a data migration configuration table, wherein the migration configuration table comprises each data table to be migrated and a mapping relation table of a migration order corresponding to the data table, and the data table comprises order information of a plurality of orders;
s200: determining a data table to be migrated from the business database at this time according to the migration configuration table;
s300: determining an order to be migrated in a data table to be migrated and generating a migration instruction;
s400: according to the migration instruction, migrating the order to be migrated from the service database A to a first database A _ Tmp;
s500: the order information of each order transferred to the data table of the first database is verified with the order information of the corresponding order in the data table of the business database A;
if an order passes verification, S600: and deleting the order from the data table of the service database A.
The migration configuration table in the Step S100 may be as shown in table 1, where different ranks may be set according to importance degrees of each data table in an actual scene for each table Tb1, Tb2 through TbN in the service database a, a large rank value is set for an important database, and a data table with rank 1 has no precedence order, the Step S200 places a data table with rank value greater than 1 in one logic according to the migration configuration table, places a data table with rank value 1 in another logic, and may migrate according to a data table name order after the data table with large rank value is migrated. Through the setting of the migration configuration table, the data migration method of the invention supports the migration according to the sequence defined by the data table.
Table 1 is a schematic diagram of a migration configuration table according to an embodiment
| Data table names | Cis position |
| Tb1 | 1 |
| Tb2 | 1 |
| Tb3 | 2 |
| ...... | ...... |
| Tb50 | 3 |
| ...... | ...... |
| Tb65 | 2 |
| ...... | ...... |
| Tb105 | 1 |
| ...... | ...... |
| TbN | 1 |
In the data migration method of the present invention, the service database a and the first database a _ Tmp belong to the same server cluster (for example, the server cluster 1), the service database is responsible for service processing, and the first database is an intermediate database. Namely, the business database a is an OLTP type database, and the first database a _ Tmp may be used as a middleware for performing data access service, independent of a data management and sharing platform, so as to facilitate deployment and uninstallation. The service database A and the first database A _ Tmp belong to the same server cluster, and the capacity of the service database A for processing the service is greatly influenced by directly reading the data memory from the service database A. In general internet application, the reading operation of other servers is concentrated in the first database a _ Tmp of the intermediate library, so that the reading efficiency can be effectively improved, and the high availability of the system is ensured.
After the data table to be migrated is determined, the step S300 of determining the order to be migrated in the data table to be migrated is executed and a migration instruction is generated; in this step, migration may be performed according to the order of each order number in the data table, or the migration order of each order number may be set.
Then the system executes the step S400 to transfer the order to be transferred from the service database A to the first database A _ Tmp according to the transfer instruction; in one embodiment, a single thread is used to migrate the order to be migrated from the business database a to the first database a _ Tmp. The single thread is used when the data service database A is migrated to the first database A _ Tmp, so that the influence of the migration process on the database can be reduced to the minimum. Of course, when data migration is performed, whether the table structures of the data tables in the service database a and the first database a _ Tmp are the same or not is checked in real time, and if they are different, the table structure in the first database a _ Tmp is corrected by referring to the table structure in the service database a during migration. The steps S300 and S400 enable stripping of the business generated data from the business database into the first database a _ tmp as quickly as possible.
S500: the order information of each order transferred to the data table of the first database is verified with the order information of the corresponding order in the data table of the business database; the check here is a consistency check. If an order passes the verification, that is, each order migrated to the data table of the first database is consistent with the order information of the corresponding order in the data table of the service database, then S600 is executed to delete the order from the data table of the service database. In some other alternative embodiments, after the step S500, the following steps are further included:
if an order passes the verification, marking the state of the order in the migration state table migrated to the first database as successful;
and if the order does not pass the verification, marking the state of the order migrated to the first database in the migration state table as failure.
Table 3 is a schematic diagram of the migration status table in an embodiment, and the migration status table may record the migration status of each order in the data table to be migrated, and may further include migration time, an order number in a new database after migration, and the like, for example, when the migration is completed.
Table 3 is a diagram illustrating a transition state table in an embodiment
In some other embodiments, the data migration method further comprises the steps of:
s700: acquiring a migration state table;
s800: and migrating each order in the first database to a second database according to the state of each order in the migration state table. The second database does not belong to the same server cluster as the service database a and the first database a _ Tmp, e.g. to server cluster 2. In the process of migrating data from the first database to the second database, a step of data consistency check can also be set to ensure that the migrated data are the same. After the data migration process is finished, the migration state table may be cleared, and at the same time, a migration report may be sent. Generally, the first database from the service database providing service processing to the intermediate database cannot have a delay, and if the delay exceeds a certain threshold, the service pressure of the service database is increased, the service database is dragged across, and even the whole server cluster is down. In some embodiments, after determining the order to be migrated of the data table to be migrated in step S300, the following steps are further included:
s310: acquiring performance parameters of the service database equipment and/or the first database equipment; the performance parameter is one or more of a number of Transactions Per Second (TPS) executed by the server cluster, a CPU usage rate, or a latency between the service database and the first database.
S320: judging whether the performance parameters meet migration conditions; for example, the performance parameter in step S310 is a CPU utilization of the server cluster, where the CPU utilization is a CPU resource occupied by the running program, and represents a situation that the service database device runs the program at a certain time point. The step S320 may be to determine whether the CPU utilization is lower than a set threshold, where a higher CPU utilization indicates that the device runs more programs in this time, and conversely runs less programs. The utilization rate is directly related to the CPU intensity of the device. If the CPU utilization is not lower than the set threshold, S330: not executing the migration instruction; if the CPU usage is lower than the set threshold, the step S400 is executed.
For another example, the performance parameter in step S310 is a delay between the service database and the first database, for example, whether there is a delay between the occurrence of the delay and the delay is determined by monitoring the value of the second _ while _ Master parameter output by the show slow status \ G command. Step S320 may be to determine whether the delay between the service database and the first database is smaller than a set threshold, and if the delay is not smaller than the set threshold, S330: not executing the migration instruction; if the delay is less than the set threshold, step S400 is executed. In the embodiment of the present invention, different thresholds may be preset for the performance parameters according to different services, and a condition that some thresholds are greater than or less than some thresholds are defined as a condition for satisfying the system execution migration instruction.
Fig. 2 is an exemplary diagram of data migration according to an embodiment of the present invention, in which a data table first migrates a first database a _ Tmp from a service database a, and then migrates from the first database a _ Tmp to a second database a _ Bak. For example, the service database a includes a plurality of data tables to be migrated, and when a data table to be migrated includes 100 ten thousand orders, whether to execute the migration instruction is determined according to the performance parameter, and the data migration process of the specific data table to be migrated is shown in table 3.
Table 3 shows a data migration procedure for a to-be-migrated data table
An embodiment of the present invention further provides a data migration system, configured to implement the data migration method, and fig. 3 is a schematic structural diagram of the data migration system according to an embodiment of the present invention, where the data migration system includes a configuration module M100, an instruction module M200, a migration module M300, a check module M400, and a deletion module M500, where:
the configuration module M100 is configured to obtain a data migration configuration table, where the migration configuration table includes each data table to be migrated and a mapping relationship table of a migration order corresponding to the data table, and the data table includes order information of multiple orders;
the instruction module M200 is configured to determine a data table to be migrated from the service database at this time according to the migration configuration table; and
the order generation module is used for determining orders to be migrated according to the order sequence in the service database and generating migration instructions;
according to the migration instruction M300, migrating the order to be migrated from the service database to the first database;
the checking module M400 is configured to verify order information of each order migrated to the data table of the first database with order information of a corresponding order in the data table of the service database;
if an order passes the verification, the deleting module M500 is configured to delete the order from the data table of the service database.
The functional implementation manner of each functional module in the data migration system of the embodiment may be implemented by using the specific implementation manner of each step in the data migration method. For example, the configuration module M100, the instruction module M200, the migration module M300, the check module M400, and the deletion module M500 may respectively implement their functions by adopting the specific implementation manners of the steps S100 to S600, which are not described herein again. The data migration system of the invention realizes that the data generated by the service is stripped from the service database A to the first database A _ tmp as fast as possible, so that other servers can acquire the part of data table at any time without influencing the service processing capacity of the service database A,
an electronic device 600 according to this embodiment of the invention is described below with reference to fig. 4. The electronic device 600 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 4, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including the memory unit 620 and the processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code which can be executed by the processing unit 610 such that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention as described in the above-mentioned method section of the present specification. For example, processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
Embodiments of the present invention further provide a computer-readable storage medium for storing a program, where the program is executed to implement the steps of the data migration method. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention described in the method part above of this description when said program product is run on the terminal device.
Referring to fig. 5, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In summary, the present invention provides a data migration method, system, device and storage medium, where the data migration method has the following advantages:
the method can meet the service requirement, and does not influence the on-line service during the data migration;
a single thread is used during migration, and the influence on the database is reduced to the minimum;
migration may be controlled according to device performance parameters and supported in tabular order.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the apparatus claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Claims (10)
1. A method of data migration, comprising the steps of:
s100: acquiring a data migration configuration table, wherein the migration configuration table comprises each data table to be migrated and a mapping relation table of a migration order corresponding to the data table, and the data table comprises order information of a plurality of orders;
s200: determining a data table to be migrated from the business database at this time according to the migration configuration table;
s300: determining an order to be migrated in a data table to be migrated and generating a migration instruction;
s400: according to the migration instruction, migrating the order to be migrated from the service database to the first database;
s500: the order information of each order transferred to the data table of the first database is verified with the order information of the corresponding order in the data table of the business database;
if an order passes verification, S600: and deleting the order from the data table of the service database.
2. The data migration method according to claim 1, further comprising the following steps after checking the data table migrated to the first database with the data table of the service database:
if an order passes the verification, marking the state of the order in the migration state table migrated to the first database as successful;
and if the order does not pass the verification, marking the state of the order migrated to the first database in the migration state table as failure.
3. The data migration method according to claim 1, wherein the order to be migrated is migrated from the business database to the first database using a single thread.
4. The data migration method according to claim 1, wherein after determining the order to be migrated for the data table to be migrated, further comprising the steps of:
acquiring performance parameters of the service database equipment and/or the first database equipment;
judging whether the performance parameters meet migration conditions;
if not, the migration instruction is not executed.
5. The data migration method according to claim 4, wherein the service database and the first database belong to a same server cluster, the service database is responsible for service processing, and the first database is an intermediate database.
6. The data migration method of claim 5, wherein the performance parameter is one or more of a number of transactions executed per second for the server cluster, a CPU usage rate, or a latency between the traffic database and the first database.
7. The data migration method according to claim 1, further comprising the steps of: acquiring a migration state table;
and migrating each order in the first database to a second database according to the state of each order in the migration state table.
8. A data migration system for implementing the data migration method according to any one of claims 1 to 7, comprising a configuration module, an instruction module, a migration module, a checking module, and a deletion module, wherein:
the configuration module is used for acquiring a data migration configuration table, the migration configuration table comprises each data table to be migrated and a mapping relation table of a migration order corresponding to the data table, and the data table comprises order information of a plurality of orders;
the instruction module is used for determining a data table to be migrated from the business database at this time according to the migration configuration table; and
the order generation module is used for determining orders to be migrated according to the order sequence in the service database and generating migration instructions;
according to the migration instruction, migrating the order to be migrated from the service database to the first database;
the checking module is used for verifying the order information of each order transferred to the data table of the first database with the order information of the corresponding order in the data table of the service database;
and if the order passes the verification, the deleting module is used for deleting the order from the data table of the service database.
9. An electronic device, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the data migration method of any one of claims 1 to 7 via execution of the executable instructions.
10. A computer-readable storage medium storing a program which, when executed by a processor, performs the steps of the data migration method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111138193.3A CN113836114B (en) | 2021-09-27 | 2021-09-27 | Data migration method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111138193.3A CN113836114B (en) | 2021-09-27 | 2021-09-27 | Data migration method, system, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113836114A true CN113836114A (en) | 2021-12-24 |
| CN113836114B CN113836114B (en) | 2024-04-26 |
Family
ID=78970649
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111138193.3A Active CN113836114B (en) | 2021-09-27 | 2021-09-27 | Data migration method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113836114B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309945A (en) * | 2013-05-15 | 2013-09-18 | 上海证券交易所 | Device for importing data to database |
| CN104008157A (en) * | 2014-05-23 | 2014-08-27 | 国家电网公司 | Power grid system data migration method |
| WO2018113580A1 (en) * | 2016-12-19 | 2018-06-28 | 腾讯科技（深圳）有限公司 | Data management method and server |
| US10185727B1 (en) * | 2015-12-30 | 2019-01-22 | Amazon Technologies, Inc. | Migrating data between databases |
| US10346374B1 (en) * | 2014-03-14 | 2019-07-09 | Open Invention Network Llc | Optimized data migration application for database compliant data extraction, loading and transformation |
| CN110362531A (en) * | 2019-06-17 | 2019-10-22 | 众安在线财产保险股份有限公司 | A kind of automatic archiving method and device |
| US10657154B1 (en) * | 2017-08-01 | 2020-05-19 | Amazon Technologies, Inc. | Providing access to data within a migrating data partition |
| CN111324596A (en) * | 2020-03-06 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Data migration method and device for database cluster and electronic equipment |
| CN111708755A (en) * | 2020-05-20 | 2020-09-25 | 北京奇艺世纪科技有限公司 | Data migration method, device, system, electronic equipment and readable storage medium |
| CN112434013A (en) * | 2020-11-30 | 2021-03-02 | 中国建设银行股份有限公司 | Data table migration method and device, electronic equipment and storage medium |
-
2021
- 2021-09-27 CN CN202111138193.3A patent/CN113836114B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309945A (en) * | 2013-05-15 | 2013-09-18 | 上海证券交易所 | Device for importing data to database |
| US10346374B1 (en) * | 2014-03-14 | 2019-07-09 | Open Invention Network Llc | Optimized data migration application for database compliant data extraction, loading and transformation |
| CN104008157A (en) * | 2014-05-23 | 2014-08-27 | 国家电网公司 | Power grid system data migration method |
| US10185727B1 (en) * | 2015-12-30 | 2019-01-22 | Amazon Technologies, Inc. | Migrating data between databases |
| WO2018113580A1 (en) * | 2016-12-19 | 2018-06-28 | 腾讯科技（深圳）有限公司 | Data management method and server |
| US10657154B1 (en) * | 2017-08-01 | 2020-05-19 | Amazon Technologies, Inc. | Providing access to data within a migrating data partition |
| CN110362531A (en) * | 2019-06-17 | 2019-10-22 | 众安在线财产保险股份有限公司 | A kind of automatic archiving method and device |
| CN111324596A (en) * | 2020-03-06 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Data migration method and device for database cluster and electronic equipment |
| CN111708755A (en) * | 2020-05-20 | 2020-09-25 | 北京奇艺世纪科技有限公司 | Data migration method, device, system, electronic equipment and readable storage medium |
| CN112434013A (en) * | 2020-11-30 | 2021-03-02 | 中国建设银行股份有限公司 | Data table migration method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 贾海军: ""一种基于OGG方式进行数据迁移的研究"", 《软件》, vol. 36, no. 5, 15 May 2015 (2015-05-15), pages 140 - 144 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113836114B (en) | 2024-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9336224B2 (en) | Systems and methods for providing increased scalability in deduplication storage systems | |
| US9223564B2 (en) | Update systems responsive to ongoing processing at a storage system | |
| US8918783B2 (en) | Managing virtual computers simultaneously with static and dynamic dependencies | |
| US11036491B1 (en) | Identifying and resolving firmware component dependencies | |
| US10169085B2 (en) | Distributed computing of a task utilizing a copy of an original file stored on a recovery site and based on file modification times | |
| US10621003B2 (en) | Workflow handling in a multi-tenant cloud environment | |
| US11669504B2 (en) | Database schema branching workflow, with support for data, keyspaces and VSchemas | |
| US11775358B2 (en) | Tenant copy operation in a microservice architecture | |
| CN113792024A (en) | Method, device, equipment and storage medium for migrating data | |
| US8712967B1 (en) | Systems and methods for simultaneously configuring multiple independent backups | |
| CN115167782A (en) | Temporary storage copy management method, system, device and storage medium | |
| US20180210664A1 (en) | System and method for managing a storage system | |
| CN110753121A (en) | Mirror image warehouse deployment method, device, equipment and medium | |
| CN116483546B (en) | Distributed training task scheduling method, device, equipment and storage medium | |
| CN113609090A (en) | Data storage method and device, computer readable storage medium and electronic equipment | |
| CN113836114B (en) | Data migration method, system, equipment and storage medium | |
| US11644997B2 (en) | Releasing data storage tracks while maintaining logical corruption protection | |
| US9733931B2 (en) | Configuration management of engineering artifacts with automatic identification of related change sets based on type system | |
| US11513718B2 (en) | Method, electronic device and computer program product for configuring buffer size associated with backup threads | |
| US11656950B2 (en) | Method, electronic device and computer program product for storage management | |
| CN115422154A (en) | Automatic maintenance method, system, equipment and storage medium for database | |
| CN115858496A (en) | Data migration method and device | |
| CN116775102A (en) | Method, device, equipment and storage medium for acquiring code change quantity | |
| CN113282571A (en) | Data transfer method and device, electronic equipment and storage medium | |
| CN116955319A (en) | Service data migration method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |