CN112148705A - Data migration method and device - Google Patents
Data migration method and device Download PDFInfo
- Publication number
- CN112148705A CN112148705A CN201910576913.0A CN201910576913A CN112148705A CN 112148705 A CN112148705 A CN 112148705A CN 201910576913 A CN201910576913 A CN 201910576913A CN 112148705 A CN112148705 A CN 112148705A
- Authority
- CN
- China
- Prior art keywords
- migration
- data
- source
- task
- account
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Abstract
The invention discloses a method and a device for data migration, and relates to the technical field of computers. One embodiment of the apparatus comprises: the migration pre-module is used for monitoring migration request information, generating a migration task according to the migration request information and writing the migration task generated by the migration pre-module into the migration task table; the task acquisition module is used for scanning the migration task table, acquiring a migration task to be processed in the migration table and sending the migration task to the migration module; and the migration module is used for acquiring source data of a source account according to the migration task and storing the source data under the target account in a form of the target account. The implementation mode overcomes the technical defect that the data migration tool needs to be issued online again when the table structure is changed or the table code is newly added in the prior art, and further achieves the beneficial effect that the code of the data migration device does not need to be modified when the table structure is changed or the table structure is newly added.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for data migration.
Background
Data migration is the process of migrating the required data from a source database to a target database. The prior art mainly adopts two ways to migrate data:
converting data in a source relational database into TLV format data, and migrating the data in the TLV format file to a target database through a migration instance;
and secondly, storing the data to be migrated to a stack memory, converting the data into row records matched with the target database according to the type of the target database and the number of field columns of the table, packaging the row records into usable data objects conforming to the ETL, and finally writing the usable data objects into the target database.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
1. the data migration in the first mode does not support part of the original data, and efficient data migration cannot be achieved.
2. The second method for data migration needs table-by-table writing, cannot perform concurrent writing of multiple tables, is low in efficiency, and the risk of system abnormal data loss is easily caused by extracting data and placing the data in a memory stack.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for data migration, which can solve the technical defect that a tool for data migration needs to be issued online again when a table structure is changed or a table code is newly added in the prior art, and further achieve the beneficial effect that the code of the data migration apparatus does not need to be modified when the table structure is changed or the table structure is newly added.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided an apparatus for data migration, including:
the migration pre-module is used for monitoring migration request information, generating a migration task according to the migration request information and writing the migration task generated by the migration pre-module into the migration task table;
the task acquisition module is used for scanning the migration task table, acquiring a migration task to be processed in the migration table and sending the migration task to the migration module;
and the migration module is used for acquiring source data of a source account according to the migration task and storing the source data under the target account in a target account data storage mode.
Optionally, generating a migration task according to the migration request information includes:
inquiring a migration table uniformly configured by the distributed cluster according to the migration request information;
assembling the migration request information and the data inquired in the migration table to generate a migration task;
wherein, the migration request information includes: source account information, target account information, and migration request number.
Optionally, the task obtaining module is further configured to: and after the migration task is sent to a migration tool, setting the state of the migration task written into the migration task table to be sent by adopting an optimistic lock.
Optionally, the migration module is further configured to: prior to storing the source data under the target account in the form of the target account,
respectively positioning a source database and a source data table of a source account, a target database and a target data table of a target account, which correspond to the migration task;
respectively calculating the information of a source account database and a data table and the information of a target account database and a data table; and/or the presence of a gas in the gas,
and storing the database connection information of the source account and the target account into the variable of the current thread.
Optionally, obtaining source data of a source account according to the migration task, and storing the source data in a target account under the target account, including:
performing paging data extraction on the data to be migrated of the source account according to the migration task;
performing data conversion on the extracted paging data in a target account form;
storing the result of the data conversion under a target account.
Optionally, the system further comprises a migration checking module, configured to:
judging whether the data total amount of the source account before the migration is consistent with that of the target account after the migration;
if the total data amount is inconsistent, storing the source data under the target account in the form of the target account again;
if the total amount of data is consistent, comparing whether the total amount of the special fields configured by the source account and the target account is consistent;
if the sum of the special fields is consistent, the data of the source account and the data of the target account are consistent; and if the total amount of the special fields is inconsistent, storing the source data under the target account in the form of the target account again.
According to an aspect of an embodiment of the present invention, there is provided a method of data migration, including:
the migration front-end module monitors migration request information, generates a migration task according to the migration request information, and writes the generated migration task into the migration task table;
the task acquisition module scans the migration task table, acquires a migration task to be processed in the migration table, and sends the migration task to the migration module;
and the migration module acquires the source data of the source account according to the migration task and stores the source data in the target account in a target account data storage mode.
Optionally, generating a migration task according to the migration request information includes:
inquiring a migration table uniformly configured by the distributed cluster according to the migration request information;
assembling the migration request information and the data inquired in the migration table to generate a migration task;
wherein, the migration request information includes: source account information, target account information, and migration request number.
Optionally, after sending the migration task to the migration tool, the optimistic lock is used to set the state of the migration task written in the migration task table to be sent.
Optionally, prior to storing the source data under the target account in the form of the target account,
respectively positioning a source database and a source data table of a source account, a target database and a target data table of a target account, which correspond to the migration task;
respectively calculating the information of a source account database and a data table and the information of a target account database and a data table; and/or the presence of a gas in the gas,
and storing the database connection information of the source account and the target account into the variable of the current thread.
Optionally, obtaining source data of a source account according to the migration task, and storing the source data in a target account under the target account, including:
performing paging data extraction on the data to be migrated of the source account according to the migration task;
performing data conversion on the extracted paging data in a target account form;
storing the result of the data conversion under a target account.
Optionally, the method further comprises a migration check module for judging whether the total data amount of the source account before the migration is consistent with that of the target account after the migration;
if the total data amount is inconsistent, storing the source data under the target account in the form of the target account again;
if the total amount of data is consistent, comparing whether the total amount of the special fields configured by the source account and the target account is consistent;
if the sum of the special fields is consistent, the data of the source account and the data of the target account are consistent; and if the total amount of the special fields is inconsistent, storing the source data under the target account in the form of the target account again.
According to another aspect of the embodiments of the present invention, there is provided a data migration electronic device, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data migration apparatus provided by the present invention.
According to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing the data migration apparatus provided by the present invention.
One embodiment of the above invention has the following advantages or benefits:
the invention adopts the technical means that the module for acquiring the task is arranged outside the migration module and is uniformly acquired by the task acquisition module, thereby solving the technical defects that in the prior art, a tool for data migration needs to be released online again when the table structure is changed or the table code is newly added, and a plurality of repeated main keys possibly exist after the data migration, and further achieving the beneficial effect that the code of the data migration device does not need to be modified when the table structure is changed or the table structure is newly added.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main modules of an apparatus for data migration according to an embodiment of the present invention;
FIG. 2 is a system diagram of an apparatus for data migration according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main flow of a data migration method according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating a detailed flow of a data migration method according to an embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 6 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of main modules of an apparatus 100 for data migration according to an embodiment of the present invention, as shown in fig. 1,
the migration pre-processing module 101 is configured to monitor migration request information, generate a migration task according to the migration request information, and write the migration task generated by the migration pre-processing module into the migration task table;
a task obtaining module 102, configured to scan the migration task table, obtain a migration task to be processed in the migration table, and send the migration task to a migration module;
and the migration module 103 is configured to obtain source data of a source account according to the migration task, and store the source data in a target account in a form of target account data storage.
The source account and the target account may be located in the same database or may be different databases.
The monitoring migration request information is obtained in an asynchronous mode through the message queue, the concurrency of the system is greatly improved, and the defects that data migration in the prior art is long in time consumption and low in efficiency are overcome.
And the migration task is sent to the migration module in an asynchronous mode, so that the concurrency performance of the system is further improved, and the efficiency of data migration is improved.
The task obtaining module may obtain a task to be data migrated in the migration task table in a timing scanning manner, or may send information to the task obtaining module by using the migration task table.
The invention adopts the technical means that the module for acquiring the task is arranged outside the migration module and is uniformly acquired by the task acquisition module, thereby solving the technical defects that in the prior art, a tool for data migration needs to be released online again when the table structure is changed or the table code is newly added, and a plurality of repeated main keys possibly exist after the data migration, and further achieving the beneficial effect that the code of the data migration device does not need to be modified when the table structure is changed or the table structure is newly added.
By means of the technical means of completing data migration based on the distributed cluster, the technical defects that only the whole table in the database can be migrated, partial data in the table cannot be migrated, and the migration speed is low when data migration is performed by multithreading in the prior art are overcome, and the technical effects of more flexible data migration and high migration efficiency are achieved. And the data migration is carried out by depending on the distributed cluster, so that the concurrency rate of the data migration can be improved, and the efficiency of the data migration is higher.
Optionally, generating a migration task according to the migration request information includes:
inquiring a migration table uniformly configured by the distributed cluster according to the migration request information;
assembling the migration request information and the data inquired in the migration table to generate a migration task;
wherein, the migration request information includes: source account information, target account information, and migration request number.
By the migration request information, the whole data table does not need to be migrated when partial data needs to be migrated, and partial data of each table can be migrated.
The migration table uniformly configured by the distributed cluster is a dynamically configured table to be migrated, so that the technical effects of flexible configuration, improvement of data migration integrity and guarantee of high efficiency and reliability of a data migration process are achieved.
Optionally, the task obtaining module further includes: and after the migration task is sent to a migration module, setting the state of the migration task written into the migration task table to be sent by adopting an optimistic lock.
Through the optimistic lock, the situation that the main key is possibly repeated after the migration in the prior art is avoided, and the technical problem of data migration consistency is solved.
Optionally, the migration module further includes: prior to storing the source data under the target account in the form of the target account,
respectively positioning a source database and a source data table of a source account, a target database and a target data table of a target account, which correspond to the migration task, so as to determine the migration position and destination of the migration task data;
respectively calculating the information of a source account database and a data table and the information of a target account database and a data table; and/or the presence of a gas in the gas,
and the database connection information of the source account and the target account is stored in the variable of the current thread, so that subsequent data query, insertion and update operations are facilitated.
The contents of the source account database and the data table information, and the database and the data table information of the target account are hash code values thereof.
Specifically, the calculation formula of the database information of the source account or the target account is as follows:
database information (hash value% (database number table number)/table number);
the calculation formula of the data table information of the source account or the target account is as follows:
table information is hash value% table number.
Optionally, obtaining source data of a source account according to the migration task, and storing the source data in a target account under the target account, including:
performing paging data extraction on the data to be migrated of the source account according to the migration task;
performing data conversion on the extracted paging data in a target account form;
storing the result of the data conversion under a target account.
And the weighting dynamic setting can be carried out according to the data quantity of the table and the weight of the table configuration.
Optionally, the method further comprises: and the migration checking module is used for checking the data migrated to the target account. The check field of each table in the database is configurable based on the distributed configuration system, so that the flexibility of the system is increased.
The technical defects that in the prior art, due to the fact that extracted data are placed in a memory stack, system abnormal data are easily lost, and a risk avoiding scheme is not provided are overcome through the technical means of verification after data migration, and the technical effect of improving the accuracy of data migration is achieved.
Specifically, the verifying the data migrated to the target account includes:
judging whether the data total amount of the source account before the migration is consistent with that of the target account after the migration;
if the total data amount is inconsistent, storing the source data under the target account in the form of the target account again;
if the total amount of data is consistent, comparing whether the total amount of the special fields configured by the source account and the target account is consistent;
if the sum of the special fields is consistent, the data of the source account and the data of the target account are consistent; and if the total amount of the special fields is inconsistent, storing the source data under the target account in the form of the target account again.
The data verification after the migration is mainly the comparison of partial data between the source account and the target account, and the comparison information can be configured more flexibly. The integrity of the migrated data can be further determined by data checking.
FIG. 2 is a system diagram of an apparatus for data migration according to an embodiment of the present invention, as shown in FIG. 2:
the migration pre-processing module 201 is configured to monitor migration request information, determine information corresponding to a migration table configured in a unified manner by the distributed cluster according to the migration request information, generate a migration task according to the two types of information, and write the migration task into the migration task table;
a task obtaining module 202, configured to scan the migration task table, obtain a migration task to be processed in the migration table, and send the migration task to a migration module;
the migration module 203 is configured to locate, according to the task sent by the task obtaining module, a source database and a source data table of a source account, a target database and a target data table of a target account, which correspond to the migration task, and further determine a location and a destination where the migration task data is migrated; then respectively calculating the information of a source account database and a data table, and the information of a target account database and a data table; storing database connection information of a source account and a target account into a variable of a current thread; and finally, storing the source data under the target account in the form of the target account.
FIG. 3 is a schematic diagram of the main flow of a data migration method according to an embodiment of the present invention;
as shown in the figure, the present embodiment provides a method for data migration, including:
step S301, a migration front-end module monitors migration request information, generates a migration task according to the migration request information, and writes the generated migration task into a migration task table;
step S302, a task acquisition module scans the migration task table, acquires a migration task to be processed in the migration table, and sends the migration task to a migration module;
step S303, the migration module acquires source data of a source account according to the migration task, and stores the source data in a target account data storage form.
Optionally, generating a migration task according to the migration request information includes:
inquiring a migration table uniformly configured by the distributed cluster according to the migration request information;
assembling the migration request information and the data inquired in the migration table to generate a migration task;
wherein, the migration request information includes: source account information, target account information, and migration request number.
Optionally, after the task obtaining module sends the migration task to the migration tool, the state of the migration task written in the migration task table is set to be sent by using an optimistic lock.
Optionally, before the migration module stores the source data under the target account in the form of the target account,
respectively positioning a source database and a source data table of a source account, a target database and a target data table of a target account, which correspond to the migration task;
respectively calculating the information of a source account database and a data table and the information of a target account database and a data table; and/or the presence of a gas in the gas,
and storing the database connection information of the source account and the target account into the variable of the current thread.
Optionally, obtaining source data of a source account according to the migration task, and storing the source data in a target account under the target account, including:
performing paging data extraction on the data to be migrated of the source account according to the migration task;
performing data conversion on the extracted paging data in a target account form;
storing the result of the data conversion under a target account.
Optionally, the system further comprises a migration checking module, configured to:
judging whether the data total amount of the source account before the migration is consistent with that of the target account after the migration;
if the total data amount is inconsistent, storing the source data under the target account in the form of the target account again;
if the total amount of data is consistent, comparing whether the total amount of the special fields configured by the source account and the target account is consistent;
if the sum of the special fields is consistent, the data of the source account and the data of the target account are consistent; and if the total amount of the special fields is inconsistent, storing the source data under the target account in the form of the target account again.
Fig. 4 is a schematic diagram of a specific flow of a data migration method according to an embodiment of the present invention, as shown in fig. 4:
s401, monitoring migration request information;
s402, inquiring a migration table uniformly configured by the distributed cluster according to the migration request information;
s403, assembling the migration request information and the data inquired in the migration table, generating a migration task, and inserting the migration task into a migration task table;
s404, scanning the migration table at regular time;
s405, judging whether the scanning result is 0 or not according to the scanning record, if so, indicating that no task needing data migration exists in the migration table, and executing S406 to finish the operation; when the scanning result is not 0, indicating that a task needing data migration exists in the migration table, and executing S407 to migrate the data to be migrated;
s408, judging whether the parameter verification is successful;
if not, executing S409 to finish the data migration; if the operation is successful, S410 is executed to respectively locate the databases and the data tables of the source account and the target account;
s411, performing extraction and conversion processing operation on data to be migrated of the source account;
s412, migrating the processed data to a target account;
s413, verifying the migrated data;
s414, judging whether the verification is successful, and if the verification is successful, ending the process; and if the data migration is unsuccessful, the data migration is carried out again.
Fig. 5 illustrates an exemplary system architecture 500 to which the data migration method or the data migration apparatus of the embodiments of the present invention may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 serves to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. The terminal devices 501, 502, 503 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 501, 502, 503. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the data migration method provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the data migration apparatus is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks, and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, a block diagram of a computer system 600 suitable for use with a terminal device implementing an embodiment of the invention is shown. The terminal device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the computer system 600 includes a central processing module (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the system of the present invention when executed by the central processing module (CPU) 601.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a sending module, an obtaining module, a determining module, and a first processing module. The names of these modules do not form a limitation on the modules themselves in some cases, and for example, the sending module may also be described as a "module sending a picture acquisition request to a connected server".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:
the migration front-end module monitors migration request information, generates a migration task according to the migration request information, and writes the generated migration task into the migration task table;
the task acquisition module scans the migration task table, acquires a migration task to be processed in the migration table, and sends the migration task to the migration module;
and the migration module acquires the source data of the source account according to the migration task and stores the source data in the target account in a target account data storage mode.
According to the technical scheme of the embodiment of the invention, the following beneficial effects can be achieved:
the invention adopts the technical means that the module for acquiring the task is arranged outside the migration module and is uniformly acquired by the task acquisition module, thereby solving the technical defects that in the prior art, a tool for data migration needs to be released online again when the table structure is changed or the table code is newly added, and a plurality of repeated main keys possibly exist after the data migration, and further achieving the beneficial effect that the code of the data migration device does not need to be modified when the table structure is changed or the table structure is newly added.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (14)
1. An apparatus for data migration, comprising:
the migration pre-module is used for monitoring migration request information, generating a migration task according to the migration request information and writing the migration task generated by the migration pre-module into the migration task table;
the task acquisition module is used for scanning the migration task table, acquiring a migration task to be processed in the migration table and sending the migration task to the migration module;
and the migration module is used for acquiring source data of a source account according to the migration task and storing the source data under the target account in a target account data storage mode.
2. The apparatus of claim 1, wherein generating a migration task according to the migration request information comprises:
inquiring a migration table uniformly configured by the distributed cluster according to the migration request information;
assembling the migration request information and the data inquired in the migration table to generate a migration task;
wherein, the migration request information includes: source account information, target account information, and migration request number.
3. The apparatus of claim 1, wherein the task obtaining module is further configured to: and after the migration task is sent to a migration tool, setting the state of the migration task written into the migration task table to be sent by adopting an optimistic lock.
4. The apparatus of claim 1, wherein the migration module is further configured to: prior to storing the source data under the target account in the form of the target account,
respectively positioning a source database and a source data table of a source account, a target database and a target data table of a target account, which correspond to the migration task;
respectively calculating the information of a source account database and a data table and the information of a target account database and a data table; and/or the presence of a gas in the gas,
and storing the database connection information of the source account and the target account into the variable of the current thread.
5. The apparatus of claim 1, wherein obtaining source data of a source account according to the migration task, and storing the source data under a target account in the form of the target account comprises:
performing paging data extraction on the data to be migrated of the source account according to the migration task;
performing data conversion on the extracted paging data in a target account form;
storing the result of the data conversion under a target account.
6. The apparatus of claim 1, further comprising a migration check module to:
judging whether the data total amount of the source account before the migration is consistent with that of the target account after the migration;
if the total data amount is inconsistent, storing the source data under the target account in the form of the target account again;
if the total amount of data is consistent, comparing whether the total amount of the special fields configured by the source account and the target account is consistent;
if the sum of the special fields is consistent, the data of the source account and the data of the target account are consistent; and if the total amount of the special fields is inconsistent, storing the source data under the target account in the form of the target account again.
7. A method of data migration, comprising:
the migration front-end module monitors migration request information, generates a migration task according to the migration request information, and writes the generated migration task into the migration task table;
the task acquisition module scans the migration task table, acquires a migration task to be processed in the migration table, and sends the migration task to the migration module;
and the migration module acquires the source data of the source account according to the migration task and stores the source data in the target account in a target account data storage mode.
8. The method of claim 7, wherein generating a migration task according to the migration request information comprises:
inquiring a migration table uniformly configured by the distributed cluster according to the migration request information;
assembling the migration request information and the data inquired in the migration table to generate a migration task;
wherein, the migration request information includes: source account information, target account information, and migration request number.
9. The method of claim 7, wherein after sending the migration task to a migration tool, an optimistic lock is used to set the status of the migration task that has been written to the migration task table to sent.
10. The method of claim 7, wherein prior to storing the source data under a target account,
respectively positioning a source database and a source data table of a source account, a target database and a target data table of a target account, which correspond to the migration task;
respectively calculating the information of a source account database and a data table and the information of a target account database and a data table; and/or the presence of a gas in the gas,
and storing the database connection information of the source account and the target account into the variable of the current thread.
11. The method of claim 7, wherein obtaining source data of a source account according to the migration task, and storing the source data under a target account in the form of the target account comprises:
performing paging data extraction on the data to be migrated of the source account according to the migration task;
performing data conversion on the extracted paging data in a target account form;
storing the result of the data conversion under a target account.
12. The method according to claim 7, further comprising a migration check module for determining whether the total data amount of the source account before the migration is consistent with that of the target account after the migration;
if the total data amount is inconsistent, storing the source data under the target account in the form of the target account again;
if the total amount of data is consistent, comparing whether the total amount of the special fields configured by the source account and the target account is consistent;
if the sum of the special fields is consistent, the data of the source account and the data of the target account are consistent; and if the total amount of the special fields is inconsistent, storing the source data under the target account in the form of the target account again.
13. An electronic device for data migration, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 7-12.
14. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 7-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910576913.0A CN112148705A (en) | 2019-06-28 | 2019-06-28 | Data migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910576913.0A CN112148705A (en) | 2019-06-28 | 2019-06-28 | Data migration method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112148705A true CN112148705A (en) | 2020-12-29 |
Family
ID=73869536
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910576913.0A Pending CN112148705A (en) | 2019-06-28 | 2019-06-28 | Data migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112148705A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023137919A1 (en) * | 2022-01-21 | 2023-07-27 | 平安科技（深圳）有限公司 | Control method and system for live migration of virtual machine, electronic device, and storage medium |
-
2019
- 2019-06-28 CN CN201910576913.0A patent/CN112148705A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023137919A1 (en) * | 2022-01-21 | 2023-07-27 | 平安科技（深圳）有限公司 | Control method and system for live migration of virtual machine, electronic device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019350B (en) | Data query method and device based on configuration information | |
| US9817879B2 (en) | Asynchronous data replication using an external buffer table | |
| CN109657174B (en) | Method and device for updating data | |
| CN110262807B (en) | Cluster creation progress log acquisition system, method and device | |
| CN111865970B (en) | Method and apparatus for implementing interface idempotency | |
| CN105205154B (en) | Data migration method and device | |
| CN112765152B (en) | Method and apparatus for merging data tables | |
| CN111338834B (en) | Data storage method and device | |
| CN112597126A (en) | Data migration method and device | |
| CN113190517B (en) | Data integration method and device, electronic equipment and computer readable medium | |
| CN112818026A (en) | Data integration method and device | |
| CN112148705A (en) | Data migration method and device | |
| CN116069725A (en) | File migration method, device, apparatus, medium and program product | |
| CN115454971A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN111026629A (en) | Method and device for automatically generating test script | |
| CN113760861A (en) | Data migration method and device | |
| CN112506490A (en) | Interface generation method and device, electronic equipment and storage medium | |
| CN109087097B (en) | Method and device for updating same identifier of chain code | |
| CN112988857A (en) | Service data processing method and device | |
| CN112214500A (en) | Data comparison method and device, electronic equipment and storage medium | |
| CN113347052A (en) | Method and device for counting user access data through access log | |
| CN112711572A (en) | Online capacity expansion method and device suitable for sub-warehouse and sub-meter | |
| CN111290927A (en) | Data monitoring method and device | |
| CN109710673B (en) | Work processing method, device, equipment and medium | |
| CN110750410B (en) | Method and device for monitoring database logs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Beijing Economic and Technological Development Zone, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Beijing Economic and Technological Development Zone, 100176 Applicant before: Jingdong Digital Technology Holding Co.,Ltd. Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Beijing Economic and Technological Development Zone, 100176 Applicant after: Jingdong Digital Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Beijing Economic and Technological Development Zone, 100176 Applicant before: JINGDONG DIGITAL TECHNOLOGY HOLDINGS Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |