CN111291023A - Data migration method, system, device and medium - Google Patents
Data migration method, system, device and medium Download PDFInfo
- Publication number
- CN111291023A CN111291023A CN202010085526.XA CN202010085526A CN111291023A CN 111291023 A CN111291023 A CN 111291023A CN 202010085526 A CN202010085526 A CN 202010085526A CN 111291023 A CN111291023 A CN 111291023A
- Authority
- CN
- China
- Prior art keywords
- data
- change record
- fragment
- migration
- change
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 230000005012 migration Effects 0.000 title claims abstract description 59
- 238000013508 migration Methods 0.000 title claims abstract description 59
- 230000008859 change Effects 0.000 claims abstract description 88
- 239000012634 fragment Substances 0.000 claims abstract description 60
- 230000008569 process Effects 0.000 claims abstract description 25
- 238000013507 mapping Methods 0.000 claims abstract description 9
- 238000013467 fragmentation Methods 0.000 claims description 20
- 238000006062 fragmentation reaction Methods 0.000 claims description 20
- 238000012546 transfer Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 14
- 230000004044 response Effects 0.000 claims description 9
- 230000000694 effects Effects 0.000 abstract description 9
- 238000000605 extraction Methods 0.000 abstract description 7
- 238000013500 data storage Methods 0.000 abstract 1
- 238000013075 data extraction Methods 0.000 description 9
- 230000007246 mechanism Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 238000012217 deletion Methods 0.000 description 5
- 230000037430 deletion Effects 0.000 description 5
- 238000007792 addition Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000004880 explosion Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data migration method, which comprises the following steps: acquiring a dump parameter; creating a change record table; creating multiple threads according to the unloading parameters and segmenting data to be migrated to obtain multiple segmented data; establishing mapping between the fragment data and each single thread in the multiple threads, so that the corresponding fragment data is migrated to a target database by using each single thread, and data change generated in the migration process is recorded by using the change record table; and responding to the end of the migration of the plurality of fragmented data, and changing the corresponding data in the target database according to the data change record in the change record table. The invention also discloses a system, a computer device and a readable storage medium. The scheme provided by the invention realizes the technical effects of non-stop cross-database parallel extraction and super-large-scale mass data storage, improves the data migration efficiency, and ensures the integrity and accuracy of data migration.
Description
Technical Field
The present invention relates to the field of databases, and in particular, to a method, a system, a device, and a storage medium for data migration.
Background
In the era, information systems are not invariable during rapid transition, and when subversion upgrading is needed, the possibility that all related data need to be migrated exists, and the method is not limited to the above situation, and various requirements for data migration are needed, and the problem of complete data migration is also needed to be solved. Generally, in an effort to achieve brevity while maintaining data migration integrity and accuracy, technicians tend to shut down operations, i.e., suspend open online services, shut down all systems that may affect the data, and then migrate the data in a relatively simple manner. When the method is used for some services which cannot be stopped and are compared with cores, the method cannot guarantee the integrity and the accuracy of data, because data migration cannot be completed in a short time, data change may exist in the migration process, and the problem that the accuracy of the migrated data is lost due to timeliness is likely to occur, so that the migrated data has the possibility of errors and loss. This problem becomes even more unattractive when the data size is large, the period for migrating data is very long, and the service affecting data is executed at a high frequency.
Disclosure of Invention
In view of the above, in order to overcome at least one aspect of the above problems, the present invention provides a data migration method, including the steps of:
acquiring a dump parameter;
creating a change record table;
creating multiple threads according to the unloading parameters and segmenting data to be migrated to obtain multiple segmented data;
establishing mapping between the fragment data and each single thread in the multiple threads, so that the corresponding fragment data is migrated to a target database by using each single thread, and data change generated in the migration process is recorded by using the change record table;
and responding to the end of the migration of the plurality of fragmented data, and changing the corresponding data in the target database according to the data change record in the change record table.
In some embodiments, the fragmenting the data to be migrated according to the unloading parameter to obtain a plurality of fragmented data further includes:
determining the fragment size of each fragment data according to the unloading parameters;
determining a plurality of segment indexes by using the segment size;
and taking data between two continuous fragment indexes as the fragment data.
In some embodiments, determining a plurality of shard indices using the shard size further comprises:
and taking the main key or the joint main key at the position of integral multiple of the size of the fragment as the fragment index, and taking the first and the last main keys or the joint main keys as the fragment index.
In some embodiments, creating the change log table further comprises:
create add triggers, modify triggers, and delete triggers.
In some embodiments, recording the data change generated during the migration process by using the change log table further includes:
and recording the parameter information corresponding to the data change generated in the migration process into the change record table by using the adding trigger, the modifying trigger and/or the deleting trigger.
In some embodiments, changing the corresponding data in the target database according to the data change record in the change record table further includes:
and in response to the data change record in the change record table being deleted, deleting the corresponding data in the target database.
In some embodiments, further comprising:
in response to the data change record in the change record table being added or modified, deleting the data corresponding to the target database;
and acquiring the changed data and updating the changed data into the target database.
Based on the same inventive concept, according to another aspect of the present invention, the present invention further provides a system for data migration, comprising:
an acquisition module configured to acquire a transfer parameter;
a creation module configured to create a change log table;
the fragmentation module is configured to create multiple threads according to the unloading parameters and fragment data to be migrated to obtain multiple pieces of fragmentation data;
the migration module is configured to establish mapping between the fragment data and each single thread in the multiple threads, so that each single thread is used for migrating the corresponding fragment data to a target database, and the change record table is used for recording data changes generated in the migration process;
a changing module configured to, in response to the completion of the migration of the plurality of fragmented data, change corresponding data in the target database according to a data change record in the change record table.
Based on the same inventive concept, according to another aspect of the present invention, the present invention also provides a computer apparatus comprising:
at least one processor; and
a memory storing a computer program operable on the processor, wherein the processor executes the program to perform any of the steps of the method of data migration described above.
Based on the same inventive concept, according to another aspect of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed by a processor, performs the steps of any of the methods of data migration described above.
The invention has one of the following beneficial technical effects: the scheme provided by the invention realizes the high-efficiency extraction and the transfer of mass data of multiple threads without mutual sensing through a fragmentation mechanism, ensures the accuracy and the integrity of the data transfer through a mechanism of changing a record table, realizes the technical effects of extracting and transferring the mass data of a super-large scale in parallel across databases without stopping, improves the data transfer efficiency, and ensures the integrity and the accuracy of the data transfer.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a flow chart illustrating a method for data migration according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a data migration method according to an embodiment of the present invention;
FIG. 3 is a block diagram of a system for data migration according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a computer device provided in an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
It should be noted that, in the embodiment of the present invention, the non-stop means that other running services affecting the migrated data are allowed to continue providing services without being temporarily stopped, and data changes occurring in the process can be completely accommodated during data extraction and data dump, and the integrity and accuracy of the final data migration can be guaranteed; data extraction and unloading are simultaneously carried out through a multi-task method, the processing efficiency is improved, and the tasks are independently executed in parallel without sensing the existence of other tasks; the cross-database not only supports the number of the cross-databases, but also supports the types of the cross-databases, supports the mutual data extraction and the transfer of all mainstream databases in the current industry, allows the data extraction from the same database, and simultaneously transfers the data to a plurality of databases of different types; finally, the scheme provided by the invention supports the migration of mass data with super-large scale, and the complete and accurate data migration can be realized even if the data volume is very large.
According to an aspect of the present invention, the present invention provides a method of data migration, as shown in fig. 1, which may include the steps of: s1, obtaining a dump parameter; s2, creating a change record table; s3, creating multiple threads according to the unloading parameters and slicing the data to be migrated to obtain multiple sliced data; s4, establishing mapping between the fragment data and each single thread in the multiple threads, so as to migrate the corresponding fragment data to a target database by using each single thread, and recording data change generated in the migration process by using the change record table; and S5, responding to the end of the migration of the plurality of fragmented data, and changing the corresponding data in the target database according to the data change record in the change record table.
The scheme provided by the invention realizes the high-efficiency extraction and the transfer of mass data of multiple threads without mutual sensing through a fragmentation mechanism, ensures the accuracy and the integrity of the data transfer through a mechanism of changing a record table, realizes the technical effects of extracting and transferring the mass data of a super-large scale in parallel across databases without stopping, improves the data transfer efficiency, and ensures the integrity and the accuracy of the data transfer.
It should be noted that it is never a problem that a single task implements data extraction, and a key core problem to be solved by the solution proposed by the present invention is how to extract data in parallel by using multiple tasks without duplication and loss and ensure data accuracy under the condition that the data is unstable and frequently fluctuates. Therefore, under the condition that multiple threads are executed in parallel and no perception exists among the threads, the data to be extracted must be fragmented according to certain logic, the fragmented data needs to be mapped into different threads, each thread only extracts and unloads the data which are mapped into the corresponding thread and are contained in the fragment, and the threads do not influence each other in data. Scientific sharding should have the characteristic that the data volume of each shard is theoretically consistent at the instant of sharding, even if data changes, the change of the number of each shard should be equal, the situation that the data volume is seriously inclined when sharding does not exist, and the problem that data explosion occurs to some shards when the data changes should be avoided as much as possible. The key point of the fragmentation process is to confirm the fragmentation basis and how to use the fragmentation basis to perform fragmentation, generally speaking, no matter what the data table is used or what data is stored in the table, a main key or a combined main key is generally designed by scientific table structure design to serve as a unique identifier of the data, the main key serves as a first choice of the non-repeatable data belonging to the fragmentation in the whole table, and under the design specification of an enterprise-level database, the main key in each table must exist, and especially in the data table containing the ultra-large-scale mass data, the role played by the main key cannot be replaced.
In some embodiments, in the step S1, the dump parameter may be the number of threads, the size of the sliced data, all target databases, etc.
Specifically, the optimal number of threads for extracting data in parallel and the amount of data extracted each time can be reasonably calculated according to the configuration of the actual operating environment, such as the parameters of a CPU and a memory. And then selecting a proper programming language according to the actual service, so that after the original database and all target databases are determined, all database drivers can be prepared according to the used programming language, the original database and all target databases are accessed through programming, and the data loading, inserting, modifying, executing and executing of other various database languages can be executed.
It should be noted that, the more and better the threads that are not started and executed in parallel, increasing the number of threads within a certain range may improve efficiency, but the resource consumption rate is increased and then limited by performance bottleneck, the performance is decreased due to the problem of system resource limitation caused by too many threads or too large loading data amount, and even more, the possibility of process deadlock exists.
In some embodiments, the creating the change log table in step S2 may further include:
create add triggers, modify triggers, and delete triggers.
In some embodiments, recording the data change generated during the migration process by using the change log table further includes:
and recording the parameter information corresponding to the data change generated in the migration process into the change record table by using the adding trigger, the modifying trigger and/or the deleting trigger.
Specifically, in order to ensure reliability and accuracy during data migration, before data fragmentation, three triggers for respectively recording changes of addition, modification and deletion of a data table are required to be established in an original database, and after the data fragmentation is completed, the triggers take effect to start recording all data change processes after the moment, and record changed data on a change record table. The change record table may include fields, which are respectively a primary key identifier, a primary key identifier of the original data (all fields are added correspondingly by combining the primary key), a change mode, and a change time, and respectively records primary key data of the change record table (the primary key identifier has no effect in the data migration process, and only the primary key identifier necessary for the change record table itself), primary key data corresponding to the changed data in the original data table, which change type (addition, modification, deletion) the change type belongs to, and a time when the change occurs.
In some embodiments, in step S3, the fragmenting the data to be migrated according to the unloading parameter to obtain a plurality of fragmented data, further includes:
determining the fragment size of each fragment data according to the unloading parameters;
determining a plurality of segment indexes by using the segment size;
and taking data between two continuous fragment indexes as the fragment data.
In some embodiments, determining a plurality of shard indices using the shard size further comprises:
and taking the main key or the joint main key at the position of integral multiple of the size of the fragment as the fragment index, and taking the first and the last main keys or the joint main keys as the fragment index.
Specifically, if there is no primary key but there is a unique index in the table during data migration, the field can also be used as the primary key, and the field can be regarded as the primary key of the table; after the data size of each piece of data, i.e. the size of each piece, obtained in step S1, the full table is sorted according to the size order of the primary keys, and the primary key data at the position of an integral multiple of the size of each piece is retrieved as the piece index. It is stated that fragmentation is a very short time to complete process, and the data size of each fragment is completely consistent at the instant except that the data size of the last fragment is uncertain.
It should be noted that, there is no possibility of data explosion in the process of performing data extraction until the end of data extraction, deletion and modification, and the fragmentation problem caused by the possibility is less affected, and as for the inserted data, the new data entering each fragment theoretically has equal chance because of the randomness. After the fragmentation is finished, the data can be extracted and transferred in parallel in a multithread execution mode, all threads only extract and transfer the data mapped into the fragments of the threads, and the fragments are sequentially executed in a single thread until the completion of all the fragments. The data is transferred in a slicing mode only to solve the problem of established data migration, and certainly, under the condition of no shutdown, the three triggers and the change record table are needed to be used for synchronizing data addition, deletion and change in the process to the target database.
In some embodiments, in step S4, a mapping between the multiple pieces of fragment data and each single thread in the multiple threads is established, so that the corresponding piece of fragment data is migrated to the target database by using each single thread, and the change record table is used to record data changes generated during the migration process, where a mapping relationship between each piece of fragment data and each single thread may be as shown in fig. 2, where a first piece of fragment data is mapped to a first thread, a second piece of fragment data is mapped to a second thread, a third piece of fragment data is mapped to a third thread, a fourth piece of fragment data is mapped to a fourth thread, a fifth piece of fragment data is mapped to a first thread, a sixth piece of fragment data is mapped to a second thread, and so on. Of course, the number of threads may be more or less than the number of threads illustrated.
In some embodiments, in step S5, changing the corresponding data in the target database according to the data change record in the change record table further includes:
and in response to the data change record in the change record table being deleted, deleting the corresponding data in the target database.
In some embodiments, the method further comprises:
in response to the data change record in the change record table being added or modified, deleting the data corresponding to the target database;
and acquiring the changed data and updating the changed data into the target database.
Specifically, after fragmentation, parallel extraction and data transfer are completed, as shown in fig. 2, data in the change record table is sequentially read from the beginning to the end according to time sequence by an independently opened thread (thread five), and is updated into the target database, for data with an operation type of addition and modification, if corresponding data can be found in the target database, the corresponding data is deleted first, then the corresponding data is found from the original database and is updated into the target database, if the original data table does not have the data, the operation is abandoned, and for data with an operation type of deletion, the corresponding data is directly found in the target database and is directly deleted, and the operation is completed until the end. It should be explained that the operation type is added data, and the reason why the corresponding data is tried to be retrieved and deleted in the target table first is that there is a certain time delay in the data extraction process, the following fragment is processed only after the previous fragment is processed, and the possibility that the extracted and dumped data is recorded by the trigger because the data is added newly exists. After the above processes are completed, the data in the target database table is completely consistent with the original data. The theoretical statement proves to have better scientificity in practice, supports the extraction of data from the same database and the data transfer to a plurality of databases at the same time, and only needs to make relevant configuration when development is realized.
The scheme provided by the invention creates a database table with the same structure in a target database, creates a change record table in an original database, wherein fields comprise main key identifications, original data main key identifications (all fields are correspondingly added by combining the main keys), change modes and change time, creates triggers for adding, modifying and deleting data in the original database aiming at an operated table, correspondingly records the added and deleted data into the change record table, obtains a fragment index according to a fragment theory of the preceding analysis, maps the fragment index to each thread to be created, creates a plurality of threads for parallel extraction to realize data transfer, creates the threads to synchronously change all changes in the record table to the target data table after all the operations are completely executed, and finally clears all the triggers.
The scheme provided by the invention fully uses a trigger mechanism of the database to ensure the integrity and accuracy of the online migration of the data, and improves the data extraction efficiency by using a scientific fragmentation strategy and a multithreading mechanism, so that the data unloading is more modularized, the service flow is clearer, and the data processing is more guaranteed. The technical effect of non-stop cross-database parallel extraction and transfer of ultra-large-scale mass data is achieved, the data transfer efficiency is improved, and the integrity and accuracy of data transfer are guaranteed. The scheme can be applied to various modes, for example, a set of client window application program can be developed for data migration, various operation configurations such as a database, a data table, thread number, fragment number and the like can be supported, a console command program can also be developed, data migration can be completed simply and efficiently through a command mode, various mainstream frameworks can also be integrated, and the effect of realizing the subject can be completed by various operation means. The introduced mechanism enables the service function to be more perfect, the operation to be more reliable, and the mechanism is popular with customers in practical use.
Based on the same inventive concept, according to another aspect of the present invention, the present invention further provides a system 400 for data migration, as shown in fig. 3, comprising:
an obtaining module 401, where the obtaining module 401 is configured to obtain a dump parameter;
a creation module 402, the creation module 402 configured to create a change log table;
the fragmentation module 403, where the fragmentation module 403 is configured to create multiple threads according to the dump parameter and fragment data to be migrated to obtain multiple pieces of fragmented data;
a migration module 404, where the migration module 404 is configured to establish mapping between the multiple pieces of fragment data and each single thread in the multiple threads, so as to migrate the corresponding piece of fragment data to a target database by using each single thread, and record data changes generated in a migration process by using the change log table;
a changing module 405, where the changing module 405 is configured to, in response to the migration of the plurality of sharded data ending, change corresponding data in the target database according to a data change record in the change record table.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 4, the present invention further provides a computer apparatus 501, comprising:
at least one processor 520; and
the memory 510, the memory 510 storing a computer program 511 executable on the processor, the processor 520 executing the program to perform the steps of any of the above methods of data migration.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 5, the present invention further provides a computer-readable storage medium 601, the computer-readable storage medium 601 stores computer program instructions 610, and the computer program instructions 610, when executed by a processor, perform the steps of any of the above methods for data migration.
Finally, it should be noted that, as will be understood by those skilled in the art, all or part of the processes of the methods of the above embodiments may be implemented by a computer program to instruct related hardware to implement the methods. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
In addition, the apparatuses, devices, and the like disclosed in the embodiments of the present invention may be various electronic terminal devices, such as a mobile phone, a Personal Digital Assistant (PDA), a tablet computer (PAD), a smart television, and the like, or may be a large terminal device, such as a server, and the like, and therefore the scope of protection disclosed in the embodiments of the present invention should not be limited to a specific type of apparatus, device. The client disclosed by the embodiment of the invention can be applied to any one of the electronic terminal devices in the form of electronic hardware, computer software or a combination of the electronic hardware and the computer software.
Furthermore, the method disclosed according to an embodiment of the present invention may also be implemented as a computer program executed by a CPU, and the computer program may be stored in a computer-readable storage medium. The computer program, when executed by the CPU, performs the above-described functions defined in the method disclosed in the embodiments of the present invention.
Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.
Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM is available in a variety of forms such as synchronous RAM (DRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.
The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A method of data migration, comprising the steps of:
acquiring a dump parameter;
creating a change record table;
creating multiple threads according to the unloading parameters and segmenting data to be migrated to obtain multiple segmented data;
establishing mapping between the fragment data and each single thread in the multiple threads, so that the corresponding fragment data is migrated to a target database by using each single thread, and data change generated in the migration process is recorded by using the change record table;
and responding to the end of the migration of the plurality of fragmented data, and changing the corresponding data in the target database according to the data change record in the change record table.
2. The method of claim 1, wherein the data to be migrated is fragmented according to the unloading parameter to obtain a plurality of fragmented data, further comprising:
determining the fragment size of each fragment data according to the unloading parameters;
determining a plurality of segment indexes by using the segment size;
and taking data between two continuous fragment indexes as the fragment data.
3. The method of claim 2, wherein determining a plurality of shard indices using the shard size further comprises:
and taking the main key or the joint main key at the position of integral multiple of the size of the fragment as the fragment index, and taking the first and the last main keys or the joint main keys as the fragment index.
4. The method of claim 1, wherein creating a change log table further comprises:
create add triggers, modify triggers, and delete triggers.
5. The method of claim 4, wherein the change record table is utilized to record data changes generated during the migration, further comprising:
and recording the parameter information corresponding to the data change generated in the migration process into the change record table by using the adding trigger, the modifying trigger and/or the deleting trigger.
6. The method of claim 1, wherein changing the corresponding data in the target database according to the data change record in the change record table further comprises:
and in response to the data change record in the change record table being deleted, deleting the corresponding data in the target database.
7. The method of claim 6, further comprising:
in response to the data change record in the change record table being added or modified, deleting the data corresponding to the target database;
and acquiring the changed data and updating the changed data into the target database.
8. A system for data migration, comprising:
an acquisition module configured to acquire a transfer parameter;
a creation module configured to create a change log table;
the fragmentation module is configured to create multiple threads according to the unloading parameters and fragment data to be migrated to obtain multiple pieces of fragmentation data;
the migration module is configured to establish mapping between the fragment data and each single thread in the multiple threads, so that each single thread is used for migrating the corresponding fragment data to a target database, and the change record table is used for recording data changes generated in the migration process;
a changing module configured to, in response to the completion of the migration of the plurality of fragmented data, change corresponding data in the target database according to a data change record in the change record table.
9. A computer device, comprising:
at least one processor; and
memory storing a computer program operable on the processor, wherein the processor executes the program to perform the steps of the method according to any of claims 1-7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010085526.XA CN111291023A (en) | 2020-02-09 | 2020-02-09 | Data migration method, system, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010085526.XA CN111291023A (en) | 2020-02-09 | 2020-02-09 | Data migration method, system, device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111291023A true CN111291023A (en) | 2020-06-16 |
Family
ID=71025558
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010085526.XA Pending CN111291023A (en) | 2020-02-09 | 2020-02-09 | Data migration method, system, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111291023A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111857890A (en) * | 2020-07-24 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN112380185A (en) * | 2020-10-30 | 2021-02-19 | 中国人寿保险股份有限公司 | Data dump method, equipment, electronic equipment and storage medium |
| CN112395360A (en) * | 2020-12-01 | 2021-02-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN112486966A (en) * | 2020-11-26 | 2021-03-12 | 中国国家铁路集团有限公司 | Expired data cleaning method and device and electronic equipment |
| CN113760861A (en) * | 2021-01-13 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data migration method and device |
| CN113778982A (en) * | 2021-03-09 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data migration method and device |
| CN113868188A (en) * | 2021-08-25 | 2021-12-31 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for modifying storage position of database file |
| CN114237519A (en) * | 2022-02-23 | 2022-03-25 | 苏州浪潮智能科技有限公司 | Method, device, equipment and medium for migrating object storage data |
| CN114528078A (en) * | 2022-01-29 | 2022-05-24 | 苏州浪潮智能科技有限公司 | Data migration method, device, equipment and medium for object storage |
| WO2023116428A1 (en) * | 2021-12-24 | 2023-06-29 | 中兴通讯股份有限公司 | Cross-database data migration comparison method and apparatus, and device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105718507A (en) * | 2016-01-06 | 2016-06-29 | 杭州数梦工场科技有限公司 | Data migration method and device |
| CN108874828A (en) * | 2017-05-12 | 2018-11-23 | 北京京东尚科信息技术有限公司 | The method and apparatus of Data Migration |
| CN110555012A (en) * | 2018-05-14 | 2019-12-10 | 杭州海康威视数字技术股份有限公司 | data migration method and device |
| CN110647579A (en) * | 2019-08-16 | 2020-01-03 | 北京百度网讯科技有限公司 | Data synchronization method and device, computer equipment and readable medium |
-
2020
- 2020-02-09 CN CN202010085526.XA patent/CN111291023A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105718507A (en) * | 2016-01-06 | 2016-06-29 | 杭州数梦工场科技有限公司 | Data migration method and device |
| CN108874828A (en) * | 2017-05-12 | 2018-11-23 | 北京京东尚科信息技术有限公司 | The method and apparatus of Data Migration |
| CN110555012A (en) * | 2018-05-14 | 2019-12-10 | 杭州海康威视数字技术股份有限公司 | data migration method and device |
| CN110647579A (en) * | 2019-08-16 | 2020-01-03 | 北京百度网讯科技有限公司 | Data synchronization method and device, computer equipment and readable medium |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111857890B (en) * | 2020-07-24 | 2021-08-31 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN111857890A (en) * | 2020-07-24 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN112380185A (en) * | 2020-10-30 | 2021-02-19 | 中国人寿保险股份有限公司 | Data dump method, equipment, electronic equipment and storage medium |
| CN112380185B (en) * | 2020-10-30 | 2024-05-28 | 中国人寿保险股份有限公司 | Data dump method, device, electronic device and storage medium |
| CN112486966A (en) * | 2020-11-26 | 2021-03-12 | 中国国家铁路集团有限公司 | Expired data cleaning method and device and electronic equipment |
| CN112395360B (en) * | 2020-12-01 | 2023-06-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN112395360A (en) * | 2020-12-01 | 2021-02-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN113760861A (en) * | 2021-01-13 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data migration method and device |
| CN113778982A (en) * | 2021-03-09 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data migration method and device |
| CN113868188A (en) * | 2021-08-25 | 2021-12-31 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for modifying storage position of database file |
| WO2023116428A1 (en) * | 2021-12-24 | 2023-06-29 | 中兴通讯股份有限公司 | Cross-database data migration comparison method and apparatus, and device and storage medium |
| CN114528078A (en) * | 2022-01-29 | 2022-05-24 | 苏州浪潮智能科技有限公司 | Data migration method, device, equipment and medium for object storage |
| CN114237519A (en) * | 2022-02-23 | 2022-03-25 | 苏州浪潮智能科技有限公司 | Method, device, equipment and medium for migrating object storage data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111291023A (en) | Data migration method, system, device and medium | |
| US8938430B2 (en) | Intelligent data archiving | |
| CN108804306B (en) | Method and system for automatic test system | |
| US11294958B2 (en) | Managing a distributed knowledge graph | |
| CN107766374B (en) | Optimization method and system for storage and reading of massive small files | |
| KR20160003682A (en) | Hydration and dehydration with placeholders | |
| US20070156778A1 (en) | File indexer | |
| US10983909B2 (en) | Trading off cache space and write amplification for Bε-trees | |
| CN116204575A (en) | Method, device, equipment and computer storage medium for importing data into database | |
| US8589454B2 (en) | Computer data file merging based on file metadata | |
| US20240061599A1 (en) | Method and system for processing file read-write service, device, and medium | |
| US8600990B2 (en) | Interacting methods of data extraction | |
| US7958083B2 (en) | Interacting methods of data summarization | |
| WO2024041022A1 (en) | Database table alteration method and apparatus, device and storage medium | |
| US10423580B2 (en) | Storage and compression of an aggregation file | |
| CN114237519A (en) | Method, device, equipment and medium for migrating object storage data | |
| CN112965939A (en) | File merging method, device and equipment | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| CN115080515A (en) | Block chain based system file sharing method and system | |
| CN109542860B (en) | Service data management method based on HDFS and terminal equipment | |
| CN111143463B (en) | Construction method and device of bank data warehouse based on topic model | |
| US20170255644A1 (en) | Systems and methods for automatically passing hints to a file system | |
| US10761940B2 (en) | Method, device and program product for reducing data recovery time of storage system | |
| CN113326288A (en) | Data processing method and device and electronic equipment | |
| US11556519B2 (en) | Ensuring integrity of records in a not only structured query language database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200616 |