CN113190534A - Database data migration method and device - Google Patents
Database data migration method and device Download PDFInfo
- Publication number
- CN113190534A CN113190534A CN202110576785.7A CN202110576785A CN113190534A CN 113190534 A CN113190534 A CN 113190534A CN 202110576785 A CN202110576785 A CN 202110576785A CN 113190534 A CN113190534 A CN 113190534A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- files
- file
- generated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 82
- 230000005012 migration Effects 0.000 title claims abstract description 82
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 41
- 230000015654 memory Effects 0.000 claims description 35
- 238000004590 computer program Methods 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000012546 transfer Methods 0.000 abstract description 11
- 238000012545 processing Methods 0.000 description 16
- 230000006870 function Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 13
- 238000004891 communication Methods 0.000 description 12
- 239000000872 buffer Substances 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000008676 import Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000003936 working memory Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Abstract
The invention provides a database data migration method and device, and relates to the technical field of big data. The method comprises the following steps: acquiring a pre-configured configuration file, wherein the configuration file comprises: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files; determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file; and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database. The invention can effectively improve the data migration efficiency and reduce the time consumption of the transfer.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a database data migration method and device.
Background
MySQL is a relational database management system that keeps data in different tables instead of putting all the data in one large repository, which increases speed and flexibility. The SQL language used by MySQL is the most common standardized language for accessing databases. MySQL software adopts a double-authorization policy and is divided into a community version and a business version, and generally MySQL is selected as a website database for development of small and medium-sized websites due to the characteristics of small volume, high speed, low total ownership cost and particularly open source codes.
The Sharding-Sphere is an ecological circle formed by a set of open-source distributed database middleware solutions, can provide standardized data fragmentation, read-write separation, flexible transaction and data management functions, and is applicable to various diversified application scenes such as Java isomorphism, heterogeneous languages, containers, cloud protogenesis and the like. Two connection modes of the execution engine of the sharing-sphere to the database are provided: a memory-bound mode (i.e., a scenario where streaming queries are to be used), and a connection-bound mode, where for each table query a database connection needs to be created. One MySQL database connection in the streaming query can only serve one ResultSet object at the same time, and if the ResultSet object is not closed, the use of other queries on the database connection is influenced.
Aiming at the existing MySQL streaming query, when data is operated from databases such as Mysql, Oracle and H2 and the like or data is read from an original system and migrated to a new data system, if the operation data volume is too large, the problems of overlarge Memory occupation and low performance are often caused, even the problem of Out of Memory (OOM) is caused, and the serious consequence of service breakdown is caused.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a database data migration method and device, which can effectively improve the data migration efficiency.
In order to solve the technical problems, the invention provides the following technical scheme:
in a first aspect, the present invention provides a database data migration method, including:
acquiring a pre-configured configuration file, wherein the configuration file comprises: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files;
determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file;
and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database.
Further, after the determining the original database according to the source database address, the method further includes:
scanning data in the original database based on a scanning thread, and determining target data which accords with a preset migration rule;
correspondingly, the step of distributing the data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files comprises the following steps:
and distributing the target data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files.
Wherein the preset migration rule comprises: the identification field of the data is to be processed or inquired.
Wherein, the distributing the data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files comprises:
and determining a hash value corresponding to the data in the original data, and distributing the data in the original data to a plurality of files based on the hash value to obtain a plurality of exported and generated files.
Wherein, the determining the hash value corresponding to the data in the raw data includes:
and determining any field of the data in the original database, and performing hash calculation according to the determined field to obtain a corresponding hash value.
Wherein the file generated by export is a CSV file.
Wherein migrating the plurality of export-generated files to the target database comprises:
filling the data in the file generated by export into a preset report template file to generate a target report file;
and storing the target report file into a target database.
In a second aspect, the present invention provides a database data migration apparatus, including:
a reading module, configured to obtain a pre-configured configuration file, where the configuration file includes: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files;
the migration module is used for determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file;
and the storage module is used for determining a target database according to the target database address and transferring a plurality of exported and generated files to the target database.
Further, still include:
the scanning module is used for scanning the data in the original database based on a scanning thread and determining target data which accords with a preset migration rule;
correspondingly, the migration module comprises:
and the migration submodule is used for distributing the target data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files.
Wherein the migration module comprises:
and the migration unit is used for determining a hash value corresponding to the data in the original data, and distributing the data in the original data to a plurality of files based on the hash value to obtain a plurality of exported and generated files.
Wherein the storage module comprises:
the filling unit is used for filling the data in the file generated by export into a preset report template file to generate a target report file;
and the storage unit is used for storing the target report file into a target database.
In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the database data migration method when executing the program.
In a fourth aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the database data migration method described herein.
As can be seen from the foregoing technical solutions, the present invention provides a method and an apparatus for migrating database data, where a pre-configured configuration file is obtained, where the configuration file includes: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files; determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file; and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database. The data migration efficiency can be effectively improved, and the time consumed by transfer is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a first flowchart of a database data migration method according to an embodiment of the present invention.
Fig. 2 is a second flowchart of the database data migration method according to the embodiment of the present invention.
FIG. 3 is a system flow diagram of a database data migration method according to an embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a database data migration apparatus according to an embodiment of the present invention.
Fig. 5 is a schematic structural diagram of a second database data migration apparatus according to an embodiment of the present invention.
Fig. 6 is a schematic structural diagram of an electronic device in an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention provides an embodiment of a database data migration method, which specifically comprises the following contents in reference to fig. 1:
s101: acquiring a pre-configured configuration file, wherein the configuration file comprises: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files;
in this step, the pre-configured configuration file may further include: account number and password of source database or target data, derived serial-parallel mark and imported serial-parallel mark.
The algorithm for distributing the data to different files is used for carrying out hash calculation according to a certain field in the database, and different values are exported to different files. Furthermore, data can be exported to a plurality of files, and when the data in each file is loaded into the memory for processing, the problem of memory leakage can be avoided.
The serial-parallel flag is used to perform serial processing or parallel processing only on a plurality of exported files. And the parallel file export is supported, and the efficiency can be greatly improved.
S102: determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file;
in this step, a hash value corresponding to data in the raw data is determined, and the data in the raw data is distributed to a plurality of files based on the hash value to obtain a plurality of export-generated files.
Determining a hash value corresponding to data in the raw data includes:
and determining any field of the data in the original database, and performing hash calculation according to the determined field to obtain a corresponding hash value.
In this embodiment, the file generated by export is a CSV file, and correspondingly, the step of distributing the data in the raw data to multiple files based on the hash value to obtain multiple files generated by export includes:
the specific file of the multiple files of the data in the original data is determined based on the hash value, and the multiple files can be divided according to the range of the numerical value at a certain position in the hash value and distributed according to the specific data at the position. After the data in the original data is allocated to a specific file, writing the data in the original data into the file to obtain a file corresponding to the file and generated by export, specifically including:
and taking the data in the original data as a data intermediate file. All the source data in the CSV file are already included in the data intermediate file, and the data intermediate file is reproduced as a CSV file of table data.
And respectively acquiring each field content corresponding to each field label from the data intermediate file. The data intermediate file stores the target source data acquired from each source file, so that the data intermediate file stores the field contents corresponding to each field tag in the CSV file, and the corresponding field contents can be acquired from the data intermediate file according to the configuration sequence of each field tag in the configuration information.
And combining each field label and each corresponding field content into table data to obtain the target CSV file. And after all the field contents corresponding to all the field labels in the CSV file are acquired from the data intermediate file, respectively taking all the field labels, namely all the corresponding field contents, as table data, and storing the table data into a preset EXCEL table to obtain a target CSV file, wherein the CSV file can be opened by an EXCEL program.
In this step, the capacity of each export generated file is smaller than the maximum capacity of the export generated file, so that the export generated files are limited to avoid the occurrence of extra large export generated files which are larger than other export generated files. The data migration efficiency is further improved, and the time consumed by transfer is reduced.
S103: and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database.
In the step, filling the data in the file generated by export into a preset report template file to generate a target report file; and storing the target report file into a target database.
Specifically, the exported data in the CSV file is imported into a first designated table of the report template file, and is listed according to macro instructions defined in the report template file, for example, the data in the CSV file is listed by comma delimiters; then, mapping the data in the first specified table to a second specified table according to the data mapping and the embedded calculation formula defined in the report template file; and finally, filling defined data into the second specified table to generate a target report file, and finally storing the target report file into a target database.
As can be seen from the above description, in the database data migration method provided in the embodiment of the present invention, by obtaining a pre-configured configuration file, the configuration file includes: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files; determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file; and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database. The data migration efficiency can be effectively improved, and the time consumed by transfer is reduced.
In an embodiment of the present invention, referring to fig. 2, after step S102 of the database data migration method, the method further includes:
s104: scanning data in the original database based on a scanning thread, and determining target data which accords with a preset migration rule;
correspondingly, the step of distributing the data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files comprises the following steps:
and distributing the target data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files.
In this embodiment, by setting the scanning thread, the target data meeting the preset migration rule may be automatically scanned from the original database based on the scanning thread, and the target data is added to the transmission queue, so that the target data is migrated subsequently.
It should be noted that the preset migration rule includes: the identification field of the data is to be processed or inquired. Namely, the data of which the identification field is to be processed or inquired is determined as target data.
As can be seen from the above description, the data migration method provided in this embodiment can effectively improve the data migration efficiency, and reduce the time consumed by the transfer.
To further illustrate the present solution, the present invention provides an embodiment of a database data migration method, which is shown in fig. 3 and specifically includes the following contents:
1. reading a configuration file:
the configuration file comprises a source database address, a target database address, a database account password, logic for exporting data, an algorithm for distributing the data to different files, an exported serial-parallel mark, an imported serial-parallel mark and the maximum capacity of a exported and generated file.
The data are distributed into different files, hash calculation can be carried out according to a certain field, and different values are exported into different files. When the data in each file is loaded into the memory for processing, the problem of memory leakage can be avoided.
The serial-parallel flag is used to perform serial processing or parallel processing only on a plurality of exported files. And the parallel file export is supported, and the efficiency can be greatly improved.
2. Adaptation to different types of databases:
according to the sql language characteristics of the database such as Mysql/H2, the adapter is automatically adapted to different databases. Export from one database into a file is supported, and after the file is processed, the file is imported into another database. For example, data is exported from Mysql, and after processing, the file is imported into H2.
3. Exporting data to a file:
according to configured rules, the data are exported from the database to a plurality of csv files concurrently to serve as data transfer, and the data are transmitted to an application side in a file form to be processed.
4. Processing the data file by the application side:
the maximum value which can be borne by the export file can be configured in the configuration, a large amount of data is exported to a plurality of data files, the whole file is conveniently loaded to the memory by the application side to process and treat the data, and the problem of memory leakage cannot occur.
5. The application side generates the processed data into a data file:
6. importing the data file into a database:
and adapting to the corresponding database according to the configuration rule and the adapter and importing the data into the database.
Based on the technical scheme, the data to be processed or inquired in the data source is concurrently exported into a plurality of csv files with specific formats by configuring the sql statement, the csv files are used as data transfer files, and then the data of the transfer files are read from the application system side in a file stream mode. And finally, the acceleration of large data transmission is realized. Because the file medium is isolated at the bottom layer, the existence of a non-sensing file system is realized at a service layer, the method can be compatible with the traditional result set operation mode of jdbc, and is convenient for developers to use. Compared with the traditional database data operation, the performance of data in a file import and export form is greatly improved, and the file system of the system supports concurrent import and export of multiple files, so that the time consumption of file dumping is reduced, and the performance is further improved; the whole process data is used as a data read-write interface in a streaming mode, and is used as a data storage medium in a file mode, so that java working memory does not need to be occupied, and pressure cannot be applied to jvm.
An embodiment of the present invention provides a specific implementation manner of a database data migration apparatus capable of implementing all contents in the database data migration method, and referring to fig. 4, the database data migration apparatus specifically includes the following contents:
a reading module 10, configured to obtain a pre-configured configuration file, where the configuration file includes: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files;
the migration module 20 is configured to determine an original database according to the source database address, and allocate data in the original database to multiple files according to an algorithm that the data is allocated to different files to obtain multiple export-generated files; wherein the size of each export generated file is less than the maximum size of the export generated file;
and the storage module 30 is configured to determine a target database according to the target database address, and migrate a plurality of exported and generated files to the target database.
Wherein the migration module comprises:
and the migration unit is used for determining a hash value corresponding to the data in the original data, and distributing the data in the original data to a plurality of files based on the hash value to obtain a plurality of exported and generated files.
Wherein the storage module comprises:
the filling unit is used for filling the data in the file generated by export into a preset report template file to generate a target report file;
and the storage unit is used for storing the target report file into a target database.
In an embodiment of the present invention, referring to fig. 5, the database data migration apparatus further includes:
the scanning module 40 is configured to scan data in the original database based on a scanning thread, and determine target data that meets a preset migration rule;
correspondingly, the migration module comprises:
and the migration submodule is used for distributing the target data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files.
The embodiment of the database data migration apparatus provided in the present invention may be specifically configured to execute the processing procedure of the embodiment of the database data migration method in the foregoing embodiment, and the functions of the embodiment are not described herein again, and refer to the detailed description of the embodiment of the method.
As can be seen from the above description, in the database data migration apparatus provided in the embodiment of the present invention, by obtaining a pre-configured configuration file, the configuration file includes: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files; determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file; and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database. The data migration efficiency can be effectively improved, and the time consumed by transfer is reduced.
The application provides an embodiment of an electronic device for implementing all or part of contents in the database data migration method, where the electronic device specifically includes the following contents:
a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between related devices; the electronic device may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the electronic device may be implemented with reference to the embodiment for implementing the database data migration method and the embodiment for implementing the database data migration apparatus in the embodiments, and the contents thereof are incorporated herein, and repeated details are not repeated herein.
Fig. 6 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 6, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this FIG. 6 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one embodiment, the database data migration functionality may be integrated into central processor 9100. The central processor 9100 may be configured to control as follows:
acquiring a pre-configured configuration file, wherein the configuration file comprises: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files; determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file; and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database.
As can be seen from the above description, an electronic device provided in an embodiment of the present application obtains a pre-configured configuration file, where the configuration file includes: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files; determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file; and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database. The data migration efficiency can be effectively improved, and the time consumed by transfer is reduced.
In another embodiment, the database data migration apparatus may be configured separately from the central processor 9100, for example, the database data migration apparatus may be configured as a chip connected to the central processor 9100, and the database data migration function is realized under the control of the central processor.
As shown in fig. 6, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 6; further, the electronic device 9600 may further include components not shown in fig. 6, which may be referred to in the art.
As shown in fig. 6, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
An embodiment of the present invention further provides a computer-readable storage medium capable of implementing all the steps in the database data migration method in the foregoing embodiment, where the computer-readable storage medium stores thereon a computer program, and when the computer program is executed by a processor, the computer program implements all the steps in the database data migration method in the foregoing embodiment, for example, when the processor executes the computer program, the processor implements the following steps:
acquiring a pre-configured configuration file, wherein the configuration file comprises: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files; determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file; and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database.
As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present invention obtains a pre-configured configuration file, where the configuration file includes: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files; determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file; and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database. The data migration efficiency can be effectively improved, and the time consumed by transfer is reduced.
Although the present invention provides method steps as described in the examples or flowcharts, more or fewer steps may be included based on routine or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an actual apparatus or client product executes, it may execute sequentially or in parallel (e.g., in the context of parallel processors or multi-threaded processing) according to the embodiments or methods shown in the figures.
As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, apparatus (system) or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention is not limited to any single aspect, nor is it limited to any single embodiment, nor is it limited to any combination and/or permutation of these aspects and/or embodiments. Moreover, each aspect and/or embodiment of the present invention may be utilized alone or in combination with one or more other aspects and/or embodiments thereof.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present invention, and they should be construed as being included in the following claims and description.
Claims (13)
1. A database data migration method is characterized by comprising the following steps:
acquiring a pre-configured configuration file, wherein the configuration file comprises: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files;
determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to an algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file;
and determining a target database according to the target database address, and migrating a plurality of exported and generated files to the target database.
2. The database data migration method according to claim 1, further comprising, after determining the original database according to the source database address:
scanning data in the original database based on a scanning thread, and determining target data which accords with a preset migration rule;
correspondingly, the step of distributing the data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files comprises the following steps:
and distributing the target data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files.
3. The database data migration method according to claim 2, wherein the preset migration rule includes: the identification field of the data is to be processed or inquired.
4. The database data migration method according to claim 1, wherein the step of distributing the data in the original database to a plurality of files according to an algorithm for distributing the data to different files to obtain a plurality of export-generated files comprises:
and determining a hash value corresponding to the data in the original data, and distributing the data in the original data to a plurality of files based on the hash value to obtain a plurality of exported and generated files.
5. The database data migration method according to claim 4, wherein the determining the hash value corresponding to the data in the original data includes:
and determining any field of the data in the original database, and performing hash calculation according to the determined field to obtain a corresponding hash value.
6. The database data migration method according to claim 4, wherein the file generated by export is a CSV file.
7. The database data migration method according to claim 1, wherein the migrating the files generated by the plurality of exports to the target database comprises:
filling the data in the file generated by export into a preset report template file to generate a target report file;
and storing the target report file into a target database.
8. A database data migration apparatus, comprising:
a reading module, configured to obtain a pre-configured configuration file, where the configuration file includes: source database address, target database address, algorithm for distributing data to different files, and maximum capacity for exporting generated files;
the migration module is used for determining an original database according to the source database address, and distributing the data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files; wherein the size of each export generated file is less than the maximum size of the export generated file;
and the storage module is used for determining a target database according to the target database address and transferring a plurality of exported and generated files to the target database.
9. The database data migration apparatus according to claim 8, further comprising:
the scanning module is used for scanning the data in the original database based on a scanning thread and determining target data which accords with a preset migration rule;
correspondingly, the migration module comprises:
and the migration submodule is used for distributing the target data in the original database to a plurality of files according to the algorithm of distributing the data to different files to obtain a plurality of exported and generated files.
10. The database data migration apparatus according to claim 8, wherein said migration module comprises:
and the migration unit is used for determining a hash value corresponding to the data in the original data, and distributing the data in the original data to a plurality of files based on the hash value to obtain a plurality of exported and generated files.
11. The database data migration apparatus according to claim 8, wherein said storage module comprises:
the filling unit is used for filling the data in the file generated by export into a preset report template file to generate a target report file;
and the storage unit is used for storing the target report file into a target database.
12. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the database data migration method according to any one of claims 1 to 7 are implemented when the program is executed by the processor.
13. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the database data migration method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110576785.7A CN113190534A (en) | 2021-05-26 | 2021-05-26 | Database data migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110576785.7A CN113190534A (en) | 2021-05-26 | 2021-05-26 | Database data migration method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113190534A true CN113190534A (en) | 2021-07-30 |
Family
ID=76985058
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110576785.7A Pending CN113190534A (en) | 2021-05-26 | 2021-05-26 | Database data migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113190534A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609226A (en) * | 2021-08-09 | 2021-11-05 | 平安国际智慧城市科技股份有限公司 | Data export method and device, computer equipment and storage medium |
| CN114077602A (en) * | 2022-01-13 | 2022-02-22 | 中兴通讯股份有限公司 | Data migration method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105205117A (en) * | 2015-09-09 | 2015-12-30 | 郑州悉知信息科技股份有限公司 | Data table migrating method and device |
| WO2019219010A1 (en) * | 2018-05-14 | 2019-11-21 | 杭州海康威视数字技术股份有限公司 | Data migration method and device and computer readable storage medium |
| CN110674108A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data processing method and device |
| CN111367895A (en) * | 2020-03-31 | 2020-07-03 | 中国建设银行股份有限公司 | Data migration method and device |
| CN111666266A (en) * | 2020-06-09 | 2020-09-15 | 金蝶软件（中国）有限公司 | Data migration method and related equipment |
| CN112015716A (en) * | 2020-08-04 | 2020-12-01 | 北京人大金仓信息技术股份有限公司 | Database data migration method, device, medium and electronic equipment |
-
2021
- 2021-05-26 CN CN202110576785.7A patent/CN113190534A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105205117A (en) * | 2015-09-09 | 2015-12-30 | 郑州悉知信息科技股份有限公司 | Data table migrating method and device |
| WO2019219010A1 (en) * | 2018-05-14 | 2019-11-21 | 杭州海康威视数字技术股份有限公司 | Data migration method and device and computer readable storage medium |
| CN110674108A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data processing method and device |
| CN111367895A (en) * | 2020-03-31 | 2020-07-03 | 中国建设银行股份有限公司 | Data migration method and device |
| CN111666266A (en) * | 2020-06-09 | 2020-09-15 | 金蝶软件（中国）有限公司 | Data migration method and related equipment |
| CN112015716A (en) * | 2020-08-04 | 2020-12-01 | 北京人大金仓信息技术股份有限公司 | Database data migration method, device, medium and electronic equipment |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609226A (en) * | 2021-08-09 | 2021-11-05 | 平安国际智慧城市科技股份有限公司 | Data export method and device, computer equipment and storage medium |
| CN114077602A (en) * | 2022-01-13 | 2022-02-22 | 中兴通讯股份有限公司 | Data migration method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20180107725A1 (en) | Data Storage Method and Apparatus, and Data Read Method and Apparatus | |
| KR101994021B1 (en) | File manipulation method and apparatus | |
| US10372383B2 (en) | Providing secure access to data in mobile devices | |
| US10701154B2 (en) | Sharding over multi-link data channels | |
| CN104750740A (en) | Data renewing method and device | |
| CN113190534A (en) | Database data migration method and device | |
| CN105824849A (en) | Data import method and adapter | |
| KR101856486B1 (en) | Method for bi-direction calling between open system and mainframe system | |
| CN104268229A (en) | Resource obtaining method and device based on multi-process browser | |
| CN110851474A (en) | Data query method, database middleware, data query device and storage medium | |
| CN110928935A (en) | Data access command processing method, device and system | |
| CN109857553B (en) | Memory management method and device | |
| CN109597825B (en) | Rule engine calling method, device, equipment and computer readable storage medium | |
| CN104156316A (en) | Hadoop cluster batch processing method and system | |
| CN107357853B (en) | Method and device for operating REDIS console and computer system | |
| CN104572638A (en) | Data reading and writing method and device | |
| CN108228842B (en) | Docker mirror image library file storage method, terminal, device and storage medium | |
| CN113434489B (en) | Real-time database online capacity expansion method, system, equipment and storage medium | |
| CN114331745B (en) | Data processing method, system, readable storage medium and electronic device | |
| CN109614089A (en) | Automatic generation method, device, equipment and the storage medium of data access code | |
| EP4187877A1 (en) | Method and apparatus for establishing communication connection, and device, and computer readable storage medium | |
| US20170364293A1 (en) | Method and apparatus for data processing | |
| CN115509972A (en) | Method for realizing DMA high-efficiency message receiving and transmitting and application | |
| CN112597113B (en) | Data rapid reading method and device, electronic equipment and storage medium | |
| CN112596949B (en) | High-efficiency SSD (solid State disk) deleted data recovery method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |