CN110674113A - One-key migration method and device for data, electronic equipment and storage medium - Google Patents
One-key migration method and device for data, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN110674113A CN110674113A CN201910906044.3A CN201910906044A CN110674113A CN 110674113 A CN110674113 A CN 110674113A CN 201910906044 A CN201910906044 A CN 201910906044A CN 110674113 A CN110674113 A CN 110674113A
- Authority
- CN
- China
- Prior art keywords
- data
- relational database
- migration
- definition file
- field definition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The embodiment of the invention provides a one-key migration method and device of data, electronic equipment and a storage medium, wherein the method comprises the following steps: determining a first field definition file of data to be migrated in a relational database, wherein the first field definition file stores fields of the data to be migrated and a first data type defined by the fields; determining a second field definition file supported by a non-relational database according to the first field definition file and a mapping relation, wherein the mapping relation is the mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the second data type corresponding to the field and the field; and migrating the data to be migrated in the relational database to the non-relational database in a one-key mode according to the second field definition file. The invention has the beneficial effects that: the one-key migration of data from a non-relational database to a relational database can be realized.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for one-key data migration, an electronic device, and a storage medium.
Background
The relational database is a database for organizing languages by adopting a relational model, for example, the Mysql database is an open-source relational database management system, is used for storing, inquiring, updating and managing a relational database system, has the advantages of small volume, high speed, high flexibility and the like, and is widely used in the Internet industry; the Non-relational Database is an open-source nonsql (Non-Structured Query Language) Database, such as a Mongodb (mango Database), which is a Database based on a document model and supporting index, copy, transaction, and fragment horizontal expansion, the number of objects of the Non-relational Database is not limited, the execution speed of the operation of the Non-relational Database such as the Mongodb can be increased by a cluster, when the Database becomes larger, a server can be added to the cluster to solve the performance problem, and the method is very suitable for storage and analysis of large data, and therefore, the method has the characteristics of high performance, easiness in deployment, easiness in use and the like.
Relational databases such as the Mysql database are used as relational databases, data models have strict row-column formats, non-relational databases such as the Mongodb database are used as non-relational databases based on document models, and data models are flexible. With the advent of the big data age, data stored using relational databases, such as the Mysql database, often needs to be migrated to non-relational databases, such as the Mongodb for use and analysis.
The existing method for migrating relational database, such as Mysql database, to non-relational database, such as Mongodb, is a manual operation, and exporting data of single table or multi-to-be-migrated from relational database, such as Mysql database, to txt (text format) or csv (Comma-Separated Values, character separation value, a file format) format, and then importing the data types into mongoimport through import tool provided by mongoidb after manual comparison.
The relational database and the non-relational database are two completely different databases, although the non-relational database has no strict data model, when an importing tool of the non-relational database reads a data file, all data are treated as character types by default, and only the field types can be manually determined and imported, so that the time is long, and errors are possible.
Disclosure of Invention
The embodiment of the invention provides a one-key data migration method, a one-key data migration device, electronic equipment and a storage medium, which are used for overcoming the defect that the efficiency of migrating data from a relational database to a non-relational database is low in the prior art and realizing the automatic and large-batch one-key data migration from the relational database to the non-relational database.
To solve the foregoing technical problem, in a first aspect, an embodiment of the present invention provides a method for migrating data by one key, where the method includes:
determining a first field definition file of data to be migrated in a relational database, wherein the first field definition file stores fields of the data to be migrated and a first data type defined by the fields;
determining a second field definition file supported by a non-relational database according to the first field definition file and a mapping relation, wherein the mapping relation is the mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the second data type corresponding to the field and the field;
and migrating the data to be migrated in the relational database to the non-relational database in a one-key mode according to the second field definition file.
In combination with another aspect of the embodiments of the present invention, the method further includes:
acquiring first parameter information of the relational database, second parameter information of the non-relational database and a migration requirement type;
the migrating the data to be migrated in the relational database to the non-relational database in a one-touch manner according to the second field definition file includes:
determining a derivation command by combining the first parameter information of the relational database and the migration requirement type, and deriving the data to be migrated from the relational database according to the derivation command;
and generating an import command according to the second parameter information of the non-relational database and the second field definition file, and migrating the data to be migrated to the non-relational database by one key according to the import command.
In combination with another aspect of the embodiment of the present invention, the determining, by combining the first parameter information of the relational database and the migration requirement type, an export command includes:
when the migration requirement type is migration of a library, determining that the export command is a first export command;
when the migration requirement type is the migration of a table, determining that the export command is a second export command;
and when the migration requirement type is migration of a library and a partial table, determining that the export command is a third export command.
In combination with another aspect of the embodiment of the present invention, the acquiring first parameter information of the relational database, second parameter information of the non-relational database, and a migration requirement type includes:
the obtaining of the first parameter information of the relational database at least comprises obtaining a port number and a network address of the relational database;
the obtaining of the second parameter information of the non-relational database at least comprises obtaining a port number and a network address of the non-relational database.
With reference to another aspect of the embodiment of the present invention, an import log is generated when the data to be migrated is imported into the non-relational database;
carrying out keyword retrieval on the import log, and determining the import log with abnormality;
and outputting the import log with the exception.
In another aspect of the embodiment of the present invention, before determining the first field definition file of the data to be migrated in the relational database, the method further includes:
determining a directory derived from the data to be migrated and the first field definition file; and/or
Determining a directory of the second field definition file.
In a second aspect, an embodiment of the present invention further provides a device for migrating data by one key, where the device includes:
the system comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining a first field definition file of data to be migrated in a relational database, and the first field definition file stores fields of the data to be migrated and a first data type defined by the fields;
a second determining module, configured to determine, according to the first field definition file determined by the first determining module and a mapping relationship, a second field definition file supported by a non-relational database, where the mapping relationship is a mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the second data type corresponding to the field and the field;
and the one-key migration module is used for performing one-key migration on the data to be migrated in the relational database to the non-relational database according to the second field definition file determined by the second determination module.
In a third aspect, the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps of the one-key migration method of the data.
In combination with another aspect of the embodiments of the present invention, the embodiments of the present invention further provide a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the one-key migration method of the above data.
According to the one-key migration method, the one-key migration device, the electronic equipment and the storage medium of the data, the first field definition file and the data to be migrated are generated when the data of the relational database are exported, and the second field definition file is generated by combining the mapping between the data type of the relational database and the data type of the non-relational database, so that columns in the data table to be migrated of the relational database can be converted according to the second field definition file during migration, the full-automatic one-key migration work is realized, and the migration efficiency is greatly improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a flow chart of a data one-key migration method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a data one-key migration method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a one-key migration apparatus for data according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention relates to a one-key data migration method, a one-key data migration device, electronic equipment and a storage medium, which are mainly applied to a data migration scene from a relational database to a non-relational database, in particular to a scene from a Mysql database to a Mongobb database, and the basic idea is as follows: firstly, parameters such as migration requirements, port numbers and network addresses of a relational database and a non-relational database, file directory and the like are confirmed, then data of the relational database are exported to generate a first field definition file and data to be migrated, and a second field definition file supported by the Mongodb database is generated by combining mapping of data types of the Mysql database and the Mongodb database, so that the data to be migrated can be rapidly migrated to the non-relational database for storage, and migration efficiency is greatly improved.
The embodiment of the present invention is applicable to a case where an intelligent terminal with a one-key migration module performs one-key migration of data, where the method may be executed by a one-key migration apparatus, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in an intelligent terminal, or may be controlled by a central control module of the intelligent terminal, as shown in fig. 1, which is a flowchart of a one-key migration method of data according to an embodiment of the present invention, and the method in the embodiment of the present invention is applicable to data migration from a relational database to a non-relational database, and specifically includes the following steps:
in step 110, a first field definition file of data to be migrated in a relational database is determined, where the first field definition file stores fields of the data to be migrated and a first data type defined by the fields.
The relational database in the embodiment of the present invention may be a Mysql database, which generally adopts a table storage manner, where the table is stored in a row and column manner, and stores data in such a row and column structured manner, each field of each data table is predefined, and then the data is stored according to the structure of the data table, the non-relational database can be a Mongolb database, the method generally adopts a storage mode of a document, data can be stored in a Json format and a key value pair mode, however, in a certain implementation scenario of the embodiment of the present invention, when the SQL Server database belonging to the relational database can provide a corresponding export tool, and a non-relational database such as a Hive database can provide a corresponding import tool, and when the same or similar configuration can be made, the method of the embodiment of the invention can still perform one-key batch data migration.
In the embodiment of the invention, the example of exporting data from the Mysql database to the Mongodb database is taken as an example for explanation.
The first field definition file stores the field of the data to be migrated and a first data type defined by the field, for the storage form of the table in the Mysql database, the column of the table is the field, the data type defined by the field is the data type contained in the Mysql database, such as char/varchar/double/float, etc., and the field is an identifier of each column in a two-dimensional table with a relational model as a data structure, which may be composed of letters, numbers, underlines, Chinese characters, etc., and generally has no special symbols.
In step 120, a second field definition file supported by the non-relational database is determined according to the first field definition file and a mapping relationship, where the mapping relationship is a mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the field and the second data type corresponding to the field.
The data types in the relational database and the data types of the non-relational database include the following table 1:
TABLE 1
The second column of table 1 is the first data type of Mysql database, the third column of table 1 is the second data type of Mongodb database, and the one-to-one correspondence of the data types between the second column and the third column forms the mapping relationship between the data types of the two databases.
When a second field definition file supported by a non-relational database, such as a number representation field in a first column in table 1, can be determined through the first field definition file and the mapping relationship, a combination between the first column and the third column at this time is the second field definition file, and the second field definition file stores the field and a second data type corresponding to the field, as shown in table 2 below:
TABLE 2
The method and the device for establishing the Mysql database and the second field definition file supported by the Mongodb database according to the first field definition file can be represented by field _ file, and the first data type of the Mysql database and the second data type of the Mongodb database are mapped one by one to generate a file name match.f, and the file name match.f is stored in a text file under an export directory, so that the first field definition file exported from the Mysql database is compared with the match.f file, and the second field definition file supported by mongoimmommommommommommommonio is further generated through conversion.
In step 130, the data to be migrated in the relational database is migrated to the non-relational database in a one-key manner according to the second field definition file.
And migrating the data to be migrated of the relational database according to the second field definition file, specifically, generating an import command according to the data file to be migrated generated by separating export data of the relational database, and executing a Mongoimport tool of the non-relational database to import the data file to be migrated to a Mongodb of the non-relational database for storage.
In this step, before determining the first field definition file of the data to be migrated in the relational database, the one-key migration method of the data further includes: and acquiring first parameter information of the relational database, second parameter information of the non-relational database and a migration requirement type.
The first parameter information of the relational database, such as the Mysql database, may be a Mysql user name, a Mysql password, a Mysql host IP address, a Mysql port number, and the like, and at least the Mysql host IP address and the Mysql port number may be adopted according to different specific application environments to ensure a data migration source address, and a content included in the first parameter information may be changed according to an actual requirement.
The second parameter information of the non-relational database, such as a Mongodb database, may be a Mongodb user name, a Monodb password, a Monodb host IP address, a Monodb port number, and the like, and at least the Monodb host IP address and the Monodb port number may be adopted according to different specific application environments to ensure a data migration target address, and the content included in the second parameter information may be changed according to actual requirements.
The migration requirement type at least comprises one of migration of a library, migration of a table and migration of the library and partial tables, the migration of the library comprises migration of a database, migration of a plurality of databases, migration of all databases and the like, the migration of the table comprises migration of a table of a certain database, migration of a plurality of tables of a plurality of databases and the like, and the migration of the library and partial tables comprises migration of partial tables of one library and other libraries, migration of partial tables of a plurality of libraries and other libraries and the like.
The export command is determined based on the migration requirement type, and since there may be multiple types of the migration requirement type according to different actual application scenarios, the embodiment of the present invention is described in only three types:
exporting all other databases which are not built-in databases in the first and Mysql databases, wherein an information _ schema.SCHEMTA table of the Mysql needs to be inquired at the moment, other service database names except built-in databases such as the information _ schema and the Mysql are excluded, and a first export command is generated;
secondly, when a single-library or multi-library full table of a non-built-in library in the Mysql database is exported, a second export command is directly generated after the table name of the full table is subjected to text processing;
thirdly, when a single-library or multi-library partial table of a non-built-in library in the Mysql database is exported, the information _ schema.TABLES table of the Mysql is inquired, the corresponding relation between the library and the table is obtained, and then a third export command is generated in batches.
The difference of the migration requirement types directly determines the difference of the data export commands to be migrated, and the following description will make a more detailed description on the above three types of migration, which is not described herein again.
According to the embodiment of the invention, the first field definition file and the data to be migrated are generated when the data of the relational database are exported, and the second field definition file is generated by combining the mapping of the data type of the relational database and the data type of the non-relational database, so that the columns in the data table to be migrated of the relational database can be converted according to the second field definition file during migration, the full-automatic one-key migration work is realized, and the migration efficiency is greatly improved.
In a specific implementation manner of the embodiment of the present invention, taking migrating data from a relational database Mysql database to a non-relational database Mongodb database as an example, the Mysql database includes an export tool Mysql, and has parameter information such as Mysql user name, password, network address, and port number, and similarly, the Mongodb database includes an import tool mongoimprort, and has parameter information such as mongoidb user name, password, network address, and port number, and adds a specific migration requirement and a second field definition file to determine a migration program.
Before data migration, a configuration file config.f is created first, the configuration file config.f includes parameter information of the two heterogeneous databases, and corresponding parameter items can be modified before migration is performed each time.
When the Mysql database exports data through an exporting tool Mysql stub, the exported data is divided into a first field definition file and a txt file or a csv format file (to-be-migrated data), client tools of the Mysql database and the Mongobb database are automatically called in the migration process, and meanwhile codes (resolvers) of the tools are used for resolving and converting so as to achieve the purpose of one-key migration.
Reading the configuration file when the migration program is started, reading the file by the migration file to obtain a corresponding parameter value, wherein specific parameter information is as follows:
MIG _ DB _ name1, DB _ name2, DB _ name — if MIG _ MODE is 1, the parameter is no longer valid, and if MIG _ MODE is 2 or 3, the parameter indicates the library name that needs to be migrated;
the MIG _ TAB ═ db _ name [% ], db _ name.tab _ name [% ] 1, db _ name.table _ name [% ] represents all tables of a certain database to be migrated, others represent table names to be migrated, and the query of the table names supports% wildcards.
After the startup program reads the parameter information, a export command is generated according to a specific migration requirement type and data export is performed, and the process includes the following steps:
when the whole library export is needed, inquiring an information _ schema of Mysql, and SCHEMTA table, excluding other service database names except built-in databases such as the information _ schema and the Mysql, and generating export commands in batches; when the requirement is single-library or multi-library full-table export, directly generating an export script after text processing is carried out according to the configuration file of the first step; when the requirement is the export of a single-library or multi-library partial table, an information _ schema.TABLES table of Mysql is inquired, the corresponding relation between the libraries and the tables is obtained, and then an export command is generated in batches and the export is executed.
Specifically, when full library export is required:
the information _ schema of the Mysql is inquired, other service database names except built-in databases such as the information _ schema and the Mysql are excluded, and a export command is generated in batches.
The implementation commands are as follows:
specifically, when a single bank or multi-bank full table export is required:
when the migration requirement is that all tables of a single library or a plurality of data are exported, only the MIG _ DB parameter of the configuration file needs to be read, and the batch export command is automatically generated and then executed, so that the command is realized as follows:
specifically, when single bank or multi-bank partial table exports are required:
when the requirement is the export of a single-library or multi-library partial table, inquiring an information _ schema.TABLES table of Mysql, acquiring the corresponding relation between the libraries and the tables, then generating export commands in batches and executing the export, wherein the realized program codes are as follows:
after generating a corresponding export command according to the migration requirement type and exporting data to be migrated, a second field definition file which is supported by the Mongodb database and can be read is established according to a mapping relation between a first field definition file of a table generated in the process of exporting the data and data types of two heterogeneous databases of the Mysql database and the Mongodb database, and the process of generating the field _ file by automatically comparing the first field definition file with the mapping table in the embodiment of the invention comprises the following steps:
when the Mysqldmump exports data, an sql file (a first field definition file) and a txt file (data to be migrated) are generated under the condition of determining the directory, the first field definition file of the Mysql database is analyzed, and a second field definition file field _ file which is read by a Mongoimport tool and is suitable for Mongodb is generated. The comparison between the first data type defined by the column or field of the Mysql database and the second data type corresponding to the column or field supported and readable by Mongoimport is shown in the above tables 1 and 2 (note: blob type data does not support).
In this step, a one-to-one mapping table of the first data type of the relational database and the second data type of the non-relational database is established, which comprises a plurality of types of char/varchar/text/date of the Mysql database, wherein the data type used for establishing the one-to-one mapping table can be modified according to different application environments, to further adapt to more complex application environments, in the embodiment of the present invention, after obtaining the one-to-one mapping table, the contents in the table above are stored in the text file under the export directory, the file name match.f, the step of comparing the first file with the one-to-one mapping table to generate the second field definition file supported by the non-relational database comprises the steps of comparing the first field definition file exported from the Mysql database with the first field definition file in the importing process, and then automatically converting to generate the fieldfile supported by the mongoimmonport.
The code for converting the match.f into the second field definition file field file of the Mongodb database import tool Mongoimport comprises the following steps:
and after a second field definition file is generated, generating a Mongoimport import command according to the txt file (to-be-migrated data) in the determined directory BASE _ DIR, and then executing the Mongoimport to import the txt data (to-be-migrated data) into the Mongodb in batch.
An implementation manner of the embodiment of the present invention is, as shown in fig. 2, a flowchart of an implementation process of a one-key data migration method according to the embodiment of the present invention, where a program is started after a configuration file is modified according to a migration requirement and a specific application environment in an early stage to enable a code to run, a specific migration requirement is determined according to reading of the configuration file, and a corresponding export command is generated, where the export command corresponds to the migration requirement, when data of each table is migrated as one migration requirement, it corresponds to one export command, when data of one library is migrated as one migration requirement, it also corresponds to one export command, therefore, when a batch is exported, it corresponds to a batch export command, exports data according to a batch export command, and simultaneously establishes a second field definition file of an export database and an import database, and generates an import command according to the second field definition file, and executing the import command to import data.
In another implementation manner of the embodiment of the present invention, the method further includes a step of generating an import log so as to monitor and record a subsequent data import process, where the process includes the following steps:
generating an import log when the second file is imported into the non-relational database;
carrying out keyword retrieval on the imported logs and determining abnormal logs with imported exceptions;
and outputting the abnormal log.
In the embodiment of the invention, an import log is generated in the process of importing the data to be migrated into the Mongobb database, the import log is stored in a log directory file, after the one-key migration operation is executed through codes, keyword retrieval is carried out on the import log in the log directory file, for example, keyword 'error' retrieval is carried out on the import log, the log with 'error' field is determined to be an abnormal log with import abnormality, the abnormal log is output, and the abnormal log is sent to relevant operation and maintenance personnel for error correction.
The method of the embodiment of the invention realizes the inspection of the import log through the automatic running of the code, can realize the automatic inspection of whether the import operation is abnormal or not, and can realize the manual intervention on the data migration of the library or the table with errors or abnormalities.
The configuration steps of the embodiment of the invention comprise the determination of the exported file directory, the determination of the log directory, the determination of the second field definition file directory, the determination of the temporary file directory and the like, and the determination of the file directories can be predefined in the configuration file according to the requirements according to different specific implementation environments.
According to the method provided by the embodiment of the invention, when the transferred data to be transferred is less, a mode of converting the data type of the first field definition file and then storing the converted data into the relational database one by one can be adopted, when the transferred data to be transferred is more, a mode of converting the data type of the first field definition file and then storing the converted data into the temporary file can be adopted, and all data in the temporary file are imported into the relational database after the conversion is finished.
Fig. 3 is a schematic structural diagram of a device for one-key data migration according to an embodiment of the present invention, where the device may be implemented by software and/or hardware, and may be implemented by a method for one-key data migration. As shown in the figure, the present embodiment may provide a device for migrating data by one key based on the above embodiments, which is applied to data migration from a relational database to a non-relational database, and mainly includes a first determining module 310, a second determining module 320, and a module for migrating data by one key 330.
The first determining module 310 is configured to determine a first field definition file of data to be migrated in a relational database, where the first field definition file stores a field of the data to be migrated and a first data type defined by the field;
the second determining module 320 is configured to determine, according to the first field definition file determined by the first determining module and a mapping relationship, a second field definition file supported by a non-relational database, where the mapping relationship is a mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the second data type corresponding to the field and the field;
the one-key migration module 330 is configured to migrate the data to be migrated in the relational database to the non-relational database in a one-key manner according to the second field definition file determined by the second determination module.
In an implementation scenario of the embodiment of the present invention, the apparatus further includes:
the acquisition module is used for acquiring first parameter information of the relational database, second parameter information of the non-relational database and a migration requirement type;
the one-key migration module 330 is configured to:
determining a derivation command by combining the first parameter information of the relational database acquired by the acquisition module and the migration requirement type, and deriving the data to be migrated from the relational database according to the derivation command; and generating an import command according to the second parameter information of the non-relational database and the second field definition file, and migrating the data to be migrated to the non-relational database by one key according to the import command.
In an implementation scenario of the embodiment of the present invention, the one-key migration module 330 includes:
the first migration sub-module is used for determining the export command as a first export command when the migration requirement type is the migration of the library;
the second migration submodule is used for determining the export command as a second export command when the migration requirement type is the migration of the table;
and the third migration submodule is used for determining that the export command is a third export command when the migration requirement type is the migration of the library and the partial table.
In an implementation scenario of the embodiment of the present invention, the one-key migration module 330 is configured to:
at least obtaining a port number and a network address of the relational database;
at least including obtaining the port number and network address of the non-relational database.
In an implementation scenario of the embodiment of the present invention, the device for migrating data by one key further includes:
the log module is used for generating an import log when the data to be migrated is imported into the non-relational database;
the retrieval module is used for carrying out keyword retrieval on the import log and determining the import log with abnormity;
and the output module is used for outputting the import log with the exception.
In an implementation scenario of the embodiment of the present invention, the apparatus further includes:
the export directory determining module is used for determining the data to be migrated and the directory exported by the first field definition file;
the column definition module exports a directory for determining a directory of the second field definition file.
The one-key data migration apparatus provided in the above embodiments may execute the one-key data migration method provided in any embodiment of the present invention, and have functional modules and beneficial effects corresponding to the execution of the one-key data migration method.
Fig. 4 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 4: a processor (processor)410, a communication Interface 420, a memory (memory)430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are communicated with each other via the communication bus 440. The processor 410 may call logic instructions in the memory 430 to perform the following method: determining a first field definition file of data to be migrated in a relational database, wherein the first field definition file stores fields of the data to be migrated and a first data type defined by the fields; determining a second field definition file supported by a non-relational database according to the first field definition file and a mapping relation, wherein the mapping relation is the mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the second data type corresponding to the field and the field; and migrating the data to be migrated in the relational database to the non-relational database in a one-key mode according to the second field definition file.
In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the transmission method provided in the foregoing embodiments when executed by a processor, and for example, the method includes: determining a first field definition file of data to be migrated in a relational database, wherein the first field definition file stores fields of the data to be migrated and a first data type defined by the fields; determining a second field definition file supported by a non-relational database according to the first field definition file and a mapping relation, wherein the mapping relation is the mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the second data type corresponding to the field and the field; and migrating the data to be migrated in the relational database to the non-relational database in a one-key mode according to the second field definition file.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A one-key migration method of data is characterized by comprising the following steps:
determining a first field definition file of data to be migrated in a relational database, wherein the first field definition file stores fields of the data to be migrated and a first data type defined by the fields;
determining a second field definition file supported by a non-relational database according to the first field definition file and a mapping relation, wherein the mapping relation is the mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the second data type corresponding to the field and the field;
and migrating the data to be migrated in the relational database to the non-relational database in a one-key mode according to the second field definition file.
2. The method for one-key migration of data according to claim 1, wherein before determining the first field definition file of the data to be migrated in the relational database, the method for one-key migration of data further comprises:
acquiring first parameter information of the relational database, second parameter information of the non-relational database and a migration requirement type;
the migrating the data to be migrated in the relational database to the non-relational database in a one-touch manner according to the second field definition file includes:
determining a derivation command by combining the first parameter information of the relational database and the migration requirement type, and deriving the data to be migrated from the relational database according to the derivation command;
and generating an import command according to the second parameter information of the non-relational database and the second field definition file, and migrating the data to be migrated to the non-relational database by one key according to the import command.
3. The method for one-touch migration of data according to claim 2, wherein said determining an export command in combination with the first parameter information of the relational database and the migration requirement type comprises:
when the migration requirement type is migration of a library, determining that the export command is a first export command;
when the migration requirement type is the migration of a table, determining that the export command is a second export command;
and when the migration requirement type is migration of a library and a partial table, determining that the export command is a third export command.
4. The method for one-touch data migration according to claim 2, wherein the obtaining of the first parameter information of the relational database, the second parameter information of the non-relational database, and the type of migration requirement comprises:
the obtaining of the first parameter information of the relational database at least comprises obtaining a port number and a network address of the relational database;
the obtaining of the second parameter information of the non-relational database at least comprises obtaining a port number and a network address of the non-relational database.
5. The method for one-key migration of data according to claim 1, further comprising:
when the data to be migrated is imported into the non-relational database, an import log is generated;
carrying out keyword retrieval on the import log, and determining the import log with abnormality;
and outputting the import log with the exception.
6. The method for one-touch data migration according to claim 1, wherein before determining the first field definition file of the data to be migrated in the relational database, the method further comprises:
determining a directory derived from the data to be migrated and the first field definition file; and/or
Determining a directory of the second field definition file.
7. An apparatus for one-touch migration of data, the apparatus comprising:
the system comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining a first field definition file of data to be migrated in a relational database, and the first field definition file stores fields of the data to be migrated and a first data type defined by the fields;
a second determining module, configured to determine, according to the first field definition file determined by the first determining module and a mapping relationship, a second field definition file supported by a non-relational database, where the mapping relationship is a mapping between the first data type and a second data type in the non-relational database, and the second field definition file stores the second data type corresponding to the field and the field;
and the one-key migration module is used for performing one-key migration on the data to be migrated in the relational database to the non-relational database according to the second field definition file determined by the second determination module.
8. The device for one-touch migration of data according to claim 7, further comprising:
the acquisition module is used for acquiring first parameter information of the relational database, second parameter information of the non-relational database and a migration requirement type;
the one-key migration module comprises:
the export submodule is used for determining an export command by combining the first parameter information of the relational database and the migration requirement type, and exporting the data to be migrated from the relational database according to the export command;
and the import submodule is used for generating an import command according to the second parameter information of the non-relational database and the second field definition file, and migrating the data to be migrated to the non-relational database by one key according to the import command.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of a key migration method of data according to any of claims 1 to 6.
10. A non-transitory computer readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of a key migration method of data according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910906044.3A CN110674113A (en) | 2019-09-24 | 2019-09-24 | One-key migration method and device for data, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910906044.3A CN110674113A (en) | 2019-09-24 | 2019-09-24 | One-key migration method and device for data, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110674113A true CN110674113A (en) | 2020-01-10 |
Family
ID=69078589
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910906044.3A Pending CN110674113A (en) | 2019-09-24 | 2019-09-24 | One-key migration method and device for data, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110674113A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111694812A (en) * | 2020-05-06 | 2020-09-22 | 五八有限公司 | Data migration method and data migration device |
| CN111752814A (en) * | 2020-05-26 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Batch processing method and system for RMT (remote metering test) data |
| CN111966597A (en) * | 2020-08-18 | 2020-11-20 | 中国银行股份有限公司 | Test data generation method and device |
| CN112860792A (en) * | 2021-01-29 | 2021-05-28 | 银盛支付服务股份有限公司 | Method and system for synchronizing mongodb cluster data and hive cluster data |
| CN113282537A (en) * | 2021-06-15 | 2021-08-20 | 成都深思科技有限公司 | ES data migration system and migration method |
| CN113407514A (en) * | 2021-05-21 | 2021-09-17 | 济南浪潮数据技术有限公司 | Method, device and equipment for migrating database and readable medium |
| CN113778988A (en) * | 2021-08-23 | 2021-12-10 | 咪咕数字传媒有限公司 | Data processing method, device, equipment and computer program product |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102308297A (en) * | 2011-07-13 | 2012-01-04 | 华为技术有限公司 | Data migration method, data migration device and data migration system |
| CN103440273A (en) * | 2013-08-06 | 2013-12-11 | 北京航空航天大学 | Data cross-platform migration method and device |
| CN103530327A (en) * | 2013-09-25 | 2014-01-22 | 清华大学深圳研究生院 | Method for migrating data from non-relational database to relational database |
| CN110019135A (en) * | 2017-12-27 | 2019-07-16 | 航天信息股份有限公司 | It is a kind of to migrate relational data to the method and device of HBase database |
-
2019
- 2019-09-24 CN CN201910906044.3A patent/CN110674113A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102308297A (en) * | 2011-07-13 | 2012-01-04 | 华为技术有限公司 | Data migration method, data migration device and data migration system |
| CN103440273A (en) * | 2013-08-06 | 2013-12-11 | 北京航空航天大学 | Data cross-platform migration method and device |
| CN103530327A (en) * | 2013-09-25 | 2014-01-22 | 清华大学深圳研究生院 | Method for migrating data from non-relational database to relational database |
| CN110019135A (en) * | 2017-12-27 | 2019-07-16 | 航天信息股份有限公司 | It is a kind of to migrate relational data to the method and device of HBase database |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111694812A (en) * | 2020-05-06 | 2020-09-22 | 五八有限公司 | Data migration method and data migration device |
| CN111752814A (en) * | 2020-05-26 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Batch processing method and system for RMT (remote metering test) data |
| CN111966597A (en) * | 2020-08-18 | 2020-11-20 | 中国银行股份有限公司 | Test data generation method and device |
| CN111966597B (en) * | 2020-08-18 | 2024-04-16 | 中国银行股份有限公司 | Test data generation method and device |
| CN112860792A (en) * | 2021-01-29 | 2021-05-28 | 银盛支付服务股份有限公司 | Method and system for synchronizing mongodb cluster data and hive cluster data |
| CN113407514A (en) * | 2021-05-21 | 2021-09-17 | 济南浪潮数据技术有限公司 | Method, device and equipment for migrating database and readable medium |
| CN113282537A (en) * | 2021-06-15 | 2021-08-20 | 成都深思科技有限公司 | ES data migration system and migration method |
| CN113778988A (en) * | 2021-08-23 | 2021-12-10 | 咪咕数字传媒有限公司 | Data processing method, device, equipment and computer program product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110674113A (en) | One-key migration method and device for data, electronic equipment and storage medium | |
| CN111061757B (en) | Language conversion method and device of database, electronic equipment and storage medium | |
| CN110908997B (en) | Data blood relationship construction method and device, server and readable storage medium | |
| EP3343411A1 (en) | Sql auditing method and apparatus, server and storage device | |
| US11061652B1 (en) | System and a method for moving legacy applications to cloud | |
| US9619537B2 (en) | Converting data objects from single- to multi-source database environment | |
| US20070073675A1 (en) | Database query translation | |
| CN112579610A (en) | Multi-data source structure analysis method, system, terminal device and storage medium | |
| CN111291074B (en) | Database query method, system, medium and device | |
| EP2904520A1 (en) | Reference data segmentation from single to multiple tables | |
| JP7131119B2 (en) | Systems and methods for merging source data from a source application with target data in a target application | |
| CN111259038A (en) | Database query and data export method, system, medium and equipment | |
| CN115757174A (en) | Database difference detection method and device | |
| CN115757481A (en) | Data migration method, device, equipment and storage medium | |
| CN115567481A (en) | Upper computer PLC address binding method and device and electronic equipment | |
| CN113448965A (en) | Method, device and equipment for determining full-table-scanning structured query statement | |
| CN114218261A (en) | Data query method and device, storage medium and electronic equipment | |
| CN111651531A (en) | Data import method, device, equipment and computer storage medium | |
| CN112749189A (en) | Data query method and device | |
| CN114547404B (en) | Big data platform system | |
| CN110727677A (en) | Method and device for tracing blood relationship of table in data warehouse | |
| CN111291026A (en) | Data access method, system, device and computer readable medium | |
| CN117331926B (en) | Data auditing method and device, electronic equipment and storage medium | |
| CN110569243B (en) | Data query method, data query plug-in and data query server | |
| CN116126902A (en) | Method and storage medium for supporting like operation by partition pruning in SQL (structured query language) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200110 |