CN111177113A - Data migration method and device, computer equipment and storage medium - Google Patents
Data migration method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111177113A CN111177113A CN201911244417.1A CN201911244417A CN111177113A CN 111177113 A CN111177113 A CN 111177113A CN 201911244417 A CN201911244417 A CN 201911244417A CN 111177113 A CN111177113 A CN 111177113A
- Authority
- CN
- China
- Prior art keywords
- data
- migration
- data migration
- parameters
- relational database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data migration method, a data migration device, computer equipment and a storage medium, wherein the method comprises the steps of operating a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background; acquiring the type of a relational database corresponding to data to be migrated, and determining front-end page parameters according to the type of the relational database; loading the determined front-end page parameters and displaying a front-end page corresponding to the data migration front end; acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page; and generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to execute the migration script according to the scheduling task through the data migration background to complete data migration. The application relates to the field of data processing, and the data migration efficiency among different types of databases can be improved.
Description
Technical Field
The present application relates to the field of data migration, and in particular, to a data migration method and apparatus, a computer device, and a storage medium.
Background
At present, in the internet era, the application of a database is very wide, the database has the effect of mainly helping a user to store and manage data, and related data can be quickly and effectively acquired and updated through operations such as querying, deleting or newly adding data and the like. Although the types of databases are more at present, the databases mainly include two types, namely, relational database Management System (RDBMS) and non-relational database (NoSQL). Because the advantages and disadvantages of relational databases and non-relational databases can be mostly complemented, there is often a need for data migration between different types of databases in many businesses.
Most of the data migration methods of today utilize scheduling tools to migrate data among databases of the same type, but migration of data among databases of different types is difficult or inefficient. For example, data in a relational database is migrated to a non-relational database, such data migration does not have an open-source data migration tool, and often needs developers to write codes to complete the data migration, which undoubtedly increases the workload of the developers and causes great troubles to users who only use SQL language. Therefore, how to improve the data migration efficiency between the relational database and the non-relational database becomes a problem which needs to be solved urgently.
Disclosure of Invention
The application provides a data migration method, a data migration device, computer equipment and a storage medium, and aims to improve data migration efficiency among different types of databases.
In a first aspect, the present application provides a data migration method, including the steps of:
running a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background;
acquiring the type of a relational database corresponding to data to be migrated, determining front-end page parameters according to the type of the relational database, wherein the relational databases of different types correspond to different front-end page parameters;
loading the determined front-end page parameters and displaying a front-end page corresponding to the data migration front end;
acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page;
generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script;
and executing the migration script according to the scheduling task through the data migration background to finish data migration.
In a second aspect, the present application further provides a data migration apparatus, including:
the tool running module is used for running a data migration tool, and the data migration tool comprises a data migration front end and a data migration background;
the type acquisition module is used for acquiring the type of a relational database corresponding to the data to be migrated, determining front-end page parameters according to the type of the relational database, wherein the relational databases of different types correspond to different front-end page parameters;
the front-end display module is used for loading the determined front-end page parameters and displaying a front-end page corresponding to the data migration front end;
the parameter acquisition module is used for acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page;
the script determining module is used for generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script;
and the data migration module is used for executing the migration script through the data migration background according to the scheduling task to complete data migration.
In a third aspect, the present application also provides a computer device comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the data migration method as described above.
In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, where the computer program, when executed by a processor, implements the steps of the data migration method as described above.
The application provides a data migration method, a data migration device, computer equipment and a storage medium, and is used for migrating data of a relational database into a non-relational database, acquiring the type of the relational database corresponding to the data to be migrated by operating a data migration tool, determining front-end page parameters according to the type of the relational database, loading the determined front-end page parameters to display a front-end page corresponding to a data migration front end, acquiring the data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page, generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, transmitting the data parameters to the migration script, and finally executing the migration script according to the scheduling task through a data migration background to complete data migration. According to the method and the device, data can be rapidly migrated from the relational database to the non-relational database, so that a complex development process is avoided, and the data migration efficiency is effectively improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram of a data migration system provided by an embodiment of the present application;
fig. 2 is a schematic flowchart of a data migration method according to an embodiment of the present application;
FIG. 3 is a schematic flow chart diagram illustrating another data migration method according to an embodiment of the present application;
fig. 4 is a schematic block diagram of a data migration method apparatus according to an embodiment of the present application;
FIG. 5 is a schematic block diagram of another data migration method apparatus provided in an embodiment of the present application;
fig. 6 is a block diagram schematically illustrating a structure of a computer device according to an embodiment of the present application.
The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The flow diagrams depicted in the figures are merely illustrative and do not necessarily include all of the elements and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
Embodiments of the present application provide a data migration method, a data migration apparatus, a computer device, and a storage medium, which are used to migrate data in a relational database to a non-relational database, so as to improve data migration efficiency.
As shown in fig. 1, fig. 1 is a data migration system provided for an embodiment of the present application, and the system includes a terminal and a server, where the terminal and the server are communicatively connected, and the server is communicatively connected to a database, where the database includes a relational database and a non-relational database.
The terminal comprises electronic equipment such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant and wearable equipment.
The server comprises an independent server or a server cluster.
Hereinafter, a data migration method provided by an embodiment of the present application will be described in detail based on the data migration system.
Referring to fig. 2, fig. 2 is a flowchart illustrating a data migration method according to an embodiment of the present application. The data migration method can be used in a terminal of a data migration system.
As shown in fig. 2, the data migration method includes steps S101 to S105.
Step S101, a data migration tool is operated, wherein the data migration tool comprises a data migration front end and a data migration background.
Before the data migration tool is run, the data migration tool needs to be established in advance, and the data migration tool comprises a data migration front end and a data migration background. Specifically, the data migration background can be developed by using the currently popular Java or Python language and based on the Django framework in Spring Boot or Python; meanwhile, a data migration front end is developed using a lightweight front end framework React or Vue.
The data migration front end is specifically a Web front end, operates in the terminal, interacts with a user through a display Web page to acquire corresponding information, and sends the information to the migration background for processing.
After the data migration tool is developed, the data migration tool is stored in a server, and when a user needs to migrate data in a relational database to a non-relational database, a pre-created data migration tool is started and operated, namely, a Web front end of the data migration tool is opened to prepare for data migration.
In an embodiment, when the Web front end of the data migration tool is opened and run, the first front end page and the data migration background of the data migration tool are run first, and the displayed first front end page may only include an input box, where the input box is used to facilitate a user to input a database identifier of a relational database corresponding to data to be migrated. The database identifier may be a database name or a database ID.
S102, obtaining the type of a relational database corresponding to the data to be migrated, and determining front-end page parameters according to the type of the relational database.
Wherein, different types of relational databases correspond to different front-end page parameters. Therefore, different pages are displayed according to different data types, a user can conveniently check or select data migration, and the display speed of the pages is increased.
In one embodiment, the method includes the steps of obtaining a type of a relational database corresponding to data to be migrated, and determining front-end page parameters according to the type of the relational database, wherein the specific steps are as follows: acquiring a database identifier corresponding to data to be migrated, inquiring the type of the relational database according to the database identifier, determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database, and analyzing field information in a data table of the relational database according to the determined field analysis rule; and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
It should be noted that different types of relational databases correspond to different front-end page parameters, because the storage data formats of the different types of relational databases are different, such as fields in a database table: the data type can be binary data type, character data type or Unicode data type, so that the data table and field information of the relational database are displayed by utilizing the front-end page, and the specific content of the front-end page can be quickly displayed only by determining the type of the relational database.
S103, loading the determined front-end page parameters, and displaying the front-end page corresponding to the data migration front end.
After the front-end page parameters are obtained, page content and field information obtained by analysis are displayed according to a display strategy by loading the front-end page parameters, namely the page content and the field information form a front-end page corresponding to the data migration front end.
The specific display process is as follows: acquiring page content, a display strategy and a field analysis rule in front-end page parameters; inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule; displaying the page content and the field information according to the display strategy; wherein, the field information is used for facilitating the user to determine the data needing to be migrated.
It should be noted that, because different types of relational databases correspond to different front-end page parameters, the display policies, page contents, and parsing rules included in the different types of front-end page parameters may be the same or different. The last displayed front page may or may not be the same.
And S104, acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page.
The method comprises the steps of obtaining data parameters and scheduling parameters corresponding to data to be migrated through a front-end page, and specifically comprises the steps of obtaining a database of the data parameters, a data table name, the type of a non-relational database, and obtaining the execution time and the execution mode of the scheduling parameters.
In an embodiment, when data migration is performed, the acquired data parameters include field information of a database list to be migrated, and the field information is displayed through a front-end page, so that a user can configure a field of the database list to be migrated on the front-end page as required, that is, the database, the name of the database list, the field information and the like of data to be migrated are input in an input box of the front-end page.
The scheduling parameters include execution time and execution mode. The execution mode includes automatic execution or manual execution, for example, when the time reaches the execution time set by the scheduling parameter, the automatic execution mode may automatically execute data migration, and the manual execution mode may manually activate an activation button disposed on the data front-end page within the execution time according to the user's needs. For example, when it is detected that the user clicks the activation button, data migration may be performed. Optionally, the activation state of the activation button may be represented by a color change, for example, a change from gray to green indicates that the activation button has been activated, or may be designed according to the user's needs, and the embodiment is not limited in this respect.
And S105, generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to execute the migration script according to the scheduling task through the data migration background to complete data migration.
Specifically, a scheduling task is generated according to the scheduling parameter, that is, a Spark scheduling task is generated according to the execution time and the execution mode, and meanwhile, the Spark scheduling task can also be packaged into a scheduling Jar packet and sent to the data migration background, so that the data migration background executes the scheduling task according to the Jar packet.
Specifically, a corresponding migration script, namely a Shell script, is determined according to the type of the non-relational database, and the specific process is as follows: and querying a preset mapping relation table between the type of the non-relational database and the migration script to obtain a query result so as to determine the corresponding migration script.
Different Shell scripts correspond to different database types, and the different database types comprise configuration and data format requirements related to the database types. Specifically, as shown in table 1, table 1 is represented as a mapping relationship table corresponding to the type of the Shell script and the non-relational database.
TABLE 1 mapping relationship Table
| Type of non-relational database | Shell script |
| HBase | Shell script I |
| MongoDB | Shell script II |
| ELASEARCH | Shell script III |
In addition, the Shell script also has a speed limiting function, and for a system with a large access amount, data can never be read and written without control, because the bandwidth of each system is limited, and IO (input/output) of a disk is also limited. Therefore, speed limitation is required, and the method specifically comprises the following steps: monitoring the consumed time of the migration script when reading and writing data; and determining the pause time according to the consumed time, and pausing according to the pause time.
For example, the Shell script monitors the consumed time during reading and writing data, and starts a short wait if the consumed time corresponding to reading the corresponding data exceeds a preset threshold, for example, it takes more than 100ms to read 100 pieces of data, the Shell script pauses for 1s, and if it takes more than 200ms, the Shell script pauses for 5 s. Therefore, the speed limit is realized, and the data migration speed is improved.
And after the Shell script is determined, transmitting data parameters corresponding to the data to be migrated into the determined Shell script, wherein the transmitted parameters are different according to different non-relational databases. For example, live data of a relational database is migrated to ELASTICSEARCH (ES for short) database of a non-relational database, the incoming data parameter is the live SQL statement, and when data of the MySQL database is migrated to the HBase database, the incoming data parameter is the data table name of MySQL.
And when the execution time in the scheduling task is reached, the data migration background calls the Shell script of the transmitted parameters, and executes the Shell script to finish data migration.
Specifically, a Shell script is called by using a spring boot of a data migration background, so that data scheduling between databases is realized, such as what time and what fields need to be scheduled each day, or what field scheduling is completed by clicking a manual button, and thus data migration from a relational database to a non-relational database is completed.
In the data migration method for migrating data of a relational database to a non-relational database provided in the above embodiment, a pre-created data migration tool is used to obtain the type of the relational database corresponding to the data to be migrated to determine the front-end page, obtain the data parameters of the data to be migrated and determine the type of the non-relational database to be migrated through the front-end page, determine the corresponding Shell script according to the type of the non-relational database, and complete data migration by using the Shell script, so that data can be quickly migrated from the relational database to the non-relational database, a cumbersome development process is avoided, and data migration efficiency is improved.
Referring to fig. 3, fig. 3 is a flowchart illustrating another data migration method according to an embodiment of the present application. The data migration method can be used in a data migration system.
As shown in fig. 3, the data migration method includes steps S201 to S207.
S201, operating a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background;
starting and running a pre-created data migration tool, namely opening a Web front end of the data migration tool, running a first front end page and a data migration background of the data migration tool, wherein the displayed first front end page can only comprise an input box, and the input box is used for facilitating a user to input data parameters of a relational database corresponding to data to be migrated.
S202, displaying an input box through a data migration front end, acquiring the type of a relational database corresponding to data to be migrated through the input box, and determining front end page parameters according to the type of the relational database.
Specifically, after the data migration front end displays the input box, the user may input data parameters corresponding to data that the user needs to migrate, such as a database name, a data table, and corresponding field information, in the input box, obtain, according to the input data parameters, a database name corresponding to the data to be migrated that the user fills in the input box, and obtain, through the database name, a type of the relational database corresponding to the data to be migrated.
Determining front-end page parameters according to the type of the relational database, and the specific process comprises the following steps: determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database, and analyzing field information in a data table of the relational database according to the determined field analysis rule; and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
S203, loading the determined front-end page parameters, and displaying a front-end page corresponding to the data migration front end;
loading the determined front-end page parameters to display a front-end page corresponding to the data migration front end, specifically including: and loading the determined front-end page parameters to display a second front-end page corresponding to the data migration front end on the first front-end page, wherein the second front-end page is related to the type of the relational database, different types of relational data correspond to different front-end pages, and the first front-end page and the second front-end page jointly form the front-end page of the data migration front end.
In an embodiment, the page content, the display policy and the field analysis rule in the page parameter are acquired through the first front-end page, the second front-end page inquires and analyzes the field information corresponding to the data to be migrated according to the field analysis rule, and the page content and the field information are displayed on the front-end page according to the display policy. Wherein, the field information is used for facilitating the user to determine the data needing to be migrated.
S204, acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page;
the method comprises the steps of obtaining data parameters and scheduling parameters corresponding to data to be migrated through a front-end page, and specifically comprises the steps of obtaining a database of the data parameters, a data table name, the type of a non-relational database, and obtaining the execution time and the execution mode of the scheduling parameters.
In an embodiment, the data parameters acquired by the front-end page are obtained by acquiring an import index or a namespace of the database, and taking the import index or the namespace as the data parameters corresponding to the data to be migrated. Specifically, the naming modes of the databases corresponding to different types of databases are different, wherein the import index refers to a concept similar to the database on the ES, the namespace is a concept of the HBASE database, and is similar to the concept of the RDBMS database, and there are many data tables under each namespace, so that the data to be migrated is stored in the data tables of the namespace.
The scheduling parameters include execution time and execution mode. The execution mode includes automatic execution or manual execution, for example, when the time reaches the execution time set by the scheduling parameter, the automatic execution mode may automatically execute data migration, and the manual execution mode may manually activate an activation button disposed on the data front-end page within the execution time according to the user's needs. For example, when it is detected that the user clicks the activation button, data migration may be performed. Optionally, the activation state of the activation button may be represented by a color change, for example, a change from gray to green indicates that the activation button has been activated, or may be designed according to the user's needs, and the embodiment is not limited in this respect.
S205, generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script.
Specifically, a scheduling task is generated according to a scheduling parameter, that is, a scheduling task, that is, a Spark scheduling task is generated according to an execution time and an execution mode, and a corresponding migration script, that is, a Shell script, is determined according to a type of a non-relational database, and the specific process is as follows: and querying a preset mapping relation table between the type of the non-relational database and the migration script to obtain a query result so as to determine the corresponding migration script, and transmitting data parameters corresponding to the data to be migrated to the determined migration script.
And S206, executing the migration script through the data migration background according to the scheduling task to complete data migration.
And when the execution time in the scheduling task is reached, the data migration background calls the Shell script of the transmitted parameters, and executes the Shell script to finish data migration.
Specifically, a Shell script is called by using a spring boot of a data migration background, so that data scheduling between databases is realized, such as what time and what fields need to be scheduled each day, or what field scheduling is completed by clicking a manual button, and thus data migration from a relational database to a non-relational database is completed.
And S207, sending a data verification instruction to the server to verify the data migrated to the non-relational database, and sending a data deletion instruction to the relational database corresponding to the data to be migrated to delete the original data when the verification is passed.
Specifically, after the data migration from the relational database to the non-relational database is completed, the method further includes verifying the data migrated into the non-relational database, and displaying a verification result at a data migration front end to determine the integrity and correctness of the migrated data.
And when the verification is passed, sending a data deleting instruction to the relational database corresponding to the data to be migrated so as to delete the original data, so that the storage space of the original relational database is saved. The specific verification process is as follows: encrypting the data to be migrated and acquiring an encrypted character string; and meanwhile, encrypting the migrated data to obtain an encrypted character string, then comparing whether the encrypted character strings of the data to be migrated and the migrated data are consistent, if so, determining that the verification is passed, otherwise, not passing the verification. It should be noted that the encryption method for the data to be migrated is the same as the encryption method for the data after migration, and for example, the MD5 encryption method is adopted.
The data migration method provided by the embodiment is used for migrating data of a relational database to a non-relational database, acquiring the type, data parameters and scheduling parameters of data to be migrated through a data migration front end, determining a corresponding migration script according to the type of the non-relational database, executing the determined migration script through a data migration background to complete data migration, and performing encryption verification on the migrated data to determine the integrity and correctness of the migrated data, so that the efficiency and accuracy of data migration are improved.
Referring to fig. 4, fig. 4 is a schematic block diagram of a data migration apparatus according to an embodiment of the present application, the data migration apparatus being configured to perform the foregoing data migration method. Wherein, the data migration device can be configured in the terminal.
As shown in fig. 4, the moving object detecting apparatus 300 includes: a tool running module 301, a type obtaining module 302, a front end display module 303, a parameter obtaining module 304 and a script determining module 305.
The tool running module 301 is configured to run a data migration tool, where the data migration tool includes a data migration front end and a data migration background;
a type obtaining module 302, configured to obtain a type of a relational database corresponding to data to be migrated, and determine front-end page parameters according to the type of the relational database, where relational databases of different types correspond to different front-end page parameters;
a front-end display module 303, configured to load the determined front-end page parameters and display a front-end page corresponding to the data migration front end;
a parameter obtaining module 304, configured to obtain, through the front-end page, a data parameter and a scheduling parameter corresponding to the data to be migrated;
the script determining module 305 is configured to generate a scheduling task according to the scheduling parameter, determine a corresponding migration script according to the type of the non-relational database, and transmit the data parameter to the migration script, so that the data migration is completed by executing the migration script according to the scheduling task by the data migration background.
In an embodiment, the type obtaining module 302 is further configured to obtain a database identifier corresponding to the data to be migrated, and query the type of the relational database according to the database identifier; determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database; analyzing field information in a data table of the relational database according to the determined field analysis rule; and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
In an embodiment, the front-end display module 303 is further configured to obtain page content, a display policy, and a field parsing rule in the front-end page parameter; inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule; and displaying the page content and the field information according to the display strategy.
In an embodiment, the parameter obtaining module 304 is further configured to obtain data parameters of the data to be migrated through a front-end page, where the data parameters include types of a database, a data table name, and a non-relational database; and acquiring scheduling parameters of the data to be migrated through a front-end page, wherein the scheduling parameters comprise execution time and execution mode.
In an embodiment, the script determining module 305 is further configured to generate a scheduling task according to the execution time and the execution mode; and querying a preset mapping relation table between the type of the non-relational database and the migration script to obtain a query result so as to determine the corresponding migration script.
Referring to fig. 5, fig. 5 is a schematic block diagram of another data migration apparatus according to an embodiment of the present disclosure. The data migration apparatus is configured to execute the foregoing data migration method. The data migration apparatus may be configured in a server and a terminal.
As shown in fig. 5, the face recognition apparatus 400 includes: the system comprises a tool running module 401, a type obtaining module 402, a front end display module 403, a parameter obtaining module 404, a script determining module 405, a data migration module 406 and a data checking module 407.
A tool running module 401, configured to run a data migration tool, where the data migration tool includes a data migration front end and a data migration background;
a type obtaining module 402, configured to display an input box through a data migration front end, obtain, through the input box, a type of a relational database corresponding to data to be migrated, and determine front end page parameters according to the type of the relational database;
a front-end display module 403, configured to load the determined front-end page parameters and display a front-end page corresponding to the data migration front end
A parameter obtaining module 404, configured to obtain, through the front-end page, a data parameter and a scheduling parameter corresponding to the data to be migrated;
a script determining module 405, configured to generate a scheduling task according to the scheduling parameter, determine a corresponding migration script according to the type of the non-relational database, and transmit the data parameter to the migration script;
and the data migration module 406 is configured to execute the migration script according to the scheduling task through the data migration background to complete data migration.
And the data verification module 407 is configured to verify the data migrated to the non-relational database, and when the data is verified, send a data deletion instruction to the relational database corresponding to the data to be migrated to delete the original data.
It should be noted that, as will be clear to those skilled in the art, for convenience and brevity of description, the specific working processes of the apparatus and the modules described above may refer to the corresponding processes in the foregoing data migration method embodiment, and are not described herein again.
The apparatus provided by the above embodiments may be implemented in the form of a computer program, which can be run on a computer device as shown in fig. 6.
Referring to fig. 6, fig. 6 is a schematic block diagram of a computer device according to an embodiment of the present disclosure. The computer device may be a terminal.
Referring to fig. 6, the computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a nonvolatile storage medium and an internal memory.
The non-volatile storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, cause a processor to perform any of the data migration methods.
The processor is used for providing calculation and control capability and supporting the operation of the whole computer equipment.
The internal memory provides an environment for the execution of a computer program on a non-volatile storage medium, which when executed by a processor, causes the processor to perform any of a variety of data migration methods.
The network interface is used for network communication, such as sending assigned tasks and the like. Those skilled in the art will appreciate that the architecture shown in fig. 6 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
It should be understood that the Processor may be a Central Processing Unit (CPU), and the Processor may be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Wherein, in one embodiment, the processor is configured to execute a computer program stored in the memory to implement the steps of:
running a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background;
acquiring the type of a relational database corresponding to data to be migrated, determining front-end page parameters according to the type of the relational database, wherein the relational databases of different types correspond to different front-end page parameters;
loading the determined front-end page parameters and displaying a front-end page corresponding to the data migration front end;
acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page;
and generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to execute the migration script according to the scheduling task through the data migration background to complete data migration.
In one embodiment, the processor is configured to, when obtaining a type of a relational database corresponding to data to be migrated and determining a front-end page parameter according to the type of the relational database, implement:
acquiring a database identifier corresponding to data to be migrated, and inquiring the type of the relational database according to the database identifier; determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database; analyzing field information in a data table of the relational database according to the determined field analysis rule; and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
In one embodiment, the processor is configured to, after obtaining the type of the relational database corresponding to the data to be migrated, implement:
and displaying an input box through the data migration front end, and acquiring the type of the relational database corresponding to the data to be migrated through the input box.
In an embodiment, when the processor implements the loading of the determined front-end page parameter and displays the front-end page corresponding to the data migration front end, the processor is configured to implement:
acquiring page content, a display strategy and a field analysis rule in front-end page parameters; inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule; and displaying the page content and the field information according to the display strategy.
In an embodiment, when the processor obtains the data parameter and the scheduling parameter corresponding to the data to be migrated through the front-end page, the processor is configured to:
acquiring data parameters of data to be migrated through a front-end page, wherein the data parameters comprise a database, a data table name and the type of a non-relational database; and acquiring scheduling parameters of the data to be migrated through a front-end page, wherein the scheduling parameters comprise execution time and execution mode.
In an embodiment, when the processor implements the generation of the scheduling task according to the scheduling parameter and determines the corresponding migration script according to the type of the non-relational database, the processor is configured to implement:
generating a scheduling task according to the execution time and the execution mode; and querying a preset mapping relation table between the type of the non-relational database and the migration script to obtain a query result so as to determine the corresponding migration script.
In an embodiment, after the data migration is completed by the data migration background executing the migration script according to the scheduling task, the processor is further configured to:
and verifying the data migrated into the non-relational database, and sending a data deleting instruction to the relational database corresponding to the data to be migrated to delete the original data when the verification is passed.
The embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, where the computer program includes program instructions, and the processor executes the program instructions to implement any one of the data migration methods provided in the embodiments of the present application.
The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device.
While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A data migration method for migrating data of a relational database to a non-relational database, comprising:
running a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background;
acquiring the type of a relational database corresponding to data to be migrated, determining front-end page parameters according to the type of the relational database, wherein the relational databases of different types correspond to different front-end page parameters;
loading the determined front-end page parameters and displaying a front-end page corresponding to the data migration front end;
acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page;
and generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to execute the migration script according to the scheduling task through the data migration background to complete data migration.
2. The data migration method according to claim 1, wherein the obtaining of the type of the relational database corresponding to the data to be migrated and the determining of the front-end page parameter according to the type of the relational database comprise:
acquiring a database identifier corresponding to data to be migrated, and inquiring the type of the relational database according to the database identifier;
determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database;
analyzing field information in a data table of the relational database according to the determined field analysis rule;
and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
3. The data migration method according to claim 2, wherein the obtaining of the type of the relational database corresponding to the data to be migrated includes:
and displaying an input box through the data migration front end, and acquiring the type of the relational database corresponding to the data to be migrated through the input box.
4. The data migration method according to claim 2, wherein the loading the determined front-end page parameter and displaying the front-end page corresponding to the data migration front-end includes:
acquiring page content, a display strategy and a field analysis rule in front-end page parameters;
inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule;
and displaying the page content and the field information according to the display strategy.
5. The data migration method according to claim 1, wherein the obtaining of the data parameter and the scheduling parameter corresponding to the data to be migrated through the front-end page includes:
acquiring data parameters of data to be migrated through a front-end page, wherein the data parameters comprise a database, a data table name and the type of a non-relational database;
and acquiring scheduling parameters of the data to be migrated through a front-end page, wherein the scheduling parameters comprise execution time and execution mode.
6. The data migration method according to claim 5, wherein the generating a scheduling task according to the scheduling parameter and determining a corresponding migration script according to a type of the non-relational database comprises:
generating a scheduling task according to the execution time and the execution mode;
and querying a preset mapping relation table between the type of the non-relational database and the migration script to obtain a query result so as to determine the corresponding migration script.
7. The data migration method of claim 1, wherein after the completion of the data migration by the data migration background executing the migration script according to the scheduling task, further comprising:
and verifying the data migrated into the non-relational database, and sending a data deleting instruction to the relational database corresponding to the data to be migrated to delete the original data when the verification is passed.
8. A data migration apparatus, comprising:
the tool running module is used for running a data migration tool, and the data migration tool comprises a data migration front end and a data migration background;
the type acquisition module is used for acquiring the type of a relational database corresponding to the data to be migrated, determining front-end page parameters according to the type of the relational database, wherein the relational databases of different types correspond to different front-end page parameters;
the front-end display module is used for loading the determined front-end page parameters and displaying a front-end page corresponding to the data migration front end;
the parameter acquisition module is used for acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page;
and the script determining module is used for generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to execute the migration script according to the scheduling task through the data migration background to complete data migration.
9. A computer arrangement comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the data migration method of any one of claims 1 to 7.
10. A computer-readable storage medium, having stored thereon a computer program, wherein the computer program, when being executed by a processor, carries out the steps of the data migration method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911244417.1A CN111177113B (en) | 2019-12-06 | 2019-12-06 | Data migration method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911244417.1A CN111177113B (en) | 2019-12-06 | 2019-12-06 | Data migration method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111177113A true CN111177113A (en) | 2020-05-19 |
| CN111177113B CN111177113B (en) | 2023-07-25 |
Family
ID=70648755
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911244417.1A Active CN111177113B (en) | 2019-12-06 | 2019-12-06 | Data migration method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111177113B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112131128A (en) * | 2020-09-29 | 2020-12-25 | 网易（杭州）网络有限公司 | Data testing method, device, storage medium and electronic device |
| CN112181489A (en) * | 2020-09-28 | 2021-01-05 | 中国平安人寿保险股份有限公司 | Code migration method and device, computer equipment and storage medium |
| CN112214473A (en) * | 2020-09-23 | 2021-01-12 | 航天信息股份有限公司 | Data migration method and system between databases |
| CN115543969A (en) * | 2022-11-28 | 2022-12-30 | 建信金融科技有限责任公司 | Data migration method, device, equipment and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20080084332A (en) * | 2007-03-16 | 2008-09-19 | (주)온디멘드 | Database migration apparatus and method |
| US20160188710A1 (en) * | 2014-12-29 | 2016-06-30 | Wipro Limited | METHOD AND SYSTEM FOR MIGRATING DATA TO NOT ONLY STRUCTURED QUERY LANGUAGE (NoSOL) DATABASE |
| CN109684298A (en) * | 2018-09-07 | 2019-04-26 | 平安科技（深圳）有限公司 | Database migration method, apparatus, equipment and storage medium |
| CN110209652A (en) * | 2019-05-20 | 2019-09-06 | 平安科技（深圳）有限公司 | Tables of data moving method, device, computer equipment and storage medium |
| CN110297813A (en) * | 2019-05-22 | 2019-10-01 | 平安银行股份有限公司 | Data migration method, device, computer equipment and storage medium |
-
2019
- 2019-12-06 CN CN201911244417.1A patent/CN111177113B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20080084332A (en) * | 2007-03-16 | 2008-09-19 | (주)온디멘드 | Database migration apparatus and method |
| US20160188710A1 (en) * | 2014-12-29 | 2016-06-30 | Wipro Limited | METHOD AND SYSTEM FOR MIGRATING DATA TO NOT ONLY STRUCTURED QUERY LANGUAGE (NoSOL) DATABASE |
| CN109684298A (en) * | 2018-09-07 | 2019-04-26 | 平安科技（深圳）有限公司 | Database migration method, apparatus, equipment and storage medium |
| CN110209652A (en) * | 2019-05-20 | 2019-09-06 | 平安科技（深圳）有限公司 | Tables of data moving method, device, computer equipment and storage medium |
| CN110297813A (en) * | 2019-05-22 | 2019-10-01 | 平安银行股份有限公司 | Data migration method, device, computer equipment and storage medium |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112214473A (en) * | 2020-09-23 | 2021-01-12 | 航天信息股份有限公司 | Data migration method and system between databases |
| CN112181489A (en) * | 2020-09-28 | 2021-01-05 | 中国平安人寿保险股份有限公司 | Code migration method and device, computer equipment and storage medium |
| CN112181489B (en) * | 2020-09-28 | 2023-07-25 | 中国平安人寿保险股份有限公司 | Code migration method, device, computer equipment and storage medium |
| CN112131128A (en) * | 2020-09-29 | 2020-12-25 | 网易（杭州）网络有限公司 | Data testing method, device, storage medium and electronic device |
| CN112131128B (en) * | 2020-09-29 | 2023-08-22 | 网易（杭州）网络有限公司 | Data testing method and device, storage medium and electronic device |
| CN115543969A (en) * | 2022-11-28 | 2022-12-30 | 建信金融科技有限责任公司 | Data migration method, device, equipment and medium |
| CN115543969B (en) * | 2022-11-28 | 2023-03-10 | 建信金融科技有限责任公司 | Data migration method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111177113B (en) | 2023-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10534773B2 (en) | Intelligent query parameterization of database workloads | |
| CN109542399B (en) | Software development method and device, terminal equipment and computer readable storage medium | |
| CN111177113B (en) | Data migration method, device, computer equipment and storage medium | |
| WO2019140828A1 (en) | Electronic apparatus, method for querying logs in distributed system, and storage medium | |
| CN107239392B (en) | Test method, test device, test terminal and storage medium | |
| CN108536745B (en) | Shell-based data table extraction method, terminal, equipment and storage medium | |
| WO2021217846A1 (en) | Interface data processing method and apparatus, and computer device and storage medium | |
| CN110955409B (en) | Method and device for creating resources on cloud platform | |
| WO2019104891A1 (en) | Method and device for importing and exporting report, storage medium, and terminal | |
| CN110659210A (en) | Information acquisition method and device, electronic equipment and storage medium | |
| WO2019161620A1 (en) | Application dependency update method, terminal and device, and storage medium | |
| WO2019161618A1 (en) | Character string parsing method, apparatus and device, and computer-readable storage medium | |
| US20070180531A1 (en) | Analyzing interpretable code for harm potential | |
| CN113010542B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN113238739A (en) | Plug-in development and data acquisition method, device, electronic equipment and medium | |
| CN117033249A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN112883088B (en) | Data processing method, device, equipment and storage medium | |
| CN112363814A (en) | Task scheduling method and device, computer equipment and storage medium | |
| US9201937B2 (en) | Rapid provisioning of information for business analytics | |
| US20230023290A1 (en) | Method for managing function based on engine, electronic device and medium | |
| CN111666201A (en) | Regression testing method, device, medium and electronic equipment | |
| CN114968725A (en) | Task dependency relationship correction method and device, computer equipment and storage medium | |
| CN110262856B (en) | Application program data acquisition method, device, terminal and storage medium | |
| CN107506299B (en) | Code analysis method and terminal equipment | |
| CN114579405A (en) | Data processing method, data processing apparatus, electronic device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |