CN115934684A - Multi-source database data migration method, device, equipment and storage medium - Google Patents
Multi-source database data migration method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115934684A CN115934684A CN202310243043.1A CN202310243043A CN115934684A CN 115934684 A CN115934684 A CN 115934684A CN 202310243043 A CN202310243043 A CN 202310243043A CN 115934684 A CN115934684 A CN 115934684A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- preset
- different types
- preset standard
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The present disclosure relates to a method and an apparatus for migrating multi-source database data, a device and a storage medium, wherein the method comprises: acquiring data of a first database, and providing a script file for the data of the first database; responding to the matching of the provided script file and the data of the first database, and extracting the specified different types of data according to the preset data extraction sequence in the script file; providing a first preset standard rule matched with the data of the first database, and comparing the data in the first preset standard rule with the extracted data of different types; according to the preset data extraction sequence and the difference information between the first database and the second database, the extracted different types of data meeting the first preset standard rule are converted into the data of the second database, a data migration set solution after systematic integration is provided, the problems of difficult data migration adaptation and the like are fundamentally solved, errors are reduced, omission is avoided, and the reconstruction cost is reduced.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a method and an apparatus for migrating multi-source database data, a device, and a storage medium.
Background
The database cannot be copied, moved and sent in a service running state, and legal migration operation needs to be carried out on the target database if necessary.
In the related art, a legal migration operation can be performed on a target database through the following two methods: the first is that a target database required by a service is deployed, data is imported into the target database from other source databases, and the problems of incompatibility, function non-support, code writing method and the like, such as syntax compatibility, function compatibility, whether a user-defined function is supported or not, and the problem of adaptation of a storage process, are solved in sequence; and secondly, installing a visual software tool of the configuration database, exporting through a visual Web interface, importing data, converting part of formats and other operations, sequentially checking in modes of codes, self-research and development tools, scripts and the like, and solving the problem of syntax compatibility of the structured query language.
However, the above two methods have the following drawbacks: the data migration is long in time consumption, the workload of adapting data transformation of the database is large, various problems such as the structured query language compatibility, the grammar compatibility, the function support and the change of the view storage process of the corresponding service system of the database under different service environments are solved, the transformation cost is very high, the transformation cost is very complex, the omission is easy, a large amount of manpower and material resources are required to be invested, and project delay and activities cannot be developed on schedule due to the difficulty in mastering the technical process of technicians with different technical levels.
Therefore, how to realize the multi-source database data migration adaptation quickly, efficiently and smoothly is a problem to be solved urgently.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, embodiments of the present disclosure provide a multi-source database data migration method and apparatus, a device, and a storage medium.
In a first aspect, an embodiment of the present disclosure provides a multi-source database data migration method, including:
acquiring data of a first database, and providing a script file for the data of the first database;
responding to the matching of the provided script file and the data of the first database, and extracting the specified different types of data according to the preset data extraction sequence in the script file;
providing a first preset standard rule matched with the data of the first database, and comparing the data in the first preset standard rule with the extracted data of different types;
and converting the extracted different types of data which accord with the first preset standard rule into the data of the second database according to the preset data extraction sequence and the difference information between the first database and the second database.
In one possible embodiment, the method further comprises:
establishing a copy set of the data of the first database, and comparing the copy set data with the extracted data of different types;
in response to the extracted different types of data not being consistent with the data of the first database, re-extracting the different types of data specified in the data of the first database.
In one possible embodiment, the extracted different types of data include at least two of structured query language data, database built-in functions, custom functions, views, stored procedures, and index structure data.
In one possible embodiment, the extracted different types of data include index structure data, and the method further includes:
determining the number of pieces of index structure data;
and splitting the index structure data in response to the number of the index structure data exceeding the preset number so as to convert the split index structure data according to a preset splitting sequence.
In a possible embodiment, the providing of the preset standard rule matching with the data of the first database includes:
extracting characteristic parameters of the data of the first database;
and according to the corresponding relation between the characteristic parameters and at least two preset standard rules, taking the preset standard rule corresponding to the characteristic parameters as the preset standard rule matched with the data of the first database.
In a possible embodiment, the providing a script file for the data of the first database includes:
extracting preset keywords from the data of the first database;
and determining the script file matched with the data of the first database according to the extracted preset keywords.
In one possible embodiment, the method further comprises:
and correcting the extracted different types of data which do not accord with the first preset standard rule until the corrected data accord with the first preset standard rule, and generating a data migration report.
In a second aspect, an embodiment of the present disclosure provides a multi-source database data migration apparatus, including:
the acquisition module is used for acquiring data of a first database and providing script files for the data of the first database;
the extraction module is used for responding to the matching of the provided script file and the data of the first database and extracting the appointed different types of data according to the preset data extraction sequence in the script file;
the comparison module is used for providing a first preset standard rule matched with the data of the first database and comparing the data in the first preset standard rule with the extracted data of different types;
and the conversion module is used for converting the extracted different types of data which accord with the first preset standard rule into the data of the second database according to the preset data extraction sequence and the difference information between the first database and the second database.
In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing the multi-source database data migration method when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the multi-source database data migration method described above.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure at least has part or all of the following advantages:
the multi-source database data transplanting method of the embodiment of the disclosure obtains data of a first database, and provides script files for the data of the first database; responding to the matching of the provided script file and the data of the first database, and extracting the specified different types of data according to the preset data extraction sequence in the script file; providing a first preset standard rule matched with the data of the first database, and comparing the data in the first preset standard rule with the extracted data of different types; according to the preset data extraction sequence and the difference information between the first database and the second database, the extracted different types of data meeting the first preset standard rule are converted into the data of the second database, a data migration set solution after systematic integration is provided, the problems of difficult data migration adaptation and the like are fundamentally solved, mistakes are reduced, omission is reduced, the reconstruction cost is reduced, the reconstruction pain point problems of different source database reconstruction processes and rear-end services are reduced, meanwhile, a plurality of differentiated contents existing in the adaptation process and the transplantation process can be correspondingly recorded and summarized and analyzed, a corresponding solution is provided for the problems existing in the adaptation process, and the problems existing in the current different data source data migration adaptation process are fundamentally solved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the related art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
Fig. 1 schematically shows a flow diagram of a multi-source database data migration method according to an embodiment of the present disclosure;
fig. 2 schematically shows a block diagram of a multi-source database data migration apparatus according to an embodiment of the present disclosure; and
fig. 3 schematically shows a block diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some embodiments of the present disclosure, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
Referring to fig. 1, an embodiment of the present disclosure provides a multi-source database data migration method, including:
s1, acquiring data of a first database, and providing a script file for the data of the first database;
in some embodiments, the first database may be an Oracle database, the second database may be a MYSQL database, and a script file provided for the data of the first database is used to extract different types of data in the Oracle database according to a preset data extraction sequence. The data of the first database can be obtained by the timing task according to the data path of the first database.
In some embodiments, the first database may be a MYSQL database and the second database is an Oracle database.
In some embodiments, in the case where the first database is an Oracle database, export data is migrated from the Oracle database through a SQLPLUS tool, wherein the SQLPLUS tool is a client tool interacting with the Oracle database, by means of which the database records can be viewed, modified, and in which SQLPLUS commands and sql statements can be run.
S2, in response to the fact that the provided script file is matched with the data of the first database, extracting the specified different types of data according to a preset data extraction sequence in the script file;
in some embodiments, the extracted different types of data include at least two of structured query language data, database built-in functions, custom functions, views, stored procedures, and index structure data.
In some embodiments, each type of data is extracted separately based on a script file conforming to PT (Percona Toolkit, which is a tool developed by Percona corporation for managing MySQL, and the functions include checking data consistency of master-slave copy, checking repeated indexes, locating a table file with high input/output occupancy, online DDL, and the like) tool specifications, so as to split data and archive data streams, so that the efficiency of later data transformation and conversion is very efficient, and the error rate is reduced.
In some embodiments, the data extraction is implemented by encapsulating a function method and referring to the name of the extracted data, and the specific flow of the data extraction and storage process is, for example, as follows:
call P1 store procedure through the custom. Callproc () function, passing in 4 parameters
Returning the obtained set, namely the SELECT FROM tmp in the storage function;
obtain the returned value in Python fixed format: @ stores the process name 0, the first return value.
S3, providing a first preset standard rule matched with the data of the first database, and comparing the data in the first preset standard rule with the extracted data of different types;
in some embodiments, the preset standard rules may be database official standards.
In some embodiments, comparing the data in the first preset standard rule with the extracted different types of data includes:
and comparing the data in the first preset standard rule with the current data for each kind of data in the extracted different types of data based on a preset comparison rule in the script file.
In some embodiments, the comparison procedure is as follows:
the automatic comparison specification of the built-in database is accessed to an official standard specification and a collected local specification to serve as a first preset standard rule;
dividing the first preset standard rule into a plurality of checking modules according to the comparison content, wherein the checking modules at least comprise SQL legality modules;
calling each checking module, extracting the content information of the current checking module and extracting different types of data to perform automatic comparison checking, for example, calling a syntax checking of an SQL validity module;
and outputting the data difference obtained by comparison to provide a basis for subsequent data conversion.
And S4, converting the extracted different types of data which accord with the first preset standard rule into data of the second database according to the preset data extraction sequence and the difference information between the first database and the second database.
In some embodiments, converting the extracted different types of data meeting the first preset standard rule into the data of the second database according to the difference information between the first database and the second database in the preset data extraction sequence, including:
and based on a conversion rule preset in the script file, converting the format, SQL writing logic rules, function functions and index structures of the extracted different types of data which accord with the first preset standard rule, and determining the compatibility, function support and storage process adaptability of SQL grammar, functions, self-defined functions and code writing methods, so that the converted data is adaptive to the data of the second database.
In some embodiments, case rule conversion may be performed using the lower () function method.
In some embodiments, the difference information between the first database and the second database includes, but is not limited to, differences in data types and differences in stored procedures.
In some embodiments, in the case that the difference information between the first database and the second database is a difference of data types, and the first database is oracle and the second database is mysql, the extracted different types of data meeting the first preset standard rule are converted into the data of the second database, including:
encapsulating a data type conversion function by using python, wherein the data type conversion function is a place function and is used for converting the data type of oracle data into the data type of mysql data;
converting the data type of the mysql data into the data type and the value range which accord with the mysql standard rule by using a blob () method;
and merging the data types and the value ranges which accord with the mysql standard rule into a standard SQL code.
In some embodiments, in the case that the difference information between the first database and the second database is a difference of a stored process, and the first database is oracle and the second database is mysql, the extracted different types of data meeting the first preset standard rule are converted into the data of the second database, including:
the Oracle PL/SQL (SQL procedure language extensions) command for creating stored procedures and functions contains an optional OR REPLACE clause, modified to a stored procedure in MySQL: using DROP PROCEDURE and then using createprecocure statement;
when creating a MySQL stored procedure or function, the code must specify other delimiters (semicolons) that are not default delimiters ". Because MySQL would treat every line ending with ";" as a new line, we suggest you
When creating a MySQL stored procedure or function, the non-default separator in Oracle "; (semicolon), the non-default separator is specified in MySQL, other separators of" (semicolon), such as using a different separator (e.g., "$") to resolve all stored procedures.
In this embodiment, the method further includes:
recording the converted data of the second database into the second database based on the script file;
recording and analyzing problems existing in the recording process so as to redefine a conversion rule according to the problems;
and according to the redefined conversion rule, converting the data of the first database again, and re-inputting the data of the second database after conversion into the second database again.
In this embodiment, entering the data of the converted second database into the second database includes:
backing up the data of the converted second database;
recording the backed-up data of the second database into the second database;
automatically checking the data volume, the data table, the fields, the index structure and the function of the input data of the second database;
and generating a data migration adaptation report according to the result of the automatic verification, wherein the data migration adaptation report comprises adaptation states of different types of data in the data of the second database.
In this embodiment, data may be extracted from the first database by way of variable parameters, or may be entered into the second database.
In this embodiment, the entered data is converted data, such as a converted SQL script.
In some embodiments, there are two types of logging: firstly, the verified data is directly written into a target database through a PYTHON code, so that the efficiency is higher, the error rate is lower, the importing operation is more standard, the time is saved, the efficiency is higher, the error rate is lower, the importing operation is more standard, and the time is saved; and secondly, the SQL script can be imported through the MYSQL client by using the SQL script with the program production specification, wherein the size of the imported data volume script can be reasonably cut according to the size of the data volume, and the imported data volume script is imported in batches. The method for directly writing Python codes into specified data comprises the following steps: and customizing an import data function, and directly importing the data into a target database after formatting the data through a splicing function.
In this embodiment, the method further includes:
extracting business data corresponding to the data of the first database based on the SQL command in the script file;
comparing the data volume of the data input into the second database with the data volume of the service data;
responding to the data volume inconsistency between the data recorded into the second database and the business data, and recording and outputting the data with the difference;
and adjusting the script file according to the different data until the data recorded into the second database is consistent with the data volume of the business data.
In this embodiment, the method further includes:
and comparing the data in the second preset standard rule with the converted data based on a second preset standard rule matched with the second database to determine the adaptability of the converted data to the second database.
In this embodiment, the method further includes:
establishing a copy set of the data of the first database, and comparing the data of the copy set with the extracted data of different types, wherein under the condition that the index table where the extracted data of different types is located is a split index table, the data of the copy set is compared with the extracted data of different types after the copy set is split according to the splitting rule of the index table;
in response to the extracted different types of data not being consistent with the data of the first database, re-extracting the different types of data specified in the data of the first database;
and responding to the fact that the extracted different types of data are consistent with the data of the first database, and not responding.
In this embodiment, the extracted different types of data include index structure data, and the method further includes:
determining the number of pieces of index structure data;
and splitting the index structure data in response to the number of the index structure data exceeding the preset number so as to convert the split index structure data according to a preset splitting sequence.
In this embodiment, the extracted different types of data include index structure data, and the method further includes:
determining a storage space occupied by index structure data;
and splitting the index structure data in response to the fact that the storage space occupied by the index structure data exceeds a preset storage space, so that the split index structure data is converted according to a preset splitting sequence.
In this embodiment, in step S3, the providing a preset standard rule matched with the data of the first database includes:
extracting characteristic parameters of the data of the first database;
and according to the corresponding relation between the characteristic parameters and at least two preset standard rules, taking the preset standard rule corresponding to the characteristic parameters as the preset standard rule matched with the data of the first database.
In this embodiment, in step S1, the providing a script file for the data of the first database includes:
extracting preset keywords from the data of the first database;
and determining a script file matched with the data of the first database according to the extracted preset keywords.
In this embodiment, the method further includes:
and correcting the extracted different types of data which do not accord with the first preset standard rule until the corrected data accord with the first preset standard rule, and generating a data migration report.
In some embodiments, the extracted different types of data that do not comply with the first preset standard rule are obtained by:
performing classified and modularized storage on the classified and extracted data to obtain various kinds of stored data;
each kind of stored data is compared with a first preset standard rule to obtain data which do not accord with the first preset standard rule, and the data are used for modifying the data which do not accord with the first preset standard rule in the next step
In some embodiments, the data that does not comply with the first predetermined standard rule is an illegal string or data that is illegal in case and case syntax, and the like.
In the embodiment, the content of the data which does not conform to the first preset standard rule is processed in a flow-based and modularized manner with high efficiency, the data which does not conform to the first preset standard rule is processed without error, high-efficiency data migration can be ensured, more complex service data stream processing modules can be transplanted and grafted, and the flow-based and modularized data stream processing method has the advantages of solving the migration problem with high efficiency in a flow-based and coded manner.
Taking the example of transplanting the data of the Oracle database to the MYSQL database, the multi-source database data transplanting method disclosed by the invention is further explained as follows:
extracting core service indexes of a MYSQL database, wherein the core service indexes comprise: functions, stored procedures, views, index structures, custom functions, functional methods, and the like;
determining a preset script matched with Oracle database data;
extracting functions, storage processes, views, index structures, custom functions, functional methods and the like in Oracle database data according to an extraction sequence in a preset script to serve as extraction data;
determining official standard rules matched with Oracle database data, and respectively checking the extracted functions, the stored process, the view, the index structure, the custom functions, the functional method and the like according to comparison rules in a preset script;
according to conversion rules in a preset script, performing compatibility adjustment conversion on the format for extracted data meeting official standard rules, adapting and rewriting SQL writing logic rules and function support, reconstructing index structure data, adapting service environment functions, verifying data accuracy and consistency, and checking service integration test to obtain data suitable for an MYSQL database;
and adding a tool for forcibly converting the extracted data for the extracted data which does not meet the official standard until all the extracted data which does not meet the official standard are converted into data suitable for the MYSQL database.
The method for transplanting the data of the multi-source database has the advantages that for different original database sources, a plurality of problems exist in the data migration adaptation process, time and labor are consumed, resources are spent to solve the data homology problem, the data of the first database are corrected through the preset standard, the data of the converted second database are corrected, in the process that the data of the converted second database are recorded into the second database, a data migration report is generated, the preset script file is adjusted according to the data migration report, and the data of the second database converted based on the preset script file is enabled to be more adapted to the second database.
Referring to fig. 2, an embodiment of the present disclosure provides a multi-source database data migration apparatus, including:
the acquisition module 21 is configured to acquire data of a first database and provide a script file for the data of the first database;
the extraction module 22 is used for extracting the data of different specified types according to a preset data extraction sequence in the script file in response to the matching of the provided script file and the data of the first database;
the comparison module 23 is configured to provide a first preset standard rule matched with the data of the first database, and compare the data in the first preset standard rule with the extracted data of different types;
and the conversion module 24 is configured to convert the extracted different types of data meeting the first preset standard rule into data of the second database according to a preset data extraction sequence and according to the difference information between the first database and the second database.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the invention. One of ordinary skill in the art can understand and implement it without inventive effort.
In the second embodiment, any multiple of the obtaining module 21, the extracting module 22, the comparing module 23 and the converting module 24 may be combined and implemented in one module, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. At least one of the obtaining module 21, the extracting module 22, the comparing module 23 and the converting module 24 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementations of software, hardware and firmware, or a suitable combination of any several of them. Alternatively, at least one of the obtaining module 21, the extracting module 22, the comparing module 23 and the converting module 24 may be at least partly implemented as a computer program module which, when executed, may perform a corresponding function.
Referring to fig. 3, an electronic device provided by an embodiment of the present disclosure includes a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, where the processor 1110, the communication interface 1120, and the memory 1130 complete communication with each other through the communication bus 1140;
a memory 1130 for storing computer programs;
the processor 1110 is configured to implement the following multi-source database data migration method when executing the program stored in the memory 1130:
acquiring data of a first database, and providing a script file for the data of the first database;
responding to the matching of the provided script file and the data of the first database, and extracting the specified different types of data according to the preset data extraction sequence in the script file;
providing a first preset standard rule matched with the data of the first database, and comparing the data in the first preset standard rule with the extracted data of different types;
and converting the extracted different types of data which accord with the first preset standard rule into the data of the second database according to a preset data extraction sequence and the difference information between the first database and the second database.
The communication bus 1140 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus 1140 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this is not intended to represent only one bus or type of bus.
The communication interface 1120 is used for communication between the electronic device and other devices.
The memory 1130 may include a Random Access Memory (RAM) or a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory 1130 may also be at least one memory device located remotely from the processor 1110.
The Processor 1110 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
A fifth exemplary embodiment of the present disclosure also provides a computer-readable storage medium based on the same inventive concept. The computer readable storage medium stores thereon a computer program, which when executed by a processor implements the multi-source database data migration method as described above.
The computer-readable storage medium may be contained in the apparatus/device described in the above embodiments; or may be present alone without being assembled into the device/apparatus. The above-mentioned computer-readable storage medium carries one or more programs which, when executed, implement the multi-source database data migration method according to an embodiment of the present disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
It is noted that, in this document, relational terms such as "first" and "second," and the like, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A multi-source database data migration method, the method comprising:
acquiring data of a first database, and providing a script file for the data of the first database;
responding to the matching of the provided script file and the data of the first database, and extracting the specified different types of data according to the preset data extraction sequence in the script file;
providing a first preset standard rule matched with the data of the first database, and comparing the data in the first preset standard rule with the extracted data of different types;
and converting the extracted different types of data which accord with the first preset standard rule into the data of the second database according to the preset data extraction sequence and the difference information between the first database and the second database.
2. The method of claim 1, further comprising:
establishing a copy set of the data of the first database, and comparing the copy set data with the extracted data of different types;
in response to the extracted different types of data not being consistent with the data of the first database, re-extracting the different types of data specified in the data of the first database.
3. The method of claim 1, wherein the extracted different types of data comprise at least two of structured query language data, database built-in functions, custom functions, views, stored procedures, and index structure data.
4. The method of claim 1, wherein the extracted different types of data comprise index structure data, the method further comprising:
determining the number of pieces of index structure data;
and splitting the index structure data in response to the number of the index structure data exceeding the preset number so as to convert the split index structure data according to a preset splitting sequence.
5. The method of claim 1, wherein providing the preset standard rule matching the data of the first database comprises:
extracting characteristic parameters of the data of the first database;
and according to the corresponding relation between the characteristic parameters and at least two preset standard rules, taking the preset standard rule corresponding to the characteristic parameters as the preset standard rule matched with the data of the first database.
6. The method of claim 1, wherein providing a script file for the data of the first database comprises:
extracting preset keywords from the data of the first database;
and determining a script file matched with the data of the first database according to the extracted preset keywords.
7. The method of claim 1, further comprising:
and correcting the extracted different types of data which do not accord with the first preset standard rule until the corrected data accord with the first preset standard rule, and generating a data migration report.
8. A multi-source database data migration apparatus, comprising:
the acquisition module is used for acquiring data of a first database and providing script files for the data of the first database;
the extraction module is used for responding to the matching of the provided script file and the data of the first database and extracting the appointed different types of data according to the preset data extraction sequence in the script file;
the comparison module is used for providing a first preset standard rule matched with the data of the first database and comparing the data in the first preset standard rule with the extracted data of different types;
and the conversion module is used for converting the extracted different types of data which accord with the first preset standard rule into the data of the second database according to the preset data extraction sequence and the difference information between the first database and the second database.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the multi-source database data migration method of any one of claims 1-7 when executing the program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the multi-source database data migration method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310243043.1A CN115934684B (en) | 2023-03-14 | 2023-03-14 | Multi-source database data migration method and device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310243043.1A CN115934684B (en) | 2023-03-14 | 2023-03-14 | Multi-source database data migration method and device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115934684A true CN115934684A (en) | 2023-04-07 |
| CN115934684B CN115934684B (en) | 2023-06-30 |
Family
ID=86651107
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310243043.1A Active CN115934684B (en) | 2023-03-14 | 2023-03-14 | Multi-source database data migration method and device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115934684B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160019242A1 (en) * | 2014-07-15 | 2016-01-21 | Makesh Balasubramanian | Migrating Federated Data to Multi-Source Universe Database Environment |
| US20160063019A1 (en) * | 2014-09-03 | 2016-03-03 | Bank Of America Corporation | Script converter |
| CN112749151A (en) * | 2021-01-13 | 2021-05-04 | 叮当快药科技集团有限公司 | Data migration method and device among different types of databases and storage medium |
| CN115408111A (en) * | 2022-09-27 | 2022-11-29 | 建信金融科技有限责任公司 | Database script control method, system, device, storage medium and program product |
| CN115757481A (en) * | 2022-11-30 | 2023-03-07 | 中国农业银行股份有限公司 | Data migration method, device, equipment and storage medium |
-
2023
- 2023-03-14 CN CN202310243043.1A patent/CN115934684B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160019242A1 (en) * | 2014-07-15 | 2016-01-21 | Makesh Balasubramanian | Migrating Federated Data to Multi-Source Universe Database Environment |
| US20160063019A1 (en) * | 2014-09-03 | 2016-03-03 | Bank Of America Corporation | Script converter |
| CN112749151A (en) * | 2021-01-13 | 2021-05-04 | 叮当快药科技集团有限公司 | Data migration method and device among different types of databases and storage medium |
| CN115408111A (en) * | 2022-09-27 | 2022-11-29 | 建信金融科技有限责任公司 | Database script control method, system, device, storage medium and program product |
| CN115757481A (en) * | 2022-11-30 | 2023-03-07 | 中国农业银行股份有限公司 | Data migration method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115934684B (en) | 2023-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10353913B2 (en) | Automating extract, transform, and load job testing | |
| CN109062780B (en) | Development method of automatic test case and terminal equipment | |
| CN110737594B (en) | Database standard conformance testing method and device for automatically generating test cases | |
| CN110275861B (en) | Data storage method and device, storage medium and electronic device | |
| CN105205053A (en) | Method and system for analyzing database incremental logs | |
| CN110990274B (en) | Data processing method, device and system for generating test cases | |
| WO2016026328A1 (en) | Information processing method and device and computer storage medium | |
| CN111190807B (en) | Buried point testing method and equipment | |
| CN104461531A (en) | Implementing method for self-defined functions of reporting system | |
| CN111667231B (en) | Automatic tax return method, device, system, computer equipment and storage medium | |
| WO2022012327A1 (en) | Code analysis method and system, and computing device | |
| CN111061733B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN112631920A (en) | Test method, test device, electronic equipment and readable storage medium | |
| CN115934684B (en) | Multi-source database data migration method and device, equipment and storage medium | |
| CN115757174A (en) | Database difference detection method and device | |
| CN112114794B (en) | Automatic generation method and device of website application program and computer storage medium | |
| CN114661715A (en) | Database data processing method and device, storage medium and electronic equipment | |
| CN104216986A (en) | Device and method for improving data query efficiency through pre-operation according to data update period | |
| CN116226788B (en) | Modeling method integrating multiple data types and related equipment | |
| CN113050925B (en) | Block chain intelligent contract repairing method and device | |
| CN112764740B (en) | Shell-based processing method applied to file interaction between systems | |
| CN110688144B (en) | Method and device for optimizing service interface configuration and electronic equipment | |
| CN116010349A (en) | Metadata-based data checking method and device, electronic equipment and storage medium | |
| CN116069735A (en) | Method and system for generating metadata report | |
| CN116204509A (en) | Data access script generation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |