CN113672591A

CN113672591A - Data migration method, system, storage medium and electronic device

Info

Publication number: CN113672591A
Application number: CN202110848424.3A
Authority: CN
Inventors: 汪月
Original assignee: Beijing Minglue Zhaohui Technology Co Ltd
Current assignee: Beijing Minglue Zhaohui Technology Co Ltd
Priority date: 2021-07-27
Filing date: 2021-07-27
Publication date: 2021-11-19

Abstract

The application discloses a data migration method, a system, a storage medium and an electronic device, wherein the data migration method comprises the following steps: the collection step comprises: collecting link information of a data source table and link information of a target table; the processing steps are as follows: performing data synchronization on the data source table and the target table through spark according to the link information of the data source table and the link information of the target table; and (3) a checking step: checking the synchronously completed data; dag construction step: and constructing an operation interface based on dag according to the data source table, the migration device and the target table. The invention provides a data method, which can simplify the complexity of data cross-platform synchronization, so that the data migration can be realized in an interface and dragging and pulling mode.

Description

Data migration method, system, storage medium and electronic device

Technical Field

The invention belongs to the field of data migration, and particularly relates to a data migration method, a data migration system, a storage medium and electronic equipment.

Background

In the big data era, data becomes more and more important and more indispensable, platforms for storing data are various, and the problem of how to realize the synchronization of data among different platforms arises. The patent provides a data migration device based on DAG (directed acyclic graph), and the migration of cross-platform data is carried out through a flexible dragging mode to this patent, also can provide very friendly function to the people who does not have the programming basis, can realize data migration through the interface. The difficulty of performing cross-platform data synchronization is greatly reduced.

Disclosure of Invention

The embodiment of the application provides a data migration method, a data migration system, a storage medium and electronic equipment, and at least solves the problem that the existing data migration method cannot solve direct data transmission of different platforms.

The invention provides a data migration method, which comprises the following steps:

the collection step comprises: collecting link information of a data source table and link information of a target table;

the processing steps are as follows: performing data synchronization on the data source table and the target table through spark according to the link information of the data source table and the link information of the target table;

and (3) a checking step: and checking the data after synchronization.

The data migration method further includes:

dag construction step: and constructing an operation interface based on dag according to the data source table, the migration device and the target table.

The data migration method comprises the following processing steps:

an adjusting step: adjusting and adjusting the data fragments according to the size of the data to be migrated;

a reading step: reading the adjusted data in a jdbc mode;

a writing step: and performing the slow and sufficient batch writing on the read data by adopting the jdbc mode.

The data migration method comprises the following steps: and synchronizing the link information of the data source table and the link information of the target table.

The invention also provides a data migration system, which comprises:

the acquisition module acquires the link information of the data source table and the link information of the target table;

the processing module is used for carrying out data synchronization on the data source table and the target table through spark according to the link information of the data source table and the link information of the target table;

a verification module that verifies the synchronously completed data.

The data migration system further includes:

dag, the Dag build module builds a dag-based operation interface from the data source table, migration device, and the target table.

The data migration system, wherein the processing module includes:

the adjusting unit is used for adjusting and adjusting the data fragments according to the size of the data to be migrated;

the reading unit reads the adjusted data in a jdbc mode;

and the writing unit is used for performing the slow and sufficient batch writing on the read data in the jdbc mode.

The data migration system, wherein the collection module includes: and synchronizing the link information of the data source table and the link information of the target table.

The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data migration method as described in any of the above when executing the computer program.

The invention also provides a storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements a data migration method as described in any of the above.

The invention has the beneficial effects that:

the invention belongs to the field of data transmission in data capacity technology. The method can simplify the complexity of data cross-platform synchronization, so that the data migration can be realized in an interface and dragging mode.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application.

In the drawings:

FIG. 1 is a flow chart of a data migration method of the present invention;

FIG. 2 is a flow chart illustrating the substeps of step S2 in FIG. 1;

FIG. 3 is a flow chart of the present invention;

FIG. 4 is a diagram of the build dag and interface operations;

FIG. 5 is a diagram of a related configuration of the present invention;

FIG. 6 is a schematic diagram of the structure of the data migration system of the present invention;

fig. 7 is a frame diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.

It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.

The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that functional, methodological, or structural equivalents or substitutions made by these embodiments are within the scope of the present invention.

Before describing in detail the various embodiments of the present invention, the core inventive concepts of the present invention are summarized and described in detail by the following several embodiments.

The first embodiment is as follows:

referring to fig. 1, fig. 1 is a flowchart of a data migration method. As shown in fig. 1, the data migration method of the present invention includes:

a collection step S1: collecting link information of a data source table and link information of a target table;

processing step S2: performing data synchronization on the data source table and the target table through spark according to the link information of the data source table and the link information of the target table;

checking step S3: checking the synchronously completed data;

dag construction step S4: and constructing an operation interface based on dag according to the data source table, the migration device and the target table.

Referring to fig. 2, fig. 2 is a flowchart of the processing step S2. As shown in fig. 2, the processing step S2 includes:

adjustment step S21: adjusting and adjusting the data fragments according to the size of the data to be migrated;

reading step S22: reading the adjusted data in a jdbc mode;

write step S23: and performing the slow and sufficient batch writing on the read data by adopting the jdbc mode.

And synchronizing the link information of the data source table and the link information of the target table.

Specifically, the purpose of the patent is to simplify the complexity of data cross-platform synchronization, so that data migration can be realized in a mode of interfacing, dragging and pulling, the key point of the patent is to construct a DAG, efficiently realize data synchronization of different platforms, support multiple types of platforms, realize data synchronization among multiple mainstream databases (live clickhouse mysql and the like), and achieve a fast migration rate, and take a live data synchronization clickhouse as an example, and can process 1000 ten thousand of data in 10 minutes.

The specific and complete technical scheme of the invention is as shown in fig. 3, fig. 4 and fig. 5, and the specific steps are as follows:

firstly, preparation work before data migration is carried out, link information of a data source is acquired, wherein the link information comprises information of a table name database, fields, full increment and the like to be synchronized, the data source is of which type, then link information of a target table to be synchronized is acquired, the link information comprises information of the table name database, the fields and the like, and then data synchronization between the data source table and the target table is carried out through spark. And the synchronized data is checked.

The data synchronization method comprises the steps that a front-stage preparation layer is used for synchronizing information of a source and information of a target before data synchronization is mainly carried out, a data processing layer adopts spark to carry out data synchronization, when the spark is used for carrying out data synchronization, adjustment of data fragmentation is carried out according to the size of data, reading and writing of the data are carried out in a jdbc mode, inserted data of a slow and sufficient batch are carried out, and frequent data insertion is prevented.

Further, Spark provides a large number of libraries including Spark Core, Spark SQL, Spark Streaming, MLlib, GraphX. Developers can use these libraries seamlessly in combination in the same application; spark supports Hadoop YARN, Apache messos, and their own independent cluster managers.

And Shark: shark basically provides a HiveQL command interface as same as Hive on the basis of a Spark framework, Spark uses an API of Hive to realize query matching and Logic Plan generation for the purpose of maintaining compatibility with Hive to the maximum extent, and Spark replaces Hadoop MapReduce in the last physical Plan execution stage. By configuring the Shark parameter, the Shark can automatically cache a specific RDD in the memory, so that data reuse is realized, and retrieval of a specific data set is accelerated. Meanwhile, the Shark realizes a specific data analysis learning algorithm through a UDF user-defined function, so that SQL data query and operation analysis can be combined together, and the repeated use of RDD is maximized.

SparkR: SparkR is an R packet that provides a lightweight spare front for R. SparkR provides a distributed data frame data structure, which solves the bottleneck that the data frame in R can only be used in a single machine, and supports many operations like select, filter, aggregate, etc. as the data frame in R. (like the function in a dplyr packet) this solves well the big data level bottleneck problem of R. SparkR also supports distributed machine learning algorithms, such as using the MLib machine learning library. [4] SparkR introduces the vitality of R language community for Spark, attracting a large number of data scientists to start the data analysis journey directly on Spark platform.

Still further, jdbc is a "low level" interface, that is, it is used to directly invoke SQL commands. In this respect it functions well and is easier to use than other database connection APIs, but it is also designed as a basic interface on which advanced interfaces and tools can be built. The high level interface is a "user friendly" interface that uses a more understandable and convenient API that is translated behind the scenes to a low level interface such as JDBC.

In the "object/relationship" mapping of a relational database, each row in the table corresponds to an instance of a class, and the value of each column corresponds to an attribute of the instance. Thus, the programmer can directly operate the Java object; the SQL calls required to access the data will be automatically generated "under the mask". More complex mappings may also be provided, such as combining rows in multiple tables into one Java class.

With increasing interest in JDBC, more and more developers have been using JDBC-based tools to make programming easier. Programmers have also been writing applications that attempt to make end-user access to databases simpler. For example, the application may provide a menu of tasks for selecting the database. After the task is selected, the application will present prompts and blanks for filling in the information needed to perform the selected task. The required information input application will automatically invoke the required SQL commands. With the aid of such a program, database tasks can be executed even if the user does not understand the syntax of SQL at all.

Still further, the DAG is the boundary of mailbox database replication, database and server switching and failover, and internal components named "activity managers". An activity manager running on each mailbox server, managing switchover and failover in the DAG; any server in the DAG may carry a copy of the mailbox database from any other server in the DAG. After a server is added to the DAG, the server, in cooperation with other servers in the DAG, provides the capability to automatically perform recovery from failures that affect the mailbox database (e.g., disk, server, or network failures).

CCR and SCR merge and have evolved into a more unified high availability architecture, where DAG becomes an essential component. That is, a DAG is used whether to deploy a local-level or site-level high-availability and disaster-recoverable solution.

In Exchange 2010, the only way to protect the mailbox database is also to use a DAG. The main component of the DAG is a new component called the Active Manager. Exchange cluster resource DLL (ex. DLL) and related cluster service resources in Exchange 2007 and earlier versions have now been replaced. Exchange 2010 now uses an active manager to manage data Exchange and failover between mailbox servers in the DAG. The active manager runs on all mailbox servers in a given DAG, and has two roles: a Primary Active Manager (PAM) and a Standby Active Manager (SAM).

Example two:

referring to fig. 6, fig. 6 is a schematic structural diagram of a data migration system according to the present invention. Fig. 6 shows a data migration system of the present invention, which includes:

a verification module that verifies the synchronously completed data.

Wherein, still include:

Wherein the processing module comprises:

the reading unit reads the adjusted data in a jdbc mode;

Wherein, the collection module comprises: and synchronizing the link information of the data source table and the link information of the target table.

Example three:

referring to fig. 7, this embodiment discloses an embodiment of an electronic device. The electronic device may include a processor 81 and a memory 82 storing computer program instructions.

Specifically, the processor 81 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 82 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 82 may include a Hard Disk Drive (Hard Disk Drive, abbreviated to HDD), a floppy Disk Drive, a Solid State Drive (SSD), flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 82 may include removable or non-removable (or fixed) media, where appropriate. The memory 82 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 82 is a Non-Volatile (Non-Volatile) memory. In particular embodiments, Memory 82 includes Read-Only Memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically rewritable ROM (EAROM), or FLASH Memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), where the DRAM may be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended data output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), and the like.

The memory 82 may be used to store or cache various data files for processing and/or communication use, as well as possible computer program instructions executed by the processor 81.

The processor 81 implements any of the data migration methods in the above embodiments by reading and executing computer program instructions stored in the memory 82.

In some of these embodiments, the electronic device may also include a communication interface 83 and a bus 80. As shown in fig. 7, the processor 81, the memory 82, and the communication interface 83 are connected via the bus 80 to complete communication therebetween.

The communication interface 83 is used for implementing communication between modules, devices, units and/or equipment in the embodiment of the present application. The communication port 83 may also be implemented with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.

The bus 80 includes hardware, software, or both to couple the components of the electronic device to one another. Bus 80 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 80 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

The electronic device may be based on data migration to implement the methods described in connection with fig. 1-2.

In addition, in combination with the data migration method in the foregoing embodiments, the embodiments of the present application may provide a computer-readable storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the data migration methods in the above embodiments.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

In summary, the beneficial effects of the invention are that the invention provides a data migration method, which can simplify the complexity of data cross-platform synchronization, so that data migration can be realized in an interfacing and dragging manner.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims

1. A method of data migration, comprising:

and (3) a checking step: and checking the data after synchronization.

2. The data migration method of claim 1, further comprising:

3. The data migration method of claim 1, wherein said processing step comprises:

a reading step: reading the adjusted data in a jdbc mode;

4. The data migration method of claim 1, wherein the collecting step comprises: and synchronizing the link information of the data source table and the link information of the target table.

5. A data migration system, comprising:

a verification module that verifies the synchronously completed data.

6. The data migration system of claim 5, further comprising:

7. The data migration system of claim 5, wherein the processing module comprises:

the reading unit reads the adjusted data in a jdbc mode;

8. The data migration system of claim 5, wherein the collection module comprises: and synchronizing the link information of the data source table and the link information of the target table.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the data migration method of any one of claims 1 to 4 when executing the computer program.

10. A storage medium on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the data migration method according to any one of claims 1 to 4.