CN112131208A

CN112131208A - Full data migration method, device and equipment and computer readable storage medium

Info

Publication number: CN112131208A
Application number: CN202011074417.4A
Authority: CN
Inventors: 何威; 吴双桥; 王珏
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Cloud Computing Beijing Co Ltd
Priority date: 2020-10-09
Filing date: 2020-10-09
Publication date: 2020-12-25

Abstract

The application provides a full data migration method, a device, equipment and a computer readable storage medium, which are applied to the technical field of cloud, and the method comprises the following steps: acquiring full data in a source database; the full amount of data includes at least one migration object; sequentially extracting a migration object from at least one migration object of the full amount of data as an object to be migrated, and migrating the object to be migrated from a source database to a target database through an idle migration process in at least one migration process; and under the condition that at least one migration object is migrated to the target database, completing the full data migration process. By the aid of the full data migration method, migration efficiency of the full data can be improved.

Description

Full data migration method, device and equipment and computer readable storage medium

Technical Field

The present application relates to data migration technologies, and in particular, to a method, an apparatus, a device, and a computer-readable storage medium for full data migration.

Background

A database is a repository that organizes, stores, and manages data according to a data structure. With the development of internet technology, in a scenario of upgrading hardware of a database server, etc., data of a database often needs to be migrated from one electronic device to another electronic device.

In the related art, in the process of transmitting the full data in the source database to the target database, the full data is exported to the external storage pool, and after the full data export is completed, the full data in the storage pool is imported into the target database. It can be seen that, in the related art, there are full export and full import processes for full data, and the full export must be completed before the full import can be started, and there is a global synchronization barrier, which seriously affects the migration efficiency of the full data.

Disclosure of Invention

The embodiment of the application provides a full data migration method, a full data migration device, full data migration equipment and a computer-readable storage medium, and migration efficiency of full data can be improved.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides a full data migration method, which comprises the following steps: a full-scale data migration method, comprising: acquiring full data in a source database; the full amount of data includes at least one migration object; sequentially extracting a migration object from at least one migration object of the full amount of data as an object to be migrated, and migrating the object to be migrated from a source database to a target database through an idle migration process in at least one migration process; and under the condition that at least one migration object is migrated to the target database, completing the full data migration process.

In some embodiments of the present application, said sequentially extracting a migration object from at least one migration object of the full amount of data as an object to be migrated, and migrating the object to be migrated from the source database to the target database through an idle migration process in the at least one migration process, includes: under the condition that an idle migration process exists in at least one migration process, extracting a migration object from at least one migration object as an object to be migrated; allocating an object to be migrated to an idle migration process; and migrating the object to be migrated from the source database to the target database through the idle migration process.

In some embodiments of the present application, migrating an object to be migrated from a source database to a target database through an idle migration process includes: taking out the object to be migrated from the source database through a reading thread in the idle migration process, and writing the object to be migrated into a data queue corresponding to the idle migration process in the memory; and taking out the object to be migrated from the data queue through a write-in thread in the idle migration process, and writing the object to be migrated into the target database.

In some embodiments of the present application, the fetching an object to be migrated from a source database through a read thread in an idle migration process, and writing the object to be migrated into a data queue corresponding to the idle migration process in a memory includes: reading the structural data of the object to be migrated from the source database to a data queue through a reading thread; and reading the source database data of the object to be migrated from the source database to the data queue through the reading process.

In some embodiments of the present application, the fetching an object to be migrated from a data queue by a write thread in an idle migration process, and writing the object to be migrated to a target database includes: taking out the structural data of the object to be migrated from the data queue through the write-in thread, and writing the structural data into the target database to complete the configuration process of the target database according to the structural data; and taking out the source database data of the object to be migrated from the data queue through the write-in thread, and writing the source database data into the target database configured by the structural data.

In some embodiments of the present application, the method further comprises: monitoring the process state of each migration process in at least one migration process; the process state comprises a read thread state and a write thread state; and determining an idle migration process in at least one migration process according to the read thread state and the write thread state corresponding to each migration process.

In some embodiments of the present application, the determining an idle migration process in at least one migration process according to the read thread state and the write thread state corresponding to each migration process includes at least one of: determining the migration process as an idle migration process under the condition that the read thread state and the write thread state corresponding to the migration process are both idle states; and determining the migration process as an idle migration process under the condition that the read thread state corresponding to the migration process is an idle state and the write thread states are working states.

In some embodiments of the present application, the method further comprises: in the process of taking out an object to be migrated from a source database through a reading thread in an idle migration process and writing the object to be migrated into a data queue corresponding to the idle migration process in a memory, if the data queue is full, suspending the reading thread; and in the process of taking out the object to be migrated from the data queue through a write-in thread in the idle migration process and writing the object to be migrated into the target database, acquiring a load quantization value in the data queue, and recovering the read thread under the condition that the load quantization value is lower than a preset threshold value.

The embodiment of the application provides a full data migration device, the device includes: the acquisition module is used for acquiring the full data in the source database; the full amount of data includes at least one migration object; the migration module is used for sequentially extracting the migration objects from at least one migration object of the full amount of data to serve as objects to be migrated, and migrating the objects to be migrated from the source database to the target database through an idle migration process in at least one migration process; and the migration module is also used for completing the full data migration process under the condition that at least one migration object is migrated to the target database.

The embodiment of the application provides a full data migration device, including:

a memory for storing executable instructions;

and the processor is used for realizing the full data migration method provided by the embodiment of the application when the executable instructions stored in the memory are executed.

The embodiment of the present application provides a computer-readable storage medium, which stores executable instructions for causing a processor to execute the method for migrating full data provided by the embodiment of the present application.

The embodiment of the application has the following beneficial effects:

according to the embodiment of the application, the total data are thinned to the plurality of migration objects, compared with the scheme that the total data are read into the read-write storage pool firstly and the total data are written into the target database after the reading process is completed in the prior art, the migration objects can be directly migrated into the target database respectively due to the thinning to the migration objects, so that the storage pool is not required to be read and written, the migration speed is increased, the storage expense is reduced, and the operation cost is reduced. The component dependence is reduced, and the overall reliability of the system is also improved. In addition, as the migration objects are migrated in parallel through the plurality of migration processes, the technical effect of simultaneously carrying out full export and full import on the full data is realized on the whole, and the overall migration efficiency of the full data is improved.

Drawings

FIG. 1 is an alternative architectural diagram of a full volume data migration system provided by an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a full-volume data migration device provided in an embodiment of the present application;

FIG. 3 is a schematic flow chart diagram illustrating an alternative method for migrating full-scale data according to an embodiment of the present application;

FIG. 4 is a schematic flow chart diagram illustrating an alternative method for migrating full-scale data according to an embodiment of the present application;

FIG. 5 is a schematic flow chart diagram illustrating an alternative method for migrating full-scale data according to an embodiment of the present application;

FIG. 6 is a schematic flow chart diagram illustrating an alternative method for migrating full-scale data according to an embodiment of the present application;

FIG. 7 is an alternative flow chart diagram of a full volume data migration method provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of an alternative migration process of the full volume data migration method according to the embodiment of the present application;

FIG. 9 is a schematic diagram of an interactive interface of an alternative database migration service provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of an alternative interactive interface secondary menu provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of an alternative migration process of the full volume data migration method according to the embodiment of the present application;

fig. 12 is a schematic diagram of an optional migration process of the full volume data migration method according to the embodiment of the present application.

Detailed Description

In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the attached drawings, the described embodiments should not be considered as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

In the following description, the terms "first \ second \ third" are used merely for distinguishing similar objects and do not represent specific ordering for the objects, and it is understood that "first \ second \ third" may be interchanged with specific order or sequence where permitted so that the embodiments of the present application described in the present embodiment can be implemented in an order other than that shown or described in the present embodiment.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the application.

First, terms referred to in the embodiments of the present application are briefly described:

(1) database (Database, DB): in short, it can be regarded as an electronic file cabinet, i.e. a place for storing electronic files, and a user can add, query, update, delete, etc. to data in the files. A "database" is a collection of data that is stored together in a manner that can be shared by multiple users, has as little redundancy as possible, and is independent of the application.

(2) DTS: data Transmission Service (Data Transmission Service), also called Data migration Service, is a Service supporting Data synchronization and Transmission among various isomorphic or heterogeneous databases.

(3) Synchronization Barrier (Barrier): is a synchronous method in parallel computing. For a group of processes or threads, one synchronization barrier in a program means that any thread/process execution must wait until then until all threads/processes reach this point before the context can continue to execute.

(4) And (3) full quantity derivation: existing structural data and data specifying the objects to be migrated are all exported from the source database.

(5) Introducing the total amount: and importing the information which is exported in full quantity into a target database.

(6) And (3) incremental derivation: in the process of database data migration, the source DB still has data written therein, and the part is incremental data, which needs to be exported.

(7) And (3) incremental import: and importing the exported increment data into the target DB.

Referring to fig. 1, fig. 1 is an alternative architecture diagram of a full data migration system 100 provided in this embodiment of the present application, in order to support a full data migration application, terminals (terminal 400-1 and terminal 400-2 are exemplarily shown) are connected to a server 200 through a network 300, and the network 300 may be a wide area network or a local area network, or a combination of both networks. Fig. 1 further shows that the server 200 may be a server cluster, where the server cluster includes servers 200-1 to 200-3, and similarly, the servers 200-1 to 200-3 may be physical machines, or virtual machines constructed by using virtualization technologies (such as container technology and virtual machine technology), which is not limited in this embodiment, and of course, a single server may also be used to provide services in this embodiment. The server 200 is also connected to a database 600, and the database 600 may be a database cluster including the databases 600-1 and 600-2.

In some embodiments of the present application, after receiving a full-volume data migration request of a user, a terminal determines a source database identifier and a target database identifier according to the full-volume data migration request, the terminal sends the source database identifier and the target database identifier to the server 200 through the network 300, and the server 200 determines, in the database 600, a source database 600-1 corresponding to the source database identifier and a target database 600-2 corresponding to the target database identifier according to the source database identifier and the target database identifier. The server 200 may obtain the full amount of data in the source database 600-1; the full amount of data includes at least two migration objects; sequentially extracting the migration objects from at least two migration objects of the full amount of data as objects to be migrated, and migrating the objects to be migrated from the source database 600-1 to the target database 600-2 through an idle migration process in at least one migration process; the full data migration process is completed with at least two migrated objects being migrated to the target database 600-2. In the migration process, the migration progress may be sent to the terminal. The terminal may display the migration progress of the full amount of data on a graphical interface (graphical interface 410-1 and graphical interface 410-2 are shown as examples).

The embodiments of the present application may be implemented by means of Cloud Technology (Cloud Technology), which refers to a hosting Technology for unifying series resources such as hardware, software, and network in a wide area network or a local area network to implement data calculation, storage, processing, and sharing.

The cloud technology is based on the general names of network technology, information technology, integration technology, management platform technology, application technology and the like applied in the cloud computing business model, can form a resource pool, is used as required, and is flexible and convenient. Cloud computing technology will become an important support. Background services of technical network systems require a large amount of computing, storage resources, such as web portals of educational systems.

As an example, the server 200 may be an independent physical server, may be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a web service, cloud communication, a middleware service, a domain name service, a security service, a CDN, and a big data and artificial intelligence platform. The terminal may be, but is not limited to, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal and the server 200 may be directly or indirectly connected through wired or wireless communication, and the embodiment of the present application is not limited thereto.

Referring to fig. 2, fig. 2 is a schematic structural diagram of a full volume data migration apparatus 500 according to an embodiment of the present application, where the full volume data migration apparatus 500 shown in fig. 2 includes: at least one processor 510, memory 550, at least one network interface 520, and a user interface 530. The various components in the full data migration apparatus 500 are coupled together by a bus system 540. It is understood that the bus system 540 is used to enable communications among the components. The bus system 540 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 540 in fig. 2.

The Processor 510 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor, or the like.

The user interface 530 includes one or more output devices 531 enabling presentation of media content, including one or more speakers and/or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components to facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The memory 550 may comprise volatile memory or nonvolatile memory, and may also comprise both volatile and nonvolatile memory. The non-volatile Memory may be a Read Only Memory (ROM), and the volatile Memory may be a Random Access Memory (RAM). The memory 550 described in embodiments herein is intended to comprise any suitable type of memory. Memory 550 optionally includes one or more storage devices physically located remote from processor 510.

In some embodiments of the present application, memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

An operating system 551 including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

a network communication module 552 for communicating to other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: bluetooth, wireless compatibility authentication (WiFi), and Universal Serial Bus (USB), etc.;

a display module 553 for enabling presentation of information (e.g., a user interface for operating peripherals and displaying content and information) via one or more output devices 531 (e.g., a display screen, speakers, etc.) associated with the user interface 530;

an input processing module 554 to detect one or more user inputs or interactions from one of the one or more input devices 532 and to translate the detected inputs or interactions.

In some embodiments of the present application, the full-data migration apparatus provided in the embodiments of the present application may be implemented by a combination of software and hardware, and for example, the full-data migration apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the full-data migration method provided in the embodiments of the present application.

In some embodiments of the present application, the full-volume data migration apparatus provided in this embodiment of the present application may be implemented in software, and fig. 2 illustrates a full-volume data migration apparatus 555 stored in a memory 550, which may be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 5551 and a migration module 5552, which are logical and thus may be arbitrarily combined or further split depending on the functionality implemented.

The functions of the respective modules will be explained below.

In other embodiments, the apparatus provided in the embodiments of the present Application may be implemented in hardware, and for example, the apparatus provided in the embodiments of the present Application may be a processor in the form of a hardware decoding processor, which is programmed to execute the full-scale data migration method provided in the embodiments of the present Application, for example, the processor in the form of the hardware decoding processor may be implemented by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), or other electronic components.

In this embodiment of the present application, a full data migration method provided in this embodiment of the present application will be described with a server as an execution subject.

Referring to fig. 3, fig. 3 is an alternative flowchart of a full data migration method provided in an embodiment of the present application, and will be described with reference to the steps shown in fig. 3.

In step 301, acquiring full data in a source database; the full amount of data includes at least one migration object.

In some embodiments of the present application, after receiving a migration request sent by an administrator, a server may determine, according to the migration request, a source database and a target database that need to be migrated. The migration request is used for requesting to migrate the full amount of data in the source database to the target database. Wherein, the full data is all data in the source database.

In some embodiments of the present application, step 301 may be implemented by: the server sends a database information acquisition request to the source database, wherein the database information acquisition request is used for acquiring full data information of the source database, and the full data information comprises storage information of at least one migration object in the full data. The server can acquire the migration object from the source database according to the storage information of each migration object and take out the migration object. The migration object may be a data table in a database.

In some embodiments of the present application, a server is communicatively connected to at least one terminal, and after receiving a database query request sent by any one terminal, the server may query database information of at least one database connected to the server, send the database information of the at least one database to the terminal, and display the at least one database and the database information of each database through a human interface on a terminal side. The terminal can also receive a migration request of an administrator through the human interactive interface, and determine a source database and a target database which need to be migrated according to the migration request.

In step 302, the migration object is sequentially extracted from the at least one migration object of the full amount of data as the object to be migrated, and the object to be migrated is migrated from the source database to the target database through an idle migration process in the at least one migration process.

In some embodiments of the present application, at least one migration process is included in the server, and the at least one migration process may be executed in parallel. In some embodiments, a migration process may simultaneously perform a migration operation of at least one object to be migrated. The migration process can provide a data migration service (DTS), and support a plurality of services for synchronizing and transmitting data between isomorphic or heterogeneous databases.

In some embodiments of the present application, a migration process may simultaneously perform a migration operation on a plurality of objects to be migrated, and a server may obtain a process load quantization value of each migration process, and determine that a process state of the migration process is a high load state when the process load quantization value of the migration process is higher than or equal to a preset load threshold. Under the condition that the process load quantitative value of the migration process is lower than the preset load threshold value, the migration process can also execute the migration operation of the object to be migrated, and therefore the process state of the migration process is determined to be a low load state; and determining the migration process in the low load state as an idle migration process.

The load quantization value may be the number of objects to be migrated at the same time, or may be a resource utilization rate when the migration process processes each object to be migrated.

For example, there are 3 migration processes, which are a first process, a second process, and a third process, and it is necessary to migrate the total amount of data including the first migration object to the fifth migration object, and in the process of starting migration, the first process, the second process, and the third process are all in a low load state, and therefore all are idle migration processes.

Under the condition that the load quantitative value is the number of objects to be migrated at the same time and the preset load threshold of each migration process is 1, extracting the first to third migration objects to obtain first to third objects to be migrated, executing the migration process of the first object to be migrated through a first (idle) process, executing the migration process of the second object to be migrated through a second (idle) process, and executing the migration process of the third object to be migrated through a third (idle) process; at this time, each migration process is in a high load state.

Under the condition that the load quantitative value is the resource utilization rate and the preset load threshold values of all the migration processes are all 80%, extracting a first migration object to a third migration object to obtain a first object to be migrated to a third object to be migrated, executing the migration process of the first object to be migrated through a first (idle) process, executing the migration process of the second object to be migrated through a second (idle) process, and executing the migration process of the third object to be migrated through the third (idle) process; at this time, the resource utilization rate of each migration process is obtained, if the resource utilization rates of the first process to the third process are respectively 40%, 50% and 90%, the first process and the second process are in a low-load state, and are still idle migration processes, and the third process is in a high-load state; and continuously extracting the fourth migration object and the fifth migration object as a fourth object to be migrated and a fifth object to be migrated, executing the migration process of the fourth object to be migrated through the first process, and executing the migration process of the fifth object to be migrated through the second process.

In some embodiments of the present application, if each of the at least one migration process is in a working state, the extraction of the migration object from the at least one migration object is stopped. Until there is at least one idle migration process.

In some embodiments of the present application, the server may, according to the number of processes of the idle migration process in the at least one migration process, obtain, from the at least one migration object in the source database, the migration object having the same number as the number of processes, use the migration object as the object to be migrated, and migrate, from the source database to the target database, the object to be migrated having the same number as the number of processes through the at least one idle migration process. Wherein, each idle migration process executes the migration operation of an object to be migrated.

In some embodiments of the present application, sequentially extracting a migration object from at least one migration object of the full amount of data as an object to be migrated may be implemented by: (1) copying a migration object in the at least one migration object, taking the copied migration object as the object to be migrated, and adding a migrated label to the copied migration object; in the subsequent migration process, acquiring the migration objects from the migration objects without the added migration labels for migration until all the migration objects in the full amount of data are added with the added migration labels; (2) directly taking out a migration object from at least one migration object in the full data as an object to be migrated; in the subsequent migration process, the migration objects are obtained from the rest migration objects in the full amount of data for migration until any migration object does not exist in the full amount of data.

Compared with the scheme that the whole data is firstly imported into the read-write storage pool and then exported to the target database after the whole import process is completed in the traditional technology, the method and the system for migrating the objects to be migrated achieve the whole export and import of the whole data simultaneously in the migration process of the objects to be migrated.

In step 303, in the case that at least one migration object is migrated to the target database, the full data migration process is completed.

In some embodiments of the present application, when the migration process of the full amount of data is replication migration, and a migration label is added to each migration object in at least one migration object in the full amount of data in the source database, the migration process of the right data of the full amount of data in the source database to the target database is completed. And under the condition that the full-amount data migration process is mobile migration, when any migration object does not exist in the full-amount data in the source database, completing the process of migrating the full-amount data in the source database to the right data in the target database.

As can be seen from the above exemplary implementation of fig. 3 in the embodiment of the present application, compared with a scheme that the full amount of data is firstly read to the read-write storage pool and the full amount of data is written to the target database after the reading process is completed in the conventional technology, in the embodiment of the present application, the full amount of data is thinned to the migration objects, and thus, each migration object can be directly migrated to the target database, and the storage pool is not read or written, which not only improves the migration speed, but also reduces the storage overhead and reduces the operation cost. The component dependence is reduced, and the overall reliability of the system is also improved. In addition, as the migration objects are migrated in parallel through the plurality of migration processes, the technical effect of simultaneously carrying out full export and full import on the full data is realized on the whole, and the overall migration efficiency of the full data is improved.

Referring to fig. 4, fig. 4 is an optional flowchart of a full data migration method provided in this embodiment of the present application, based on fig. 3, step 302 in fig. 3 may be updated to step 401 to step 403, which will be described with reference to the steps shown in fig. 4.

In step 401, in the case that there is an idle migration process in at least one migration process, a migration object is extracted from at least one migration object as an object to be migrated.

In some embodiments of the present application, a server may obtain a process state of each migration process, and determine that the process state of the migration process is a working state when the migration process is executing a migration operation of an object to be migrated; determining the process state of the migration process as an idle state under the condition that the migration process does not start or the migration operation of the object to be migrated is completed; and determining the migration process in the idle state as an idle migration process.

In step 402, an object to be migrated is allocated to an idle migration process.

In some embodiments of the present application, the number of extracted migration objects (objects to be migrated) may be determined according to the number of idle migration processes. For an idle migration process, a migration object can be extracted and allocated to the idle migration process as an object to be migrated, so that the idle migration process completes the migration process of the object to be migrated.

For example, there are 3 migration processes, which are a first process, a second process, and a third process, respectively, and it is necessary to migrate the entire data including the first to fifth migration objects, and in the process of starting migration, the first process, the second process, and the third process are all in an idle state, and therefore all are idle migration processes, at this time, the first to third migration objects may be simultaneously extracted to obtain the first to third objects to be migrated, and the migration process of the first object to be migrated is executed through the first (idle) process, the migration process of the second object to be migrated is executed through the second (idle) process, and the migration process of the third object to be migrated is executed through the third (idle) process; at this time, each migration process is in a working state, so that the extraction of the migration object from the at least one migration object is temporarily stopped; if the second process has completed the migration process of the second object to be migrated, at this time, the second process is in an idle state again, at this time, the fourth object to be migrated may be continuously extracted from the full amount of data in the source data as the fourth object to be migrated, and the migration process of the fourth object to be migrated is executed through the second (idle) process, and so on until the migration processes of the first object to the fifth object to be migrated are completed.

In step 403, the object to be migrated is migrated from the source database to the target database through the idle migration process.

As can be seen from the above exemplary implementation of fig. 4 in the embodiment of the present application, the migration process of the object to be migrated is completed through the idle migration process in the at least one migration process, so that the migration efficiency of the object to be migrated can be ensured, the problem that the overall data migration efficiency of the full amount of data is low due to uneven distribution of multiple migration objects is avoided, and the overall migration efficiency of the full amount of data is improved.

In some embodiments of the present application, referring to fig. 5, fig. 5 is an optional flowchart of a full volume data migration method provided in the embodiments of the present application, and based on fig. 4, step 403 shown in fig. 4 may be updated to be step 501 to step 502.

In step 501, the object to be migrated is fetched from the source database by the read thread in the idle migration process, and the object to be migrated is written into the data queue corresponding to the idle migration process in the memory.

In some embodiments of the present application, each migration process (idle migration process) corresponds to one data queue in a memory, and in a migration process of an object to be migrated by using a migration process (idle migration process), the object to be migrated needs to be taken out from a source database and written into a data queue corresponding to the migration process (idle migration process) in the memory in an asynchronous manner, and multiple objects to be migrated may be simultaneously stored in the data queue, and the object to be migrated is taken out from the data queue and written into the target database, so as to complete a migration process of data to be migrated from the source database to the target database.

In some embodiments of the present application, each migration process (idle migration process) includes a read thread, where the read thread is configured to take out an object to be migrated from a source database, and write the object to be migrated into a data queue corresponding to the idle migration process in a memory.

In some embodiments of the present application, the above step 501 may be implemented by: in step 5011, reading the structure data of the object to be migrated from the source database to the data queue through the read thread; in step 5012, the source database data of the object to be migrated is read from the source database to the data queue through the read process.

Each migration object in the full amount of data comprises source database data and structural data, wherein the source database data are specific data in the full amount of data which are required to be migrated to the target database in the source database; the structure data is a schema (schema) of the source database, and is a general name of a schema object, where the schema object includes at least one of objects such as tables (tables), columns (columns), data types (data types), views (views), stored procedures (stored procedures), relationships (relationships), primary keys (primary keys), and foreign keys (foreign keys).

In step 501, when the data queue is a first-in first-out data queue, it is necessary to first obtain the structural data of the object to be migrated and write the structural data into the data queue, so as to ensure that, in the process of writing the structural data into the target database, the target database is configured by using the structural data, and then the source database data of the object to be migrated is written into the data queue, so that when the source database data is written into the target database, the target database completes the configuration process according to the structural data of the object to be migrated. When the data queue is in a first-in-last-out state, the source library data of the object to be migrated may be obtained first, and written into the data queue, and the structure data may be obtained and written into the data queue. In step 502, the object to be migrated is fetched from the data queue by the write thread in the idle migration process, and the object to be migrated is written into the target database.

In some embodiments of the present application, each migration process (idle migration process) includes a write thread, where the write thread is configured to fetch an object to be migrated from a data queue and write the object to be migrated to a target database.

In some embodiments of the present application, in a process of taking out and writing an object to be migrated from a data queue into a target database, in order to improve writing efficiency and writing accuracy, structural data of the object to be migrated may be taken out first, and the structural data is written into the target database, so that the target database completes its own configuration process according to the structural data, after configuration, suction efficiency and writing accuracy of source database data in the object to be migrated may be improved, and then, source database data of the object to be migrated is taken out, and the source database data is written into the configured target database.

In some embodiments of the present application, step 502 described above may be implemented by: in step 5021, the structural data of the object to be migrated is taken out of the data queue through the write-in thread, and the structural data is written into the target database, so that the configuration process of the target database according to the structural data is completed; in step 5022, the source database data of the object to be migrated is taken out of the data queue through the write thread, and the source database data is written into the target database configured by the structural data.

In some embodiments of the present application, the read thread and the write thread in the migration process are executed in parallel, and when the migration process does not execute other migration tasks and only migrates one object to be migrated, the migration process may be divided into the following stages: the first stage, a read thread acquires the structural data corresponding to the process to be migrated from a source database and writes the structural data into a data queue, and at the moment, the write thread is in an idle state; and in the second stage, the reading thread finishes the process of writing the structural data into the data queue, at the moment, the reading thread starts to acquire the source database data of the process to be migrated from the source database and writes the source database data into the data queue, and meanwhile, the writing thread writes the structural data in the data queue into the target database so as to configure the target database. It can be seen that, at this stage, the processes of reading and writing can be executed in parallel for the object to be migrated; and in the third stage, the write thread finishes the configuration process of the target database, and starts to write the source database into the configured target database under the condition that the read thread writes the source database into the data queue. Meanwhile, if the migration object which is not migrated still exists in the source database, the structure data of the next object to be migrated is continuously read through the reading thread, and at this time, the writing process of the previous object to be migrated and the reading process of the next object to be migrated are executed in parallel.

In some embodiments of the present application, the migration objects in the full amount of data may include both source library data and structure data, may include only source library data, and may include only structure data. When the migration object includes the source library data and the structure data, the read-write method in the above embodiment needs to be adopted, the structure data of the migration object is migrated to the target database, and then the source library data of the migration object is migrated to the target database. In the case where the migration object includes only structural data, the source database data of the migration object need not be migrated after the structural data of the migration object is migrated into the target database. And in the case that the migration object only comprises the data of the source database, the structure data of the migration object is directly migrated to the target database without migrating the structure data of the migration object. On the basis of the implementation, the full-volume data migration method provided by the application may further include: obtaining a migration instruction of each migration object in the at least one migration object, where the migration instruction may be one of: a first migration instruction for simultaneously migrating the source library data and the structure data of the migration object; second migration instructions to migrate the source library data of the migrated object without migrating the structure data of the migrated object; and a third migration instruction for migrating the structure data of the migration object and not migrating the source library data of the migration object.

As can be seen from the foregoing exemplary implementation of fig. 5 in the embodiment of the present application, in the migration process of an object to be migrated from a source database to a target database, since a data queue is used to cache the object to be migrated, an asynchronous migration process of each object to be migrated is implemented, and data migration efficiency is improved; in addition, in the migration process of each object to be migrated, the method of migrating the structural data of the object to be migrated first, configuring the target database, and then migrating the source database data of the object to be migrated to the configured target database is adopted, so that the data migration efficiency and the migration accuracy can be further improved.

In some embodiments of the present application, referring to fig. 6, fig. 6 is an optional flowchart of a full-volume data migration method provided in the embodiments of the present application, and based on fig. 4, the method may further include steps 601 to 602.

In step 601, monitoring a process state of each migration process in at least one migration process; the process state includes a read thread state and a write thread state.

In some embodiments of the present application, the presence of an idle migration process may be determined by detecting the process state of each migration process. According to the above embodiment, each migration process includes a read thread and a write thread, and the read thread and the write thread are executed in parallel, so the process status of each migration process can be determined by monitoring the read thread status of the read thread and the write thread status of the write thread in each migration process.

Wherein, the process state of each migration process may include: the read thread state and the write thread state are both in working states; the read thread state is in a working state and the write thread state is in an idle state; the write thread state is in a working state and the read thread state is in an idle state; the read thread state and the write thread state are both in an idle state.

In step 602, an idle migration process is determined in at least one migration process according to the read thread state and the write thread state corresponding to each migration process.

In some embodiments of the present application, the step 602 may include: step 6021, determining the migration process as an idle migration process when the read thread state and the write thread state corresponding to the migration process are both idle states. By the embodiment, the migration process can be ensured to complete the migration task of the next migration object after completing the migration task of one migration object, the risk of errors during data migration can be reduced, and the accuracy of data migration is ensured.

In some embodiments of the present application, the step 602 may further include: step 6022, determining the migration process as an idle migration process when the read thread state corresponding to the migration process is an idle state and the write thread states are all working states. By the embodiment, under the condition that the reading rate of the source database and the writing rate of the target database are uncontrollable, the risk of low data migration efficiency caused by fluctuation of the reading rate and the writing rate is reduced, and the overall data migration efficiency of the whole data is improved.

As can be seen from the above exemplary implementation of fig. 6, according to the present application, by detecting the thread states of the read thread and the write thread in each migration process, and then determining the idle migration thread according to the read thread state and the write thread state, it can be ensured that after a migration task of one migration object is completed by a migration process, a migration task of a next migration object is completed by the migration process, so that a risk of an error occurring during data migration can be reduced, and a correct rate of data migration can be ensured; and under the condition that the reading rate of the source database and the writing rate of the target database are uncontrollable, the risk of low data migration efficiency caused by fluctuation of the reading rate and the writing rate is reduced, and the overall data migration efficiency of the full amount of data is improved.

In some embodiments of the present application, referring to fig. 7, fig. 7 is an optional flowchart of a full-volume data migration method provided in the embodiments of the present application, and based on fig. 2, the method may further include steps 701 to 703.

In step 701, taking out the object to be migrated from the source database through a read thread in the idle migration process, and writing the object to be migrated into a data queue corresponding to the idle migration process in the memory; and taking out the object to be migrated from the data queue through a write-in thread in the idle migration process, and writing the object to be migrated into the target database.

In some embodiments of the present application, the above detailed description of step 701 may refer to the description in fig. 5.

In step 702, in the process of fetching the object to be migrated from the source database by the read thread in the idle migration process and writing the object to be migrated into the data queue corresponding to the idle migration process in the memory, if the data queue is full, the read thread is suspended.

In some embodiments of the present application, for a migration process, in a case that a read thread in the migration process is in an idle state, the migration process may be determined as an idle migration process, and the structure data and the source library data of the object to be migrated are continuously read from the source database and written into a data queue corresponding to the migration process. Because the data queue is located in the memory and the capacity size of the data queue is limited, when the write-in rate of the write-in thread is low, the fixed capacity of the data queue may be exceeded in the process that the read thread reads the structure data and the source library data of the object to be migrated and writes the data into the data queue, and therefore, when the data queue is full, the read thread is suspended and only the write-in thread is operated.

In one embodiment of the present application, whether the data queue is full may be determined by: acquiring the number of objects to be migrated temporarily stored in the data queue, and determining that the data queue is full under the condition that the number of the objects to be migrated reaches a preset number threshold; and acquiring the data volume of the objects to be migrated temporarily stored in the data queue, and determining that the data queue is full when the ratio of the data volume to the capacity of the data queue exceeds a first ratio.

In step 703, in the process of taking out the object to be migrated from the data queue by the write-in thread in the idle migration process and writing the object to be migrated into the target database, the load quantization value in the data queue is obtained, and the read thread is resumed when the load quantization value is lower than the preset threshold.

In some embodiments of the present application, since the data queue is full, the read thread is temporarily suspended, the object to be migrated is taken out from the data queue through the write thread, and the object to be migrated is written into the target database, and it is possible to determine whether to resume the read thread by obtaining a load quantization value in the data queue. The load quantization value may be the number of the objects to be migrated, the preset threshold is the preset number threshold, and the read thread is resumed on a day when the number of the objects to be migrated is less than the preset number threshold; the load quantization value may also be a ratio of the data amount to the capacity of the data queue, the preset threshold is a second ratio, and the read thread is resumed when the ratio is lower than the second ratio, which is less than or equal to the first ratio.

As can be seen from the above exemplary implementation of fig. 7 in the embodiment of the present application, when the data queue corresponding to the migration process is full, the read thread of the migration process is suspended, so that the stability of the data queue in the memory can be ensured; meanwhile, under the condition that the load quantization value of the data queue is judged to be lower than the preset threshold value, the reading thread is recovered in time, and the overall transmission efficiency of the full data can be effectively improved while the stability is ensured.

Next, an exemplary application of the embodiment of the present application in a practical application scenario will be described.

The embodiment of the application provides a new migration flow and a new migration framework aiming at the full export and import stages of the database data migration, improves the migration efficiency, saves the migration time, improves the stability and reduces the cost.

The database data migration full process mainly comprises a full stage and an increment stage, wherein the full stage comprises the following steps: a full export phase and a full import phase; the increment stage is divided into: an increment export and an increment import phase. After research, the inventors found that the structural data (schema) defines the organization form of the data (data), and therefore the structural data must be imported before the data is imported into the target database. The related scheme leads the database data out of the storage pool, and leads the database data into the target end from the storage pool after the leading-out is finished.

The embodiment of the application optimizes the process as follows:

(1) the full import does not need to wait for all the migrated objects to start after the full export is finished. According to the scheme, the full-scale stage scheduling granularity of the migration task is refined to a specific migration object. When the migration object imports the structure data from the source end to the target end, the data can be read from the source end and the read data is imported to the target end. Therefore, macroscopically, the full-scale export and the full-scale import are executed in parallel, and the overall migration efficiency is improved.

(2) In the full export phase, the structure data and the data do not need to enter the storage pool, but the structure data and the data are directly imported into the target database. The read-write storage pool is removed, so that the migration speed is improved, the storage overhead is reduced, and the operation cost is reduced. The component dependence is reduced, and the overall reliability of the system is also improved. The storage pool in the related art refers to a disk, or a disk array, or an object storage system, or a message queue.

In the related art, for data migration of a database, for example, logical export based on mysqldump, mydumper and the like or physical export based on xtrabeckup and the like all split the full-scale export and full-scale import phases. That is, the full export of all objects to be migrated must be completed before the full import can be started, and a global synchronization barrier exists. This process is illustrated in fig. 8.

Referring to the migration process diagram in fig. 8, the amount of data derived from the whole volume is huge, and due to the existence of the synchronization barrier, the data cannot be cached in the memory. Therefore, the full export phase must put the exported information in external storage for the full import phase to read. The exported data may be stored in a local disk, an object store, a cloud disk, etc., which are abstracted into a storage pool.

The applicant has found that the following disadvantages exist in the above-mentioned related art:

(1) the total amount is not simultaneously introduced and is not discharged. When the source database is exported, the target database is completely idle and waits for the export to be completed. For example, there is a database with a capacity of 6T, and in an absolutely ideal case (no user load, all resources are in full operation for data migration), the export of 6T data takes about 18 hours. Full load import is the process of executing and writing, and can be much longer than 18 hours. Therefore, the process has great optimization value.

(2) An export needs to be written to a storage pool from which the export will be read. The capacity of the storage pool limits the capacity of the full import and export. The space of the storage pool must be larger than the source database instance, increasing the economic cost.

(3) The time consumed for exporting and importing the whole data is too long, and the time and the cost of the whole data migration are increased. Since incremental data export is initiated from the migration initiation, the incremental data also needs to be stored in the storage pool. Because the remote database is readable and writable during the data migration process, the source database is frequently changed, which is a very common scenario, and the incremental data generated between the full export and the full import is much. The incremental data occupies the storage resource for a long time, and the workload and the time of the incremental import stage are increased.

In response to this deficiency, the applicant removed this synchronization barrier: the total export and import are fused, and the overall time is reduced in the process. With the synchronization barrier removed, the storage pool is no longer a necessity, and data is "streamed" directly from the source to the target database. The performance is improved, the economic cost is reduced, and the system reliability is improved.

Referring to FIG. 9, a schematic diagram of an interactive interface of an alternative database migration service is shown. In the interactive interface 90, a plurality of migration tasks are displayed, and an administrator can perform operations such as viewing, modifying, verifying, starting, deleting, canceling, and data comparison on each migration task, for example, after performing a selection operation on one of the migration tasks 91, a secondary menu diagram as shown in fig. 10 can be obtained, in a partial interface 911, a secondary menu 912 corresponding to the migration task 91 can be obtained, and a migration request for the full amount of data by the administrator can be obtained by detecting the selection operation of the administrator on a "full amount migration" option of the secondary menu 912.

In some embodiments of the present application, referring to fig. 11, fig. 11 is a schematic diagram of an optional migration process of the full-volume data migration method provided in the embodiments of the present application.

This scheme exports the total and the leading-in integration back of total, divide into 2 modules: a management module 111 and an execution module 112. Wherein:

the management module 111: the object to be migrated is known, e.g., which specific library table is. The objects to be migrated are packaged into jobs to form job queues, and asynchronously dispatched to the execution module 112 through the job queues. Wherein, in the case of an idle execution module, a job is distributed to the execution module; when the job queue is empty, i.e., the job is completed, then the full export and import is completed. Each table is an object to be migrated. An object to be migrated is divided into a "structure data part" (also called schema, i.e. structure data) and a "data part" (also called data, which is specific data in a table, i.e. source library data). The migration instruction of the administrator for each object to be migrated may be received, where the migration instruction may include a first migration instruction for migrating the "structure data portion" and the "data portion" simultaneously, may also include a second migration instruction for migrating only the "data portion", and may also include a third migration instruction for migrating only the "structure data portion". If the migration instruction of the administrator for each object to be migrated is not received, the structure data part and the data part are migrated at the same time by default. If the administrator wants to migrate the structural data part of the object to be migrated to the target database only, no specific data is needed, and a third migration instruction can be selected; if the target database has already established the data structure, the administrator wishes to migrate only the "data portion" of the object to be migrated to the target database, and may select the second migration instruction.

The execution module 112: the execution module has a plurality of entities (programming term instance may be understood as a process, or a service, which may have any number of execution modules, the more the migration is better, but the more the cost is higher), and after initialization, the connection work of the source DB and the target DB is completed, and then the job dispatch of the management module 111 is waited. And after receiving the operation, reading the structural data of the object to be migrated in the operation from the source DB, sending the structural data to the target DB for execution, and constructing the structural data in the target DB. And then, reading the source database data of the object to be migrated from the source DB, sending the source database data to the target DB for execution, and constructing the source database data in the target DB. After the source library data is constructed, the execution module completes the full export and the full import of the object to be migrated, and can execute new operation to perform the full export and import on other migrated objects.

The above steps are that in the conventional scheme, a global synchronization barrier object: all objects to be migrated are scattered into each job and executed in parallel by a plurality of execution modules without waiting for all objects to complete a certain stage. This is a parallelization transformation from the global task allocation stage, improving performance.

In some embodiments of the present application, referring to fig. 12, fig. 12 is a schematic diagram of an optional migration process of the full-volume data migration method provided in the embodiments of the present application.

The read (dump) thread 121 of the execution module continuously reads the structure data or data from the source DB and asynchronously writes to the high speed memory data queue, which replaces the memory pool of the conventional scheme. And the asynchronous mode ensures that the structure data or the data are continuously acquired from the source end when the queue is not full. When the queue is full, the fetch action is suspended until the queue has space to allow writes. dump threads and jobs are in a one-to-one correspondence. One thread processes one job and the best performance can be achieved.

At the same time, the execution module writes (apply) to thread 122, reads the structure data or data from the queue, writes to the target DB until the dump thread completes the work and the queue is empty.

The source DB has user use load, dump speed may be different, and the high-speed memory queue exists to smooth the speed change. Is beneficial to improving the overall efficiency. The implementation mode of the execution module removes a synchronization barrier that the whole amount is exported first and then imported from the data migration flow of each migration object, and removes the dependence on the storage pool. The use of a storage pool is eliminated, performance is improved, cost is reduced, component dependence is reduced, and system reliability is improved.

By the full data migration method provided by the embodiment, the following technical effects can be achieved: (1) the migration speed is improved; after the synchronous barriers of full export and full import are removed, each migration object can be independent without waiting, and full migration is completed. The storage pool is removed, and the negative influence of the storage pool performance on the migration speed is reduced. The migration of the full-scale stage can fully utilize the network speed and the DB performance, and the restriction of unnecessary components on the speed is reduced. (2) Reliability is improved, and component dependence is reduced; the elimination of the storage pool reduces component complexity and improves reliability. (3) The operation cost is reduced; the storage pool is removed without the need to allocate a full amount of data to be migrated for each migration task. And after the migration speed of the full-scale stage is improved, the storage space required by incremental data is also reduced. The factors are superposed, so that the operation cost can be greatly reduced.

Continuing with the exemplary structure of the full data migration apparatus 555 implemented as software modules provided by the embodiments of the present application, in some embodiments of the present application, as shown in fig. 2, the software modules stored in the full data migration apparatus 555 in the memory 550 may include:

an obtaining module 5551, configured to obtain full data in a source database; the full amount of data includes at least one migration object;

the migration module 5552 is configured to sequentially extract a migration object from at least one migration object of the full amount of data as an object to be migrated, and migrate the object to be migrated from the source database to the target database through an idle migration process in the at least one migration process;

the migration module 5552 is further configured to complete the full data migration process if all of the at least one migration object is migrated to the target database.

In some embodiments of the present application, the migration module 5552 is further configured to, in a case that there is an idle migration process in the at least one migration process, extract a migration object from the at least one migration object as an object to be migrated; allocating an object to be migrated to an idle migration process; and migrating the object to be migrated from the source database to the target database through the idle migration process.

In some embodiments of the present application, the migration module 5552 is further configured to take out an object to be migrated from the source database through a read thread in the idle migration process, and write the object to be migrated into a data queue corresponding to the idle migration process in the memory; and taking out the object to be migrated from the data queue through a write-in thread in the idle migration process, and writing the object to be migrated into the target database.

In some embodiments of the present application, the migration module 5552 is further configured to read, by a read thread, the structure data of the object to be migrated from the source database to the data queue; and reading the source database data of the object to be migrated from the source database to the data queue through the reading process.

In some embodiments of the present application, the migration module 5552 is further configured to fetch the structural data of the object to be migrated from the data queue through a write thread, and write the structural data into the target database, so as to complete a configuration process of the target database according to the structural data; and taking out the source database data of the object to be migrated from the data queue through the write-in thread, and writing the source database data into the target database configured by the structural data.

In some embodiments of the present application, the migration module 5552 is further configured to monitor a process status of each of the at least one migration process; the process state comprises a read thread state and a write thread state; and determining an idle migration process in at least one migration process according to the read thread state and the write thread state corresponding to each migration process.

In some embodiments of the present application, the migration module 5552 is further configured to suspend the read thread if the data queue is full in a process of taking out the object to be migrated from the source database through the read thread in the idle migration process and writing the object to be migrated into the data queue corresponding to the idle migration process in the memory; and in the process of taking out the object to be migrated from the data queue through a write-in thread in the idle migration process and writing the object to be migrated into the target database, acquiring a load quantization value in the data queue, and recovering the read thread under the condition that the load quantization value is lower than a preset threshold value.

Embodiments of the present disclosure provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the full-data migration method described in the embodiment of the present application.

Embodiments of the present disclosure provide a computer-readable storage medium having stored therein executable instructions that, when executed by a processor, will cause the processor to perform the full data migration method provided by embodiments of the present disclosure, for example, the method as illustrated in fig. 3 to 7.

In some embodiments of the present application, the computer readable storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

In some embodiments of the application, the executable instructions may be in the form of a program, software module, script, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.

In summary, the following technical effects can be achieved through the embodiments of the present application:

(1) compared with the scheme that the full data are read to the read-write storage pool and written into the target database after the reading process is completed in the traditional technology, the full data are thinned to the migration objects, and the migration objects can be directly migrated to the target database respectively, so that the storage pool is not required to be read and written, the migration speed is increased, the storage expense is reduced, and the operation cost is reduced. The component dependence is reduced, and the overall reliability of the system is also improved. In addition, as the migration objects are migrated in parallel through the plurality of migration processes, the technical effect of simultaneously carrying out full export and full import on the full data is realized on the whole, and the overall migration efficiency of the full data is improved.

(2) According to the embodiment of the application, the migration process of the object to be migrated is completed through the idle migration process in the at least one migration process, so that the migration efficiency of the object to be migrated can be guaranteed, the problem that the overall data migration efficiency of the full amount of data is low due to uneven distribution of a plurality of migrated objects is avoided, and the overall data migration efficiency is improved.

(3) In the migration process of the object to be migrated from the source database to the target database, the data queue is adopted to cache the object to be migrated, so that the asynchronous migration process of each object to be migrated is realized, and the data migration efficiency is improved; in addition, in the migration process of each object to be migrated, the method of migrating the structural data of the object to be migrated first, configuring the target database, and then migrating the source database data of the object to be migrated to the configured target database is adopted, so that the data migration efficiency and the migration accuracy can be further improved.

(4) By detecting the thread states of the read thread and the write thread in each migration process and determining the idle migration thread according to the read thread state and the write thread state, the migration process can be ensured to complete the migration task of one migration object and then complete the migration task of the next migration object through the migration process, the risk of errors during data migration can be reduced, and the accuracy of data migration is ensured; and under the condition that the reading rate of the source database and the writing rate of the target database are uncontrollable, the risk of low data migration efficiency caused by fluctuation of the reading rate and the writing rate is reduced, and the overall data migration efficiency of the full amount of data is improved.

(5) The method and the device have the advantages that under the condition that the data queue corresponding to the migration process is full, the reading thread of the migration process is suspended, and the stability of the data queue in the memory can be guaranteed; meanwhile, under the condition that the load quantization value of the data queue is judged to be lower than the preset threshold value, the reading thread is recovered in time, and the overall transmission efficiency of the full data can be effectively improved while the stability is ensured.

The above description is only an example of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A full-scale data migration method, comprising:

acquiring full data in a source database; the full amount of data includes at least one migration object;

sequentially extracting a migration object from at least one migration object of the full amount of data as an object to be migrated, and migrating the object to be migrated from the source database to a target database through an idle migration process in at least one migration process;

and under the condition that the at least one migration object is migrated to the target database, completing the full data migration process.

2. The method according to claim 1, wherein the sequentially extracting the migration object from the at least one migration object of the full amount of data as the object to be migrated, and migrating the object to be migrated from the source database to the target database through an idle migration process of the at least one migration process, includes:

under the condition that the idle migration process exists in the at least one migration process, extracting a migration object from the at least one migration object as an object to be migrated;

allocating the object to be migrated to the idle migration process;

and migrating the object to be migrated from the source database to a target database through the idle migration process.

3. The method according to claim 2, wherein the migrating the object to be migrated from the source database to the target database through the idle migration process comprises:

taking out the object to be migrated from the source database through a reading thread in the idle migration process, and writing the object to be migrated into a data queue corresponding to the idle migration process in a memory;

and taking out the object to be migrated from the data queue through a write-in thread in the idle migration process, and writing the object to be migrated into the target database.

4. The method according to claim 3, wherein the fetching the object to be migrated from the source database by a read thread in the idle migration process and writing the object to be migrated into a data queue corresponding to the idle migration process in a memory comprises:

reading the structural data of the object to be migrated from the source database to the data queue through the reading thread;

and reading the source database data of the object to be migrated from the source database to the data queue through the reading process.

5. The method according to claim 3 or 4, wherein the fetching the object to be migrated from the data queue and writing the object to be migrated into the target database by a write thread in the idle migration process comprises:

taking out the structural data of the object to be migrated from the data queue through the write-in thread, and writing the structural data into the target database to complete the configuration process of the target database according to the structural data;

and taking out the source database data of the object to be migrated from the data queue through the write-in thread, and writing the source database data into a target database configured by the structural data.

6. The method of claim 2, further comprising:

monitoring the process state of each migration process in the at least one migration process; the process state comprises a read thread state and a write thread state;

and determining the idle migration process in the at least one migration process according to the read thread state and the write thread state corresponding to each migration process.

7. The method of claim 6, wherein determining the idle migration process among the at least one migration process according to the read thread state and the write thread state corresponding to each of the migration processes comprises at least one of:

determining the migration process as an idle migration process under the condition that the read thread state and the write thread state corresponding to the migration process are both idle states;

and determining the migration process as an idle migration process under the condition that the read thread state corresponding to the migration process is an idle state and the write thread states are working states.

8. A full-scale data migration apparatus, comprising:

the acquisition module is used for acquiring the full data in the source database; the full amount of data includes at least one migration object;

the migration module is used for sequentially extracting a migration object from at least one migration object of the full amount of data to serve as an object to be migrated, and migrating the object to be migrated from the source database to the target database through an idle migration process in at least one migration process;

the migration module is further configured to complete the full data migration process when all of the at least one migration object is migrated to the target database.

9. A full-volume data migration apparatus, comprising:

a memory for storing executable instructions;

a processor for implementing the method of any one of claims 1 to 7 when executing executable instructions stored in the memory.

10. A computer-readable storage medium having stored thereon executable instructions for, when executed by a processor, implementing the method of any one of claims 1 to 7.