CN113051245A

CN113051245A - Method, device and system for migrating data

Info

Publication number: CN113051245A
Application number: CN201911366408.XA
Authority: CN
Inventors: 李欣; 高灿锦
Original assignee: Yunding Network Technology Beijing Co Ltd
Current assignee: Yunding Network Technology Beijing Co Ltd
Priority date: 2019-12-26
Filing date: 2019-12-26
Publication date: 2021-06-29

Abstract

The embodiment of the application discloses a method, a device and a system for migrating data. One embodiment of the method comprises: acquiring data to be migrated and data migration configuration information, wherein the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure; creating at least one data migration task based on the data migration configuration information and the data to be migrated; and executing at least one data migration task to store the data respectively corresponding to the at least one data migration task in the corresponding target database in the corresponding target data structure. The implementation mode realizes data migration among different databases and different data structures, and meets the data migration requirement of complex data structures.

Description

Method, device and system for migrating data

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a method, a device and a system for migrating data.

Background

Data migration is a common technology for system upgrade, and mainly migrates data in a database.

With the development of database technology, the types of databases are more and more diverse, the data structures are more and more complex, and the data migration requirements of complex data structures cannot be met by the current data migration work.

Disclosure of Invention

The embodiment of the application provides a method, a device and a system for migrating data.

In a first aspect, an embodiment of the present application provides a method for migrating data, where the method includes:

acquiring data to be migrated and data migration configuration information, wherein the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure; creating at least one data migration task based on the data migration configuration information and the data to be migrated; and executing at least one data migration task to store the data respectively corresponding to the at least one data migration task in the corresponding target database in the corresponding target data structure.

In some embodiments, the data migration configuration information includes a data category set, and each data category in the data category set corresponds to a migration priority. In some embodiments, the data categories in the data category set include user data, device data, a core log and a non-core log, and migration priorities of the user data, the device data, the core log and the non-core log are sequentially decreased.

In some embodiments, creating at least one data migration task based on the data migration configuration information and the data to be migrated includes: dividing the data to be migrated into a plurality of data groups based on the data types in the data type set, wherein the data types of different data groups are different; sorting the plurality of data groups according to the migration priority of the data category to obtain a data group sequence; and sequentially creating at least one data migration task for each data group from the data group with the highest migration priority in the data group sequence based on the data migration configuration information, wherein the migration priority of the data group is the same as the migration priority of the corresponding data category.

In some embodiments, executing the at least one data migration task to store data respectively corresponding to the at least one data migration task in a respective target database in a respective target data structure includes: and sequentially executing the data migration tasks created for each data group according to the migration priority of the data groups from high to low so as to sequentially store the data in the data groups with different migration priorities in the corresponding target database in the corresponding target data structure.

In some embodiments, before creating at least one data migration task based on the data migration configuration information and the data to be migrated, the method further comprises: under the condition of receiving a detection request which is sent by a user and carries a detection range of data to be migrated, creating a pre-migration detection task according to the detection range; splitting a pre-migration detection task into a plurality of detection tasks, wherein different detection tasks respectively correspond to different data groups in a detection range of data to be migrated; and executing a plurality of detection tasks corresponding to the pre-migration detection task to obtain an execution result of each detection task.

In some embodiments, executing a plurality of detection tasks corresponding to the premigration detection task to obtain an execution result of each detection task includes: for each detection task, the following detection operations are performed: performing aggregation calculation on the data group corresponding to the detection task in the corresponding source database to obtain a first aggregation result of the data group corresponding to the detection task in the corresponding source database; the simulation stores the data group corresponding to the detection task in a corresponding target database in a corresponding target data structure, and performs aggregation calculation on the data group stored due to the execution of the detection task in the corresponding target database to obtain a second aggregation result of the data group corresponding to the detection task in the corresponding target database; and determining whether the data group corresponding to the detection task has a migration problem or not by comparing the first aggregation result with the second aggregation result.

In some embodiments, the method for migrating data further comprises: acquiring a state detection task, wherein the state detection task comprises a task identifier; and inquiring the execution state of the task corresponding to the task identification.

In some embodiments, the method for migrating data further comprises: and under the condition that the data migration task corresponding to the user data is completely finished, if the fact that the user operates in the original system corresponding to the data to be migrated is detected, pushing prompt information for indicating to log in a new system corresponding to the data to be migrated to the user.

In a second aspect, an embodiment of the present application provides an apparatus for migrating data, where the apparatus includes: the data migration system comprises an acquisition unit, a migration unit and a migration unit, wherein the acquisition unit is used for acquiring data to be migrated and data migration configuration information, the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure; the creating unit is used for creating at least one data migration task based on the data migration configuration information and the data to be migrated; and the execution unit is used for executing the at least one data migration task so as to store the data respectively corresponding to the at least one data migration task in the corresponding target database in the corresponding target data structure.

In some embodiments, the data migration configuration information includes a set of data categories, each data category in the set of data categories respectively corresponding to a migration priority.

In some embodiments, the data categories in the data category set include user data, device data, a core log, and a non-core log, and migration priorities of the user data, the device data, the core log, and the non-core log are sequentially decreased.

In some embodiments, the creating unit is specifically configured to: dividing the data to be migrated into a plurality of data groups based on the data types in the data type set, wherein the data types of different data groups are different; sorting the plurality of data groups according to the migration priority of the data category to obtain a data group sequence; and sequentially creating at least one data migration task for each data group from the data group with the highest migration priority in the data group sequence based on the data migration configuration information, wherein the migration priority of the data group is the same as the migration priority of the corresponding data category.

In some embodiments, the execution unit is specifically configured to: and sequentially executing the data migration tasks created for each data group according to the migration priority of the data groups from high to low so as to sequentially store the data in the data groups with different migration priorities in the corresponding target database in the corresponding target data structure.

In some embodiments, the apparatus for migrating data further includes a premigration detection unit, where the premigration detection unit is specifically configured to: under the condition of receiving a detection request which is sent by a user and carries a detection range of data to be migrated, creating a pre-migration detection task according to the detection range; splitting a pre-migration detection task into a plurality of detection tasks, wherein different detection tasks respectively correspond to different data groups in a detection range of data to be migrated; and executing a plurality of detection tasks corresponding to the pre-migration detection task to obtain an execution result of each detection task.

In some embodiments, in the premigration detection unit, executing a plurality of detection tasks corresponding to the premigration detection task to obtain an execution result of each detection task, where the executing result includes: for each detection task, the following detection operations are performed: performing aggregation calculation on the data group corresponding to the detection task in the corresponding source database to obtain a first aggregation result of the data group corresponding to the detection task in the corresponding source database; the simulation stores the data group corresponding to the detection task in a corresponding target database in a corresponding target data structure, and performs aggregation calculation on the data group stored due to the execution of the detection task in the corresponding target database to obtain a second aggregation result of the data group corresponding to the detection task in the corresponding target database; and determining whether the data group corresponding to the detection task has a migration problem or not by comparing the first aggregation result with the second aggregation result.

In some embodiments, the apparatus for migrating data further includes: the state detection unit is used for acquiring a state detection task, and the state detection task comprises a task identifier; and inquiring the execution state of the task corresponding to the task identification.

In some embodiments, the apparatus for migrating data further includes: and the login prompting unit is used for pushing prompting information for indicating to login a new system corresponding to the data to be migrated to the user if the fact that the user operates in the original system corresponding to the data to be migrated is detected under the condition that the data migration task corresponding to the user data is completely finished is inquired.

In a third aspect, the present application provides a computer readable medium, on which a computer program is stored, where the program, when executed by a processor, implements the method as described in any implementation manner of the first aspect.

In a fourth aspect, the present application provides a processor, where the processor is configured to execute a program, where the program when executed implements the method described in any implementation manner of the first aspect.

In a fifth aspect, an embodiment of the present application provides an electronic device, including:

one or more processors;

a storage device having one or more programs stored thereon;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as described in any implementation of the first aspect.

In a sixth aspect, an embodiment of the present application provides a system for migrating data, where the system includes a scheduling device and at least one task execution device; the scheduling device is configured to acquire data to be migrated and data migration configuration information, the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure; creating at least one data migration task based on the data migration configuration information and the data to be migrated; assigning the at least one data migration task to a task execution device of the at least one task execution device; and the task execution device is configured to execute the received data migration task so as to store the data corresponding to the received data migration task in the corresponding target database in the corresponding target data structure.

According to the method, the device and the system for migrating data, provided by the embodiment of the application, the mapping relation between the source database and the target database and the mapping relation between the source data structure and the target data structure are defined by presetting the data migration configuration information, so that data migration between different databases and between different data structures is realized, and the data migration requirement of a complex data structure is met.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some examples or embodiments of the present application, and that for a person skilled in the art, other drawings can be obtained from the provided drawings without inventive effort, and that the present application can also be applied to other similar scenarios from the provided drawings. Unless otherwise apparent from the context, or otherwise indicated, like reference numbers in the figures refer to the same structure or operation.

FIG. 1 is an exemplary system architecture diagram to which some embodiments of the present application may be applied;

FIG. 2 is a schematic flow chart diagram illustrating one embodiment of a method for migrating data in accordance with the present application;

FIG. 3 is a schematic illustration of an application scenario of a method for migrating data according to the present application;

FIG. 4 is a schematic flow chart diagram illustrating one embodiment of a method for creating at least one data migration task based on data migration configuration information and data to be migrated in accordance with the present application;

FIG. 5 is a schematic flow chart diagram illustrating yet another embodiment of a method for migrating data in accordance with the present application;

FIG. 6 is a schematic block diagram illustrating one embodiment of an apparatus for migrating data according to the present application;

FIG. 7 is a timing diagram for one embodiment of a system for migrating data according to the present application;

FIG. 8 is a schematic diagram of an electronic device suitable for use in implementing some embodiments of the present application.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. The described embodiments are only some embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present application may be combined with each other without conflict.

It should be understood that "system", "apparatus", "unit" and/or "module" as used herein is a method for distinguishing different components, elements, parts or assemblies at different levels. However, other words may be substituted by other expressions if they accomplish the same purpose.

As used in this application and the appended claims, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are intended to be inclusive in the plural unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements. An element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

Flow charts are used herein to illustrate operations performed by systems according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or several steps of operations may be removed from the processes.

The inventor of the application finds out through research that: the existing data migration tool, such as DataX, is an offline data synchronization tool, and has the following problems:

firstly, only supporting data migration of basic data structure and not supporting data migration of complex data structure

In order to ensure compatibility with enough databases, the DataX only has several basic data structure types (such as Long, Double, String, Date, Boolean, Bytes) inside, and cannot process complex data structure types such as Array, JSON, even Array (JSON) nesting and the like encountered in the migration process.

Secondly, only supporting the synchronous migration work among single data sources and not supporting the simultaneous migration of many-to-many data sources

The source data storage and the target data storage of the DataX only have to be one, which greatly limits the use scene of the DataX.

In order to solve the technical problem, the application provides a method, a device and a system for migrating data, by presetting data migration configuration information and defining a mapping relationship between a source database and a target database and a mapping relationship between a source data structure and a target data structure, data migration between different databases and between different data structures is realized, and the data migration requirement of a complex data structure is met.

Fig. 1 illustrates an exemplary system architecture to which some embodiments of the method for migrating data or the apparatus for migrating data of the present application may be applied. As shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. Various client applications, such as a browser, an internet of things management application, a system management application, and the like, may be installed on the

terminal devices

101, 102, and 103.

The

terminal apparatuses

101, 102, and 103 may be hardware or software. When the

terminal devices

101, 102, 103 are hardware, they may be various electronic devices, including but not limited to smart phones, tablet computers, ultra-mobile personal computers (UMPCs), netbooks, Personal Digital Assistants (PDAs), laptop portable computers, desktop computers, and other electronic devices. When the

terminal apparatuses

101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services) or as a single piece of software or software module. The embodiment of the present application does not set any limit to the specific type of the electronic device.

The server 105 may be a server that provides various services, such as a background server that provides data migration services to the

terminal devices

101, 102, 103. A user sends a data migration request through a browser or an application in the

terminal device

101, 102, or 103, and the background server may, in response to receiving the data migration request sent by the

terminal device

101, 102, or 103, perform processing such as analysis and execution on the request, obtain a processing result (e.g., a data migration result), and return the processing result.

The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.

It should be noted that the method for migrating data provided in the embodiment of the present application is generally performed by the server 105, and accordingly, the apparatus for migrating data is generally disposed in the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

With continuing reference to FIG. 2, a flow 200 of one embodiment of a method for migrating data of the present application is shown, the subject of execution of which may be the server 105 shown in FIG. 1, the method comprising the steps of:

s201: acquiring data to be migrated and data migration configuration information, wherein the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure;

the method and the device can realize data migration of different user systems, namely data migration based on user dimensionality, for example, data in different user systems deployed in an IoT (Internet of Things, Chinese name) platform is migrated to a SaaS (Software as a Service, Chinese name) platform.

One way to obtain the data to be migrated and the data migration configuration information may be: and then obtaining the data to be migrated and the data migration configuration information corresponding to the system identification according to the preset corresponding relation between the system identification and the data to be migrated and the data migration configuration information.

It should be noted that the above is only one optional embodiment for acquiring the data to be migrated and the data migration configuration information, and the present invention is not limited thereto.

The data to be migrated is stored in at least one source database in the original system, and the data to be migrated is stored in at least one target database in the new system after data migration, that is, the invention can support synchronous data migration between multiple data sources and multiple data sources.

The data to be migrated is stored in at least one source database in a source data structure, the data to be migrated needs to be stored in a target database in a target data structure, and for a certain data to be migrated, when the source data structure is different from the target data structure, data structure conversion needs to be performed.

As in the scenario of migrating the IoT platform to the SaaS platform, the source database may be MongoDB, Redis, RabbitMQ, etc., the target database may be PostgreSQL, hitsbb, Redis, etc., and the source data structure and the target data structure may be basic data structures, such as Long, Double, String, Date, Boolean, Bytes, etc., or complex data structures, such as Array, JSON, Array (JSON) nest, etc.

S202: creating at least one data migration task based on the data migration configuration information and the data to be migrated;

and creating at least one data migration task based on the data migration configuration information and the data to be migrated, and improving the data migration efficiency by executing the data migration tasks in parallel.

It can be understood that the larger the amount of data to be migrated, the more data migration tasks need to be created.

Based on the data migration configuration information, data migration tasks may be created by data category. Data migration tasks may also be created by the storage time of the data.

S203: and executing at least one data migration task to store the data respectively corresponding to the at least one data migration task in the corresponding target database in the corresponding target data structure.

According to the data volume to be migrated and the processing capacity of the system, whether all data migration tasks corresponding to the data to be migrated are executed in parallel or in batch can be determined. If the data migration tasks are executed in batch, the data migration tasks in the same batch are executed in parallel.

With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario of the method for migrating data according to the present embodiment. In the application scenario of fig. 3, the original system is deployed on the IoT platform, the new system is deployed on the SaaS platform, and the data is migrated from the IoT platform to the SaaS platform. Data to be migrated in an original system are stored in MongoDB, Redis and RabbitMQ, namely a source database is MongoDB, Redis and RabbitMQ, the data to be migrated need to be stored in PostgreSQL, HiTSDB and Redis in a new system, namely target databases are PostgreSQL, HiTSDB and Redis, a source data structure of the data to be migrated in the source database can be Long, Double, String, Date, Boolean and Bytes, and the data to be migrated needs to be converted into a target data structure JSON, Array and Array (JSON) before being stored in the target database. In order to implement data migration in the application scenario, data migration configuration information needs to be preset, and a mapping relationship between a source database of MongoDB, Redis, RabbitMQ and a target database of PostgreSQL, hitsbb, Redis, and a mapping relationship between a source data structure Long, Double, String, Date, Boolean, Bytes and a target data structure JSON, Array (JSON) are configured in the data migration configuration information.

According to the method for migrating data disclosed by the embodiment, the mapping relationship between the source database and the target database and the mapping relationship between the source data structure and the target data structure are defined by presetting the data migration configuration information, so that data migration between different databases and between different data structures is realized, and the data migration requirement of a complex data structure is met.

In some optional implementation manners of the foregoing embodiment, the data migration configuration information includes a data category set, each data category in the data category set corresponds to one migration priority, and the data migration task is created and executed according to the migration priority by setting the migration priority of each data category in the data migration configuration information.

On this basis, please refer to fig. 4, which shows the step S202 in the above embodiment: the method for creating at least one data migration task based on the data migration configuration information and the data to be migrated comprises the following steps:

s401: dividing the data to be migrated into a plurality of data groups based on the data types in the data type set in the data migration configuration information, wherein the data types of different data groups are different;

s402: sorting the plurality of data groups according to the migration priority of the data category to obtain a data group sequence;

s403: and sequentially creating at least one data migration task for each data group from the data group with the highest migration priority in the data group sequence based on the data migration configuration information, wherein the migration priority of the data group is the same as the migration priority of the corresponding data category.

Accordingly, in the above embodiment, S203: executing at least one data migration task to store data respectively corresponding to the at least one data migration task in a corresponding target database in a corresponding target data structure, including:

and sequentially executing the data migration tasks created for each data group according to the migration priority of the data groups from high to low so as to sequentially store the data in the data groups with different migration priorities in the corresponding target database in the corresponding target data structure.

For example, the data category in the data category combination includes user data, device data, a core log and a non-core log, and the migration priorities of the user data, the device data, the core log and the non-core log are sequentially decreased. In the migration process, user data, equipment data, a core log and a non-core log are sequentially migrated, a main key of the user data is obtained after the data migration of the user data is completed, the migration process is controlled based on the main key of the user data, the migration of the equipment data, the core log and the non-core log is carried out, the strict and controllable migration data range is ensured, and offline migration and online migration can be realized.

Specifically, in the process of sequentially migrating user data, device data, a core log and a non-core log, at least one data migration task is created for the user data, the device data is temporarily stored in a source message queue, the data migration task is sent to an available work process, and the work process executes the corresponding data migration task until all data migration tasks of the user data are executed; then creating at least one data migration task for the device data in the source message queue, wherein the source message queue executes the data migration task, the device data in the source message queue is migrated to the target message queue, and the target message queue stores the device data in the target database until all the device data in the source message queue is stored in the target database; then creating at least one data migration task for the core log, sending the data migration task to an available work process, and executing the corresponding data migration task by the work process until all the data migration tasks of the core log are executed; and finally, creating at least one data migration task for the non-core log, sending the data migration task to an available work process, and executing the corresponding data migration task by the work process until all the data migration tasks of the non-core log are executed.

In the online migration process, newly added user data can be automatically migrated, invalid data (namely data of which the primary key does not exist in a new system) of the device data, the core log and the non-core log in the migration process can not be scanned, and further the invalid data can not be migrated into the new system, the problem of system crash caused by the fact that the primary key of the data in the new system does not exist can not occur, and the stability and reliability of the whole data migration process can be ensured.

The whole data migration process is transparent to the user, the user can continue to use the original system for operation, the server 105 pushes prompt information for indicating to log in the new system to the user when the server inquires that the data migration task corresponding to the user data is completely completed and if the fact that the user operates in the original system is detected, the user logs in the new system and continues to perform related operation, it is guaranteed that the online service of the user does not need to be stopped in the data migration process, the user is friendly to switch the new system and the old system, data is safe and not lost, and user experience is improved.

With further reference to FIG. 5, a flow 500 of yet another embodiment of a method for migrating data is shown. The method for migrating data comprises the following steps:

s501: acquiring data to be migrated and data migration configuration information, wherein the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure;

s502: under the condition of receiving a detection request which is sent by a user and carries a detection range of data to be migrated, creating a pre-migration detection task according to the detection range;

the user can select the data range for performing the pre-migration detection, perform the pre-migration detection on all the data to be migrated, and perform the pre-migration detection on part of the data to be migrated.

S503: splitting a pre-migration detection task into a plurality of detection tasks, wherein different detection tasks respectively correspond to different data groups in a detection range of data to be migrated;

it should be noted that, when the detection range is large, the pre-migration detection task may be split into multiple detection tasks according to the data migration configuration information.

If the detection range includes more than one data type, the pre-migration detection task can be split into more than one data type detection task according to the data type, and then each data type detection task is further split to obtain a plurality of detection tasks.

S504: executing a plurality of detection tasks corresponding to the pre-migration detection task to obtain an execution result of each detection task;

and detecting that the execution result of the task is migration success or migration failure, and for the data with migration failure, further modifying the relevant information in the data migration configuration information, such as the mapping relationship between the source database and the target database and/or the mapping relationship between the source data structure and the target data structure.

For each detection task, the following detection operations are performed:

performing aggregation calculation on the data group corresponding to the detection task in the corresponding source database to obtain a first aggregation result of the data group corresponding to the detection task in the corresponding source database;

the simulation stores the data group corresponding to the detection task in a corresponding target database in a corresponding target data structure, and performs aggregation calculation on the data group stored due to the execution of the detection task in the corresponding target database to obtain a second aggregation result of the data group corresponding to the detection task in the corresponding target database;

and determining whether the data group corresponding to the detection task has a migration problem or not by comparing the first aggregation result with the second aggregation result.

Specifically, when the first aggregation result is consistent with the second aggregation result, the data group corresponding to the detection task has no problem, and the migration is successful, otherwise, the data group corresponding to the detection task has a problem, and the migration fails.

S505: under the condition that the execution result of each detection task of the pre-migration detection tasks is successful in migration, at least one data migration task is created based on the data migration configuration information and the data to be migrated;

s506: and executing at least one data migration task to store the data respectively corresponding to the at least one data migration task in the corresponding target database in the corresponding target data structure.

Compared with the embodiment corresponding to fig. 2, the flow 500 of the method for migrating data in this embodiment adds premigration detection, and premigration detection may be performed on all or part of data to be migrated before data migration is formally performed, and if the execution result of the detection task of premigration detection is migration failure, it indicates that there is data conflict or incompatibility, and a technician may modify and perfect the relevant information in the data migration configuration information according to the execution result, thereby ensuring the reliability of formally performing data migration.

Based on all the embodiments described above, in relation to the execution of the data migration task and the execution of the pre-migration detection task, in order to enable the user to obtain the execution states of the tasks in real time, the execution states of the tasks may be queried.

Specifically, a state detection task is first obtained, where the state detection task includes a task identifier, and the task identifier may be an identifier of a data migration task or an identifier of a detection task in a pre-migration detection task. Methods of obtaining status detection tasks include, but are not limited to including: a user inputs a task needing state detection through a browser, and if the user inputs a data migration task of user data through the browser, the execution states of the data migration tasks of all the user data are detected; automatically detecting a detection task in a data migration detection task or a pre-migration detection task which enters an execution state; and automatically detecting a certain preset task entering an execution state.

And then inquiring the execution state of the task corresponding to the task identification.

Further, the execution statuses of the plurality of data migration tasks or the plurality of detection tasks in the pre-migration detection tasks of each data category may also be summarized, for example, the percentage of the tasks in the to-be-executed status, the percentage of the tasks in the execution success status, the percentage of the tasks in the execution failure status, and the like in the data migration tasks of the user data. The method is convenient for more intuitively knowing the execution state and progress of the task.

It should be noted that after the state detection task including the task identifier is acquired and the execution state of the task corresponding to the task identifier is queried, the corresponding query result may be returned to the terminal device that sent the state detection task, so that the user can view the state detection task.

With further reference to fig. 6, the present application provides an embodiment of an apparatus for migrating data, which corresponds to the embodiment of the method shown in fig. 2, and which may be applied in various electronic devices in particular.

As shown in fig. 6, the apparatus 600 for migrating data of the present embodiment includes: an acquisition unit 601, a creation unit 602, an execution unit 603.

The obtaining unit 601 is configured to obtain data to be migrated and data migration configuration information, where the data to be migrated originates from at least one source database, and the data migration configuration information includes a mapping relationship between the source database and a target database and a mapping relationship between a source data structure and a target data structure; a creating unit 602 configured to create at least one data migration task based on the data migration configuration information and the data to be migrated; an executing unit 603 configured to execute at least one data migration task to store data corresponding to the at least one data migration task in a corresponding target database in a corresponding target data structure.

In this embodiment, specific processes of the obtaining unit 601, the creating unit 602, and the executing unit 603 and technical effects brought by the specific processes may refer to related descriptions of the embodiments of step 201, step 202, and step 203 in the corresponding embodiment of fig. 2, which are not described herein again.

In some embodiments, the creating unit 602 is specifically configured to: dividing the data to be migrated into a plurality of data groups based on the data types in the data type set, wherein the data types of different data groups are different; sorting the plurality of data groups according to the migration priority of the data category to obtain a data group sequence; and sequentially creating at least one data migration task for each data group from the data group with the highest migration priority in the data group sequence based on the data migration configuration information, wherein the migration priority of the data group is the same as the migration priority of the corresponding data category.

In some embodiments, the executing unit 603 is specifically configured to: and sequentially executing the data migration tasks created for each data group according to the migration priority of the data groups from high to low so as to sequentially store the data in the data groups with different migration priorities in the corresponding target database in the corresponding target data structure.

In some embodiments, the apparatus 600 for migrating data further includes a premigration detection unit, which is specifically configured to: under the condition of receiving a detection request which is sent by a user and carries a detection range of data to be migrated, creating a pre-migration detection task according to the detection range; splitting a pre-migration detection task into a plurality of detection tasks, wherein different detection tasks respectively correspond to different data groups in a detection range of data to be migrated; and executing a plurality of detection tasks corresponding to the pre-migration detection task to obtain an execution result of each detection task.

In some embodiments, in the premigration detection unit, executing a plurality of detection tasks corresponding to the premigration detection task to obtain an execution result of each detection task, where the executing result includes:

for each detection task, the following detection operations are performed: performing aggregation calculation on the data group corresponding to the detection task in the corresponding source database to obtain a first aggregation result of the data group corresponding to the detection task in the corresponding source database; the simulation stores the data group corresponding to the detection task in a corresponding target database in a corresponding target data structure, and performs aggregation calculation on the data group stored due to the execution of the detection task in the corresponding target database to obtain a second aggregation result of the data group corresponding to the detection task in the corresponding target database; and determining whether the data group corresponding to the detection task has a migration problem or not by comparing the first aggregation result with the second aggregation result.

In some embodiments, the apparatus 600 for migrating data further includes: the state detection unit is used for acquiring a state detection task, and the state detection task comprises a task identifier; and inquiring the execution state of the task corresponding to the task identification.

In some embodiments, the apparatus 600 for migrating data further includes: and the login prompting unit is used for pushing prompting information for indicating to login a new system corresponding to the data to be migrated to the user if the fact that the user operates in the original system corresponding to the data to be migrated is detected under the condition that the data migration task corresponding to the user data is completely finished is inquired.

It should be noted that the apparatus 600 for migrating data may be a chip, a component or a module, the apparatus 600 for migrating data may include a processor and a memory, the obtaining unit 601, the creating unit 602, the executing unit 603, and the like are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

The processor may include a kernel, which calls the corresponding program unit from the memory. One or more of the kernels can be set, and the efficiency of data migration is improved by adjusting the parameters of the kernels.

The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.

The apparatus for migrating data provided in the foregoing embodiment of the present application sets data migration configuration information in advance, defines a mapping relationship between a source database and a target database and a mapping relationship between a source data structure and a target data structure, and creates a data migration task and executes the data migration task according to the data migration configuration information and data to be migrated, so that data migration between different databases and between different data structures is implemented, and a data migration requirement of a complex data structure is met.

With further reference to FIG. 7, a timing diagram of one embodiment of a system for migrating data is shown. Among other things, the system may be applied to the server 105 shown in fig. 1.

In this embodiment, a system for migrating data includes a scheduling device and at least one task execution device.

The scheduling device is configured to acquire data to be migrated and data migration configuration information, the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure; creating at least one data migration task based on the data migration configuration information and the data to be migrated; assigning the at least one data migration task to a task execution device of the at least one task execution device;

and the task execution device is configured to execute the received data migration task so as to store the data corresponding to the received data migration task in the corresponding target database in the corresponding target data structure.

As shown in fig. 7, in step 701, the scheduling apparatus obtains data to be migrated and data migration configuration information. The data to be migrated originates from at least one source database, and the data migration configuration information includes a mapping relationship between the source database and the target database and a mapping relationship between the source data structure and the target data structure.

In step 702, the scheduling device creates at least one data migration task based on the data migration configuration information and the data to be migrated.

In step 703, the scheduling device allocates the at least one data migration task to a task execution device of the at least one task execution device.

In step 704, the task execution device executes the received data migration task to store the data corresponding to the received data migration task in the corresponding target database in the corresponding target data structure.

In some optional implementation manners of this embodiment, the data migration configuration information includes a data category set, and each data category in the data category set corresponds to one migration priority.

In some embodiments, the scheduling apparatus is specifically configured to: dividing the data to be migrated into a plurality of data groups based on the data types in the data type set, wherein the data types of different data groups are different; sorting the plurality of data groups according to the migration priority of the data category to obtain a data group sequence; and sequentially creating at least one data migration task for each data group from the data group with the highest migration priority in the data group sequence based on the data migration configuration information, wherein the migration priority of the data group is the same as the migration priority of the corresponding data category.

In some embodiments, the task execution device is specifically configured to: and sequentially executing the data migration tasks created for each data group according to the migration priority of the data groups from high to low so as to sequentially store the data in the data groups with different migration priorities in the corresponding target database in the corresponding target data structure.

In some embodiments, the scheduling apparatus is further configured to create a pre-migration detection task according to a detection range when receiving a detection request carrying the detection range of the data to be migrated and sent by a user; splitting a pre-migration detection task into a plurality of detection tasks, wherein different detection tasks respectively correspond to different data groups in a detection range of data to be migrated; assigning the plurality of detection tasks to task execution devices of the at least one task execution device; the at least one task execution device is further configured to execute the received detection task to obtain an execution result of the detection task.

In some embodiments, for each of the plurality of detection tasks, the task performing device that received the detection task may perform the following detection operations: performing aggregation calculation on the data group corresponding to the detection task in the corresponding source database to obtain a first aggregation result of the data group corresponding to the detection task in the corresponding source database; the simulation stores the data group corresponding to the detection task in a corresponding target database in a corresponding target data structure, and performs aggregation calculation on the data group stored due to the execution of the detection task in the corresponding target database to obtain a second aggregation result of the data group corresponding to the detection task in the corresponding target database; and determining whether the data group corresponding to the detection task has a migration problem or not by comparing the first aggregation result with the second aggregation result.

In some embodiments, the scheduling device is further configured to acquire a status detection task, where the status detection task includes a task identifier; and inquiring the execution state of the task corresponding to the task identification.

In some embodiments, the scheduling device is further configured to, when it is found that the data migration task corresponding to the user data is completely completed, push, to the user, prompt information for instructing to log in a new system corresponding to the data to be migrated if it is detected that the user performs an operation in the original system corresponding to the data to be migrated.

When allocating tasks, such as a data migration task and a detection task, the scheduling device may preferentially allocate the tasks to be allocated to the task execution devices that have not allocated the tasks or have completed the tasks.

It should be understood that the scheduling device and the task execution device may be chips, components or modules. The scheduling device and the task execution device may include a processor and a memory, respectively. For any one of the scheduling apparatus and the task execution apparatus, the memory included in the apparatus may store a program unit for implementing operations performed by the apparatus, and the processor included in the apparatus may execute the program unit stored in the memory to implement corresponding functions.

The system provided by the embodiment of the application presets the data migration configuration information, defines the mapping relationship between the source database and the target database and the mapping relationship between the source data structure and the target data structure, and creates the data migration task and executes the data migration task according to the data migration configuration information and the data to be migrated, so that the data migration between different databases and between different data structures is realized, and the data migration requirement of a complex data structure is met.

Referring now to FIG. 8, shown is a schematic diagram of an electronic device 800 suitable for use in implementing some embodiments of the present application. The electronic device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 8, the electronic device 800 may include a processor 801, a memory 802, a communication interface 803, an input unit 804, an output unit 805, and a communication bus 806. Wherein the processor 801 and the memory 802 are connected to each other by a communication bus 806. A communication interface 803, an input unit 804 and an output unit 805 are also connected to the communication bus 806.

The communication interface 803 may be an interface of a communication module, such as an interface of a GSM module. The communication interface 803 may be used to obtain a data migration request, a pre-migration detection request, a task status query request, and the like sent by the terminal device.

In the embodiment of the present application, the processor 801 may be a Central Processing Unit (CPU), an application-specific integrated circuit (ASIC), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices.

In one possible implementation, the memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function (such as a task creation function, a task execution function, a task state query function, and the like), and the like; the storage data area may store data created during use of the computer, such as data migration configuration information and the like.

Further, the memory 802 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device.

The processor 801 may call a program stored in the memory 802, and in particular, the processor 801 may execute the method for migrating data as shown in any of the embodiments of fig. 2, fig. 4, and fig. 5 above.

The memory 802 is used for storing one or more programs, the programs may include program codes, the program codes include computer operation instructions, and in this embodiment, the memory 802 stores at least the programs for implementing the following functions:

acquiring data to be migrated and data migration configuration information, wherein the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure;

creating at least one data migration task based on the data migration configuration information and the data to be migrated;

and executing at least one data migration task to store the data respectively corresponding to the at least one data migration task in the corresponding target database in the corresponding target data structure.

The present application may further include an input unit 804, and the input unit 804 may include at least one of a touch sensing unit that senses a touch event on the touch display panel, a keyboard, a mouse, a camera, a microphone, and the like.

The output unit 805 may include: at least one of a display, a speaker, a vibration mechanism, a light, and the like. The display may comprise a display panel, such as a touch display panel or the like. In one possible case, the Display panel may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. The vibration mechanism may displace the electronic device 800 during operation, and in one possible implementation, the vibration mechanism includes a motor and an eccentric vibrator, and the motor drives the eccentric vibrator to rotate so as to generate vibration. The brightness and/or color of the lamp can be adjusted, in a possible implementation manner, different information can be embodied through at least one of the on-off, brightness and color of the lamp, for example, the alarm information can be embodied through red light emitted by the lamp.

Of course, the structure of the electronic device 800 shown in fig. 8 does not constitute a limitation of the electronic device in the embodiment of the present application, and in practical applications, the electronic device may include more or less components than those shown in fig. 8, or some components may be combined.

The present application provides a computer readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method for migrating data described in the above method embodiments.

The embodiment of the present application provides a processor, which is configured to execute a program, where the program executes a method for migrating data, which is described in the above method embodiments.

The present application further provides a computer program product which, when executed on a data processing device, causes the data processing device to implement the method for migrating data described in the above method embodiments.

In addition, the electronic device, the processor, the computer-readable medium, or the computer program product provided in the foregoing embodiments of the present application may be all used for executing the corresponding method provided above, and therefore, the beneficial effects achieved by the electronic device, the processor, the computer-readable medium, or the computer program product may refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, which include both non-transitory and non-transitory, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The above description is only for the purpose of illustrating the preferred embodiments of the present application and the technical principles applied, and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. The scope of the invention according to the present application is not limited to the specific combinations of the above-described features, and may also cover other embodiments in which the above-described features or their equivalents are arbitrarily combined without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

Claims

1. A method for migrating data, the method comprising:

and executing the at least one data migration task to store the data respectively corresponding to the at least one data migration task in the corresponding target database in the corresponding target data structure.

2. The method of claim 1, wherein the data migration configuration information comprises a set of data categories, and each data category in the set of data categories corresponds to a migration priority.

3. The method of claim 2, wherein the data categories in the set of data categories include user data, device data, a core log, and a non-core log, and wherein migration priorities of the user data, the device data, the core log, and the non-core log are sequentially decreased.

4. The method of claim 2, wherein creating at least one data migration task based on the data migration configuration information and the data to be migrated comprises:

dividing the data to be migrated into a plurality of data groups based on the data types in the data type set, wherein the data types of different data groups are different;

sorting the plurality of data groups according to the migration priority of the data category to obtain a data group sequence;

and sequentially creating at least one data migration task for each data group from the data group with the highest migration priority in the data group sequence based on the data migration configuration information, wherein the migration priority of the data group is the same as the migration priority of the corresponding data category.

5. The method of claim 4, wherein the executing the at least one data migration task to store data respectively corresponding to the at least one data migration task in a corresponding target database in a corresponding target data structure comprises:

6. The method of claim 4, wherein prior to the creating at least one data migration task based on the data migration configuration information and the data to be migrated, the method further comprises:

under the condition that a detection request which is sent by a user and carries the detection range of the data to be migrated is received, creating a pre-migration detection task according to the detection range;

splitting the pre-migration detection task into a plurality of detection tasks, wherein different detection tasks respectively correspond to different data groups in the detection range of the data to be migrated;

and executing a plurality of detection tasks corresponding to the pre-migration detection task to obtain an execution result of each detection task.

7. The method according to claim 6, wherein the executing the plurality of detection tasks corresponding to the premigration detection task to obtain the execution result of each detection task comprises:

for each of the detection tasks, performing the following detection operations:

8. The method according to any one of claims 1-7, further comprising:

acquiring a state detection task, wherein the state detection task comprises a task identifier;

and inquiring the execution state of the task corresponding to the task identification.

9. The method of claim 3, further comprising:

and under the condition that the data migration task corresponding to the user data is completely finished, if the fact that the user operates in the original system corresponding to the data to be migrated is detected, pushing prompt information for indicating to log in a new system corresponding to the data to be migrated to the user.

10. An apparatus for migrating data, the apparatus comprising:

the data migration system comprises an acquisition unit, a migration unit and a migration unit, wherein the acquisition unit is used for acquiring data to be migrated and data migration configuration information, the data to be migrated originates from at least one source database, and the data migration configuration information comprises a mapping relation between the source database and a target database and a mapping relation between a source data structure and a target data structure;

the creating unit is used for creating at least one data migration task based on the data migration configuration information and the data to be migrated;

and the execution unit is used for executing the at least one data migration task so as to store the data respectively corresponding to the at least one data migration task in the corresponding target database in the corresponding target data structure.

11. A computer-readable medium, on which a computer program is stored, which program, when being executed by a processor, carries out the method for migrating data according to any one of claims 1-9.

12. A processor for running a program, wherein the program when running implements the method for migrating data of any of claims 1-9.

13. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method for migrating data as recited in any of claims 1-9.

14. A system for migrating data, the system comprising scheduling means and at least one task execution means;

the scheduling device is configured to acquire data to be migrated and data migration configuration information, wherein the data to be migrated originates from at least one source database, and the data migration configuration information includes a mapping relationship between the source database and a target database and a mapping relationship between a source data structure and a target data structure; creating at least one data migration task based on the data migration configuration information and the data to be migrated; assigning the at least one data migration task to a task execution device of the at least one task execution device;

the at least one task execution device is configured to execute the received data migration task, so that data corresponding to the received data migration task is stored in a corresponding target database in a corresponding target data structure.