CN113986825A

CN113986825A - System, method and device for data migration, electronic equipment and readable storage medium

Info

Publication number: CN113986825A
Application number: CN202111607777.0A
Authority: CN
Inventors: 刘波; 孙元涛; 王旭; 辛玉龙; 卢华军; 李德怀
Original assignee: Beijing Xinghan Future Network Technology Co ltd
Current assignee: Beijing Xinghan Future Network Technology Co ltd
Priority date: 2021-12-27
Filing date: 2021-12-27
Publication date: 2022-01-28
Anticipated expiration: 2041-12-27
Also published as: CN113986825B

Abstract

The application belongs to the technical field of communication, and discloses a system, a method, a device, electronic equipment and a readable storage medium for data migration, wherein the method is applied to a scheduling server in a data migration system, the data migration system also comprises at least one cloud platform, and the method comprises the following steps: after determining that a data migration request aiming at target data to be migrated is received, acquiring data configuration information of the target data; according to the data configuration information, carrying out elastic capacity expansion on at least one cloud platform to obtain a server set, wherein the server set comprises at least two cloud servers; calling a cloud server in the server set, and carrying out data migration on target data; and when the data migration is determined to be completed, releasing the server set. When mass data are migrated in a long distance, the data migration cost is reduced.

Description

System, method and device for data migration, electronic equipment and readable storage medium

Technical Field

The present application relates to the field of communications technologies, and in particular, to a system, a method, an apparatus, an electronic device, and a readable storage medium for data migration.

Background

With the development of internet technology, internet data is more and more, and the difficulty of remote migration of mass data is also more and more. For example, because the east region has high energy consumption cost and large limitation, and the west region has low energy consumption cost and small limitation, mass data in the east region needs to be migrated to the west region to reduce the energy consumption cost, which requires remote migration of mass data. In the prior art, a dedicated line mode is usually adopted for remote migration of mass data, however, by adopting the mode, a plurality of dedicated lines need to be constructed, which consumes a large amount of hardware cost and time cost.

Therefore, when mass data is migrated in a long distance, how to reduce the data migration cost is a technical problem to be solved.

Disclosure of Invention

Embodiments of the present application provide a system, a method, an apparatus, an electronic device, and a readable storage medium for data migration, so as to reduce data migration cost when migrating mass data in a long distance.

In one aspect, a data migration system is provided, comprising a scheduling server and at least one cloud platform, wherein,

a scheduling server: the data migration method comprises the steps of obtaining data configuration information of target data after determining that a data migration request aiming at the target data to be migrated is received; according to the data configuration information, performing elastic capacity expansion on at least one cloud platform to obtain a server set; calling a cloud server in the server set, and carrying out data migration on target data; when the data migration is determined to be completed, releasing a server set, wherein the server set comprises at least two cloud servers;

each cloud platform: and the method is used for performing elastic capacity expansion to obtain the cloud server when the elastic capacity expansion instruction of the scheduling server is received.

In one embodiment, the data migration system is constructed based on a cloud native architecture, and the server set comprises at least one sending server and at least one receiving server; the data configuration information includes at least one of the following information: directory information, target migration time, source address information of the source device, and destination address information of the target device.

In one embodiment, the scheduling server is configured to:

acquiring the total capacity of the file to be transmitted according to the data configuration information;

determining the number of servers in the server set according to the total file capacity, the target migration duration and the data migration rate of the server set;

and according to the number of the servers, performing elastic expansion on at least one cloud platform to obtain a server set.

In one embodiment, the scheduling server is configured to:

determining a cloud platform corresponding to the source equipment according to source address information contained in the data configuration information;

based on the number of the servers, carrying out elastic expansion on the cloud platform corresponding to the source equipment, and determining a sending server after elastic expansion;

determining a cloud platform corresponding to the target equipment according to the destination address information contained in the data configuration information;

and based on the number of the servers, performing elastic expansion on the cloud platform corresponding to the target equipment to obtain the receiving server after elastic expansion.

In one embodiment, the scheduling server is configured to:

distributing target data, and respectively determining the distributed data blocks of each sending server, wherein the data blocks are obtained by dividing the target data;

each sending server is used for: reading the distributed data blocks from the source equipment based on the source address information in the data configuration information, splitting the distributed data blocks to obtain at least one data slice, and sending each data slice to a corresponding receiving server;

each receiving server is used for: and writing each received data slice into the target equipment based on the destination address information in the data configuration information.

In one embodiment, each sending server is configured to:

randomly selecting at least one data slice from the at least one data slice, encrypting each selected data slice respectively to obtain encrypted data slices, and sending the at least one data slice containing the encrypted data slices to a corresponding receiving server;

each receiving server is used for:

and decrypting the received encrypted data slice to obtain a decrypted data slice, and writing at least one data slice containing the decrypted data slice into the target device.

In one aspect, a method for data migration is provided, which is applied to a scheduling server in a data migration system, where the data migration system further includes at least one cloud platform, and the method includes:

after determining that a data migration request aiming at target data to be migrated is received, acquiring data configuration information of the target data;

according to the data configuration information, carrying out elastic capacity expansion on at least one cloud platform to obtain a server set, wherein the server set comprises at least two cloud servers;

calling a cloud server in the server set, and carrying out data migration on target data;

and when the data migration is determined to be completed, releasing the server set.

In the implementation process, when data migration is performed, a server set for data migration is obtained in an elastic capacity expansion mode, a special line does not need to be established, the running service is not affected, the complex steps of configuration of data migration equipment and software and hardware cost are reduced, the number of servers can be determined according to target data to be migrated, the data migration efficiency is improved, the server set can be released after the migration is completed, and the cost is further reduced.

In one embodiment, the elastically expanding at least one cloud platform according to the data configuration information to obtain a server set includes:

In the implementation process, the number of the servers can be determined according to the target data to be migrated, the data migration efficiency is improved,

in one embodiment, elastically expanding at least one cloud platform according to the number of servers to obtain a server set includes:

In the implementation process, the cloud platforms near the source device and the target device are respectively subjected to elastic expansion, so that the sending device near the source device and the receiving device near the target device can be obtained, data transmission can be performed through the sending device and the receiving device, and the transmission efficiency is improved.

In one embodiment, invoking a cloud server in a server set to perform data migration on target data includes:

calling each sending server respectively, and executing the following steps: reading the distributed data blocks from the source equipment based on the source address information in the data configuration information, splitting the distributed data blocks to obtain at least one data slice, and sending each data slice to a corresponding receiving server;

calling each receiving server respectively, and executing the following steps: and writing each received data slice into the target equipment based on the destination address information in the data configuration information.

In the implementation process, the data slices are used as data transmission units, and the data slices contained in the data block are transmitted in parallel, so that the data transmission efficiency is improved.

In one aspect, an apparatus for data migration is provided, and is applied to a scheduling server in a data migration system, where the data migration system further includes at least one cloud platform, and the apparatus includes:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring data configuration information of target data after determining that a data migration request aiming at the target data to be migrated is received;

the capacity expansion unit is used for elastically expanding the capacity of at least one cloud platform according to the data configuration information to obtain a server set, and the server set comprises at least two cloud servers;

the migration unit is used for calling the cloud servers in the server set and carrying out data migration on the target data;

and the releasing unit is used for releasing the server set when the data migration is determined to be completed.

In one embodiment, the capacity expansion unit is configured to:

In one embodiment, the migration unit is configured to:

In one aspect, an electronic device is provided, comprising a processor and a memory, the memory storing computer readable instructions which, when executed by the processor, perform the steps of the method provided in any of the various alternative implementations of data migration described above.

In one aspect, a readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, is adapted to carry out the steps of the method as provided in any of the various alternative implementations of data migration described above.

In one aspect, a computer program product is provided which, when run on a computer, causes the computer to perform the steps of the method as provided in any of the various alternative implementations of data migration described above.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 is a schematic view of an application scenario of data migration according to an embodiment of the present application;

fig. 2 is a flowchart of an implementation of a method for data migration according to an embodiment of the present application;

fig. 3 is a schematic diagram of data slice transmission according to an embodiment of the present application;

fig. 4 is a block diagram illustrating a structure of a data migration apparatus according to an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.

First, some terms referred to in the embodiments of the present application will be described to facilitate understanding by those skilled in the art.

The terminal equipment: may be a mobile terminal, a fixed terminal, or a portable terminal such as a mobile handset, station, unit, device, multimedia computer, multimedia tablet, internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system device, personal navigation device, personal digital assistant, audio/video player, digital camera/camcorder, positioning device, television receiver, radio broadcast receiver, electronic book device, gaming device, or any combination thereof, including the accessories and peripherals of these devices, or any combination thereof. It is also contemplated that the terminal device can support any type of interface to the user (e.g., wearable device), and the like.

A server: the cloud server can be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, and can also be a cloud server for providing basic cloud computing services such as cloud service, a cloud database, cloud computing, cloud functions, cloud storage, network service, cloud communication, middleware service, domain name service, security service, big data and artificial intelligence platform and the like.

In order to reduce data migration cost when mass data is migrated in a long distance, embodiments of the present application provide a system, a method, an apparatus, an electronic device, and a readable storage medium for data migration.

Fig. 1 is a schematic view of an application scenario of data migration according to an embodiment of the present application. The application scene comprises the following steps: the system comprises user equipment, a data migration system, source equipment and target equipment.

The data migration system is constructed based on a cloud native architecture, and comprises a scheduling server and at least one cloud platform.

The user equipment: the data migration system is used for sending a data migration request aiming at target data to be migrated to a scheduling server in the data migration system based on a data migration instruction of a user.

Optionally, the user equipment may be a terminal device, and may also be a server, which is not limited herein.

A source device: is a device that stores target data.

The target device: and migrating the target data to the equipment indicated by the user.

In one embodiment, the source device and the target device may each include at least one of the following databases:

object Storage Service (OSS), Network Attached Storage (NAS), table Storage (Tablestore), Distributed File System (HDFS), Database File Storage (DBFS), Block Storage (EBS), and parallel File System (CPFS).

Optionally, the source device and the target device may be both servers or terminal devices, which is not limited herein.

A scheduling server: the data migration method comprises the steps of obtaining data configuration information of target data after determining that a data migration request aiming at the target data to be migrated is received; according to the data configuration information, performing elastic capacity expansion on at least one cloud platform to obtain a server set; calling a cloud server in the server set, and carrying out data migration on target data; and when the data migration is determined to be completed, releasing the server set.

In one embodiment, the server set includes at least two cloud servers (ECSs). The server set comprises at least one sending server and at least one receiving server; the data configuration information includes at least one of the following information: directory information, target migration time, source address information of the source device, and destination address information of the target device.

Alternatively, the server set may also be a set of other types of servers, and is not limited herein.

In one embodiment, when obtaining the server set, the scheduling server may adopt the following steps: acquiring the total capacity of the file to be transmitted according to the data configuration information; and determining the number of servers in the server set according to the total file capacity, the target migration duration and the data migration rate of the server set, and performing elastic expansion on at least one cloud platform according to the number of the servers to obtain the server set.

In one embodiment, when performing flexible capacity expansion, the scheduling server may adopt the following steps: according to the source address information contained in the data configuration information, determining a cloud platform corresponding to the source equipment, performing elastic expansion on the cloud platform corresponding to the source equipment based on the number of servers, determining a sending server after elastic expansion, determining a cloud platform corresponding to the target equipment based on the destination address information contained in the data configuration information, and performing elastic expansion on the cloud platform corresponding to the target equipment based on the number of servers to obtain a receiving server after elastic expansion.

For example, if the source device is located in beijing, the cloud platform of beijing is elastically expanded, and if the target device is located in wulan scotch, the cloud platform of wulan scotch is elastically expanded.

In one embodiment, the following steps may be adopted during data migration:

and the dispatching server distributes the target data and respectively determines the distributed data blocks of each sending server. The data block is obtained by dividing target data; each sending server reads the distributed data block from the source equipment based on the source address information in the data configuration information, splits the distributed data block to obtain at least one data slice, and sends each data slice to the corresponding receiving server. And each receiving server writes each received data slice into the target equipment based on the destination address information in the data configuration information.

In one embodiment, when transmitting the data slice, the following steps can be adopted:

each sending server randomly selects at least one data slice from the at least one data slice, encrypts the selected data slice respectively to obtain an encrypted data slice, and sends the at least one data slice containing the encrypted data slice to the corresponding receiving server. And each receiving server decrypts the received encrypted data slice to obtain a decrypted data slice, and writes at least one data slice containing the decrypted data slice into the target equipment.

In the embodiment of the application, the execution main body is a scheduling server. Referring to fig. 2, an implementation flowchart of a method for data migration according to an embodiment of the present application is described with reference to the application scenario shown in fig. 1, where the method includes the following specific implementation flows:

step 200: and after determining that a data migration request aiming at target data to be migrated is received, acquiring data configuration information of the target data.

In one embodiment, a user device creates a data migration task in response to a task creation operation of a user for a data migration page, obtains data configuration information of target data to be migrated in response to an input operation of the user, and sends a data migration request for the target data to a scheduling server based on a data migration instruction of the user. And after receiving the data migration request, the scheduling server acquires the data configuration information in the data migration request.

In one embodiment, after receiving and storing data migration requests respectively sent by one or more user equipments, a scheduling server scans the stored data migration requests, and sequentially executes subsequent data migration operations for each scanned data migration request.

It should be noted that the operation of submitting the data migration request by the user equipment and the operation of executing the data migration request by the server may be synchronous or asynchronous, and are not limited herein.

Optionally, the data configuration information may include at least one of the following information: data identification information, directory information, target migration duration, source address information of the source device, destination address information of the target device, data type, file system, mount point, private network, switch, and account information.

In practical application, the data configuration information may be set according to a practical application scenario, which is not limited herein.

Thus, the data configuration information of the target data can be obtained based on the data migration request of the user.

Step 201: and performing elastic capacity expansion on at least one cloud platform according to the data configuration information to obtain a server set.

In one embodiment, the following steps may be adopted when step 201 is executed:

s2011: and acquiring the total volume of the file to be transmitted according to the data configuration information.

In one embodiment, the total file capacity contained in the data configuration information is obtained.

In one embodiment, according to directory information in the data configuration information, the source device is called to compress the target data, and the total file capacity of the compressed file after processing is obtained.

In one embodiment, the implementation process of compressing the target data includes: respectively acquiring the file capacity of each file in the target data according to the directory information, and screening out files with the file capacity larger than the preset capacity; calling source equipment, respectively carrying out logic splitting on each screened file according to preset capacity to obtain a logic subfile corresponding to each file, enabling the file capacity of each logic subfile to be smaller than the preset capacity, dividing the files which are not screened and the logic subfiles according to the preset capacity to obtain at least one file set, and respectively compressing each file set to obtain a compressed file.

It should be noted that, in order to facilitate file transmission and file integrity, in the embodiment of the present application, a large file (i.e., a screened file) is logically divided, rather than physically divided.

Therefore, the target data can be packed and compressed, and a compressed file convenient for data transmission is obtained.

S2012: and determining the number of servers in the server set according to the total file capacity, the target migration duration and the data migration rate of the server set.

In one embodiment, the implementation process for determining the number of servers in the server set according to the total file capacity, the target migration duration, and the data migration rate of each cloud server includes: and acquiring the target migration time length and the data migration rate of the server set, and determining the number of servers of the server set according to the total file capacity, the target migration time length and the data migration rate of the server set.

In practical application, the target migration duration may be set according to a practical application scenario, for example, 5 days, and is not limited herein.

The server set comprises at least one sending server and at least one receiving server. The sending server and the receiving server are cloud servers. The data migration rate of the server set refers to: the rate of data transmission between the sending server and the receiving server. The data migration rates of different server combinations may be the same or different, and are not limited herein.

The number of the servers is positively correlated with the total file capacity, and is negatively correlated with the target migration duration and the data migration rate.

In one embodiment, the product of the target migration duration and the data migration rate of a single server is determined, the file transfer capacity of the single server is obtained, and the number of servers is determined according to the ratio of the total file capacity to the file transfer capacity.

In this way, the required number of servers, i.e. the required scheduling resources, can be obtained.

S2013: and according to the number of the servers, performing elastic expansion on at least one cloud platform to obtain a server set.

In one embodiment, when performing S2013, the following steps may be adopted:

step 1: and determining a cloud platform corresponding to the source equipment according to the source address information contained in the data configuration information.

Specifically, the source address information may include a network address of the source device and may also include a geographic address of the source device. The cloud platform corresponding to the source device is determined according to the network distance or the geographic distance between the cloud platform and the source device.

In one embodiment, a network address of each cloud platform in the data migration system is obtained, and according to the network address in the source address information and the network address of each cloud platform, a network distance between the source device and each cloud platform is determined, and a cloud platform corresponding to the source device is determined, and is a cloud platform corresponding to a minimum network distance in each network distance.

In one embodiment, the geographic address of each cloud platform in the data migration system is obtained, the geographic distance between the source device and each cloud platform is determined according to the geographic address in the source address information and the geographic address of each cloud platform, and the cloud platform corresponding to the source device is determined and is the cloud platform corresponding to the minimum geographic distance in each geographic distance.

Thus, a cloud platform that is relatively close to the source device can be determined.

Step 2: and based on the number of the servers, performing elastic expansion on the cloud platform corresponding to the source equipment to obtain the sending server after the elastic expansion.

In one embodiment, an elastic capacity expansion service is called to elastically expand a cloud platform corresponding to a source device, and a sending server after elastic capacity expansion and a network address of the sending server are obtained.

Optionally, the flexible capacity expansion service may be a computing power scheduling engine (BridgX) flexible capacity expansion service.

In one embodiment, the cloud platform is elastically expanded, that is, a cloud server of a cloud manufacturer is called as a sending server through an interface of the cloud manufacturer, and the cloud server is configured according to a set configuration and a set transmission mirror image, so that the configured cloud server is configured to the set configuration and installed with the set transmission mirror image.

Wherein the transfer image is a program for data migration. In practical application, both setting the memory and setting the transmission mirror image may be performed according to a practical application scenario, which is not limited herein. The network address may be an Internet Protocol (IP) address.

And step 3: and determining a cloud platform corresponding to the target equipment according to the destination address information contained in the data configuration information.

In one embodiment, the destination address information may include a network address of the target device and may also include a geographic address of the target device. The cloud platform corresponding to the target device is determined according to the network distance or the geographic distance between the cloud platform and the target device.

In one embodiment, network addresses of cloud platforms in the data migration system are obtained, and according to the network addresses in the destination address information and the network addresses of the cloud platforms, network distances between the target device and each cloud platform are determined, and a cloud platform corresponding to the target device is determined and is a cloud platform corresponding to the minimum network distance in the network distances.

In one embodiment, the geographic address of each cloud platform in the data migration system is obtained, the geographic distance between the target device and each cloud platform is determined according to the geographic address in the destination address information and the geographic address of each cloud platform, and the cloud platform corresponding to the target device is determined and is the cloud platform corresponding to the minimum geographic distance in each geographic distance.

Thus, a cloud platform which is relatively close to the target device can be determined.

And 4, step 4: and based on the number of the servers, performing elastic expansion on the cloud platform corresponding to the target equipment to obtain the receiving server after elastic expansion.

In an embodiment, based on a principle similar to that of obtaining the sending server, the cloud platform corresponding to the target device is elastically expanded to obtain the receiving server and the network address of the receiving server, which is not described herein again.

And 5: a server set is constructed that includes a sending server and a receiving server.

In one embodiment, a correspondence between a sending server and a receiving server in a server set is established.

Optionally, the correspondence between the sending server and the receiving server may also be one-to-one, one-to-many, many-to-one, or many-to-many, which is not limited herein.

In one embodiment, a one-to-one correspondence between a sending server and a receiving server is established based on the network address of the sending server and the network address of the receiving server.

In one embodiment, the network address of the sending server and the network address of the receiving server in the server set are stored in association with a database, such as a Database (DB), according to the above correspondence.

In practical applications, the database may be set according to practical application scenarios, which is not limited herein.

Therefore, the cloud platforms around the source device and the target device can be elastically expanded according to the target data to be migrated to obtain the cloud server for migrating the target data, and the sending server and the corresponding receiving server can be called to perform data migration in the subsequent steps according to the corresponding relation.

Step 202: and calling a cloud server in the server set to perform data migration on the target data.

In one embodiment, the following steps may be adopted when performing step 202:

s2021: and distributing the target data, and respectively determining the distributed data blocks of each sending server.

Specifically, the data blocks are obtained by dividing the target data according to the size of the compressed file and the number of servers. Optionally, the data block may be 1G or 10G, and in actual application, the size of the data block may also be set according to an actual application scenario, which is not limited herein.

In one embodiment, a transmission task for transmitting a compressed file is constructed based on the compressed file corresponding to target data, the compressed file corresponding to each transmission task is logically divided according to the size of the compressed file and the number of servers, data blocks required to be transmitted by each transmission task are obtained, and the data blocks are distributed to each sending server in a server set.

Therefore, the target data can be logically divided to obtain the divided data blocks, and the divided data blocks are distributed to the cloud servers, so that the data blocks can be transmitted in parallel through the cloud servers in the subsequent steps.

S2022: calling each sending server respectively, and executing the following steps: and reading the distributed data blocks from the source equipment based on the source address information in the data configuration information, splitting the distributed data blocks to obtain at least one data slice, and sending each data slice to a corresponding receiving server.

The data slice (Chunk) is a data transmission unit when the sending server performs data transmission, that is, the sending server sends one data slice to the receiving server each time.

Optionally, the size of the data slice may be 10M, and in practical application, the size of the data slice may be set according to a practical application scenario, which is not limited herein.

In one embodiment, S2022 is described by taking one sending server as an example, and when S2022 is executed, the following steps may be adopted:

step 1: the scheduling server transmits a data transmission instruction for transmitting the allocated data block to one transmission server.

In one embodiment, the data transmission instruction includes source address information and a network address of a receiving server corresponding to the transmitting server.

Step 2: and the sending server reads the distributed data block from the source equipment based on the data sending instruction of the scheduling server and the source address information in the data configuration information, and splits the distributed data block to obtain at least one split data slice.

And step 3: and the sending server respectively sends each data slice to the receiving server in the server set.

In one embodiment, at least one data slice is randomly selected from the at least one data slice, each selected data slice is encrypted to obtain an encrypted data slice, and the at least one data slice including the encrypted data slice is sent to a corresponding receiving server.

In one embodiment, the sending server further stores the data piece information of the encrypted data piece to the database. The data piece information may include at least one of the following information: the size of the data piece, the serial number of the encrypted data piece and the encryption and decryption modes.

Therefore, the randomly selected data slices can be encrypted, an attacker does not know which data slice is the encrypted data slice, so that the encrypted data slice is difficult to position and decrypt, and the attacker needs to acquire the complete target data after the data slices are combined, so that the problem that the attacker acquires the complete target data is solved by adopting a random encryption mode, the safety of the data is improved, all the data slices do not need to be encrypted by adopting the random encryption mode, the system resources and time consumed by data encryption are less, the resource cost and the time cost are reduced, and the efficiency of data slice transmission is improved.

S2023: calling each receiving server respectively, and executing the following steps: and writing each received data slice into the target equipment based on the destination address information in the data configuration information.

In one embodiment, taking one receiving server as an example for explanation, when executing S2022, the following steps may be adopted:

step 1: the scheduling server sends a data reception indication for receiving the data block to the receiving server.

In one embodiment, the data reception indication includes destination address information.

Step 2: the receiving server receives the data pieces sent by the sending server based on the data transmission instruction of the scheduling server, and sends the received data pieces to the target equipment based on the destination address information.

In one embodiment, the received encrypted data piece is decrypted to obtain a decrypted data piece, and at least one data piece containing the decrypted data piece is written to the target device.

In one embodiment, the receiving server obtains the data piece information of the encrypted data piece through the database, obtains the data piece sequence number and the encryption and decryption mode in the data piece information of the encrypted data piece, decrypts the encrypted data piece based on the encryption and decryption mode and the data piece sequence number, obtains the decrypted data piece, and sends the decrypted data piece to the target device. The target device stores each piece of data received.

It should be noted that the cloud servers may transmit the data pieces in a parallel manner.

Fig. 3 is a schematic diagram of data slice transmission according to an embodiment of the present application. Fig. 3 includes: the system comprises a source device, a plurality of sending servers, a plurality of receiving servers and a target device. And the sending server reads the data slice in the data block of the source equipment and sends the read data slice to the receiving server. And the receiving server sends the received data piece to the target equipment.

Further, the target device determines the data pieces with abnormal migration according to the data piece serial numbers of the stored data pieces, and sends the data piece serial numbers with abnormal migration to the scheduling server. And the scheduling server calls the server set to resend the data pieces with abnormal migration according to the data piece serial numbers with abnormal migration.

Optionally, the data slice with the abnormal migration may be a data slice with a failed transmission or a data slice with a failed storage, or may be other abnormal, which is not limited herein.

It should be noted that, during data transmission, data may be transmitted asynchronously or synchronously, and during data storage, data may be stored asynchronously or synchronously, which is not limited herein.

Step 203: and when the data migration is determined to be completed, releasing the server set.

In one embodiment, after the data migration is determined to be completed, the flexible capacity reduction service is called, the server set is released, that is, the called server set is returned to the cloud manufacturer through an interface of the cloud manufacturer, and the server charging is stopped.

In the embodiment of the application, the data migration system is constructed based on the cloud native architecture, server resources can be loaded according to the data migration requirements of users, the effective utilization of the server resources is realized, the data migration system has the technical characteristics of operation and maintenance automation, elastic expansion and contraction capacity, strong isolation, agile deployment and the like, the software and hardware cost and the time cost of data migration are reduced, the data migration tasks of the users can be uniformly scheduled, the starting and the recovery of the server resources can be reasonably distributed according to the size of the tasks, the distributed operation is supported, the problems of resource deficiency and long-time occupation which easily occur when single-machine mass data are migrated are avoided, the transmission efficiency is improved through the parallel transmission of data blocks, and randomly selected data pieces are encrypted, the safety of data transmission is ensured, and the high efficiency of data transmission is also ensured, the method can be applied to application scenes of east data migration and west migration of enterprise data remote backup, enterprise business remote sharing, east-west-arithmetic strategic engineering and the like, supports data migration among public clouds, private clouds, mixed clouds and cross-cloud manufacturers, and is wide in application range and strong in flexibility.

Based on the same inventive concept, the embodiment of the present application further provides a data migration apparatus, and because the principle of the apparatus and the device for solving the problem is similar to that of a data migration method, the implementation of the apparatus may refer to the implementation of the method, and repeated details are not repeated.

As shown in fig. 4, which is a schematic structural diagram of a data migration apparatus provided in an embodiment of the present application, and is applied to a scheduling server in a data migration system, where the data migration system further includes at least one cloud platform, and the data migration apparatus includes:

an obtaining unit 401, configured to obtain data configuration information of target data after determining that a data migration request for the target data to be migrated is received;

the capacity expansion unit 402 is configured to perform elastic capacity expansion on at least one cloud platform according to the data configuration information to obtain a server set, where the server set includes at least two cloud servers;

a migration unit 403, configured to invoke a cloud server in the server set, and perform data migration on the target data;

a releasing unit 404, configured to release the server set when it is determined that the data migration is completed.

In one embodiment, the capacity expansion unit 402 is configured to:

In one embodiment, the migration unit 403 is configured to:

Fig. 5 shows a schematic structural diagram of an electronic device 5000. Referring to fig. 5, the electronic device 5000 includes: the processor 5010 and the memory 5020 can optionally include a power supply 5030, a display unit 5040, and an input unit 5050.

The processor 5010 is a control center of the electronic apparatus 5000, connects various components using various interfaces and lines, and performs various functions of the electronic apparatus 5000 by running or executing software programs and/or data stored in the memory 5020, thereby monitoring the electronic apparatus 5000 as a whole.

In an embodiment of the application, the processor 5010, when calling a computer program stored in the memory 5020, executes the method of data migration as provided by the embodiment shown in fig. 2.

Optionally, the processor 5010 can include one or more processing units; preferably, the processor 5010 can integrate an application processor, which mainly handles operating systems, user interfaces, applications, etc., and a modem processor, which mainly handles wireless communications. It is to be appreciated that the modem processor described above may not be integrated into processor 5010. In some embodiments, the processor, memory, and/or memory may be implemented on a single chip, or in some embodiments, they may be implemented separately on separate chips.

The memory 5020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, and the like; the storage data area may store data created according to the use of the electronic device 5000, and the like. Further, the memory 5020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

The electronic device 5000 also includes a power supply 5030 (e.g., a battery) that provides power to the various components and that may be logically connected to the processor 5010 via a power management system to provide management of charging, discharging, and power consumption via the power management system.

The display unit 5040 may be configured to display information input by a user or information provided to the user, and various menus of the electronic device 5000, and in the embodiment of the present invention, the display unit is mainly configured to display a display interface of each application in the electronic device 5000 and objects such as texts and pictures displayed in the display interface. The display unit 5040 may include a display panel 5041. The Display panel 5041 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

The input unit 5050 may be used to receive information such as numbers or characters input by a user. Input units 5050 may include touch panel 5051 as well as other input devices 5052. Among other things, the touch panel 5051, also referred to as a touch screen, may collect touch operations by a user on or near the touch panel 5051 (e.g., operations by a user on or near the touch panel 5051 using a finger, a stylus, or any other suitable object or attachment).

Specifically, the touch panel 5051 can detect a touch operation by a user, detect signals resulting from the touch operation, convert the signals into touch point coordinates, transmit the touch point coordinates to the processor 5010, and receive and execute a command transmitted from the processor 5010. In addition, the touch panel 5051 may be implemented in various types, such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. Other input devices 5052 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on/off keys, etc.), a trackball, a mouse, a joystick, and the like.

Of course, the touch panel 5051 may cover the display panel 5041, and when the touch panel 5051 detects a touch operation thereon or thereabout, it is transmitted to the processor 5010 to determine the type of touch event, and then the processor 5010 provides a corresponding visual output on the display panel 5041 according to the type of touch event. Although in fig. 5, the touch panel 5051 and the display panel 5041 are implemented as two separate components to implement input and output functions of the electronic device 5000, in some embodiments, the touch panel 5051 and the display panel 5041 may be integrated to implement input and output functions of the electronic device 5000.

The electronic device 5000 may also include one or more sensors, such as pressure sensors, gravitational acceleration sensors, proximity light sensors, and the like. Of course, the electronic device 5000 may further include other components such as a camera according to the requirements of a specific application, and these components are not shown in fig. 5 and are not described in detail since they are not components used in this embodiment of the present application.

Those skilled in the art will appreciate that fig. 5 is merely an example of an electronic device and is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or different components.

In an embodiment of the present application, a readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the communication device may perform the steps in the above embodiments.

For convenience of description, the above parts are separately described as modules (or units) according to functional division. Of course, the functionality of the various modules (or units) may be implemented in the same one or more pieces of software or hardware when implementing the present application.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A data migration system is characterized by comprising a scheduling server and at least one cloud platform,

the scheduling server: the method comprises the steps of obtaining data configuration information of target data after determining that a data migration request aiming at the target data to be migrated is received; according to the data configuration information, performing elastic capacity expansion on the at least one cloud platform to obtain a server set; calling a cloud server in the server set to perform data migration on the target data; when the data migration is determined to be completed, releasing the server set, wherein the server set comprises at least two cloud servers;

each cloud platform: and the system is used for performing elastic capacity expansion to obtain the cloud server when the elastic capacity expansion instruction of the scheduling server is received.

2. The system of claim 1, wherein the data migration system is built based on a cloud-native architecture, a set of servers including at least one sending server and at least one receiving server; the data configuration information includes at least one of the following information: directory information, target migration time, source address information of the source device, and destination address information of the target device.

3. The system of claim 2, wherein the dispatch server is to:

and according to the number of the servers, performing elastic expansion on the at least one cloud platform to obtain the server set.

4. The system of claim 3, wherein the dispatch server is to:

based on the number of the servers, performing elastic expansion on the cloud platform corresponding to the source device, and determining a sending server after elastic expansion;

5. The system of claim 3, wherein the dispatch server is to:

distributing the target data, and respectively determining the distributed data blocks of each sending server, wherein the data blocks are obtained by dividing the target data;

6. The system of claim 5, wherein each sending server is to:

each receiving server is used for:

7. A data migration method is applied to a scheduling server in a data migration system, wherein the data migration system further comprises at least one cloud platform, and the method comprises the following steps:

according to the data configuration information, performing elastic capacity expansion on the at least one cloud platform to obtain a server set, wherein the server set comprises at least two cloud servers;

calling a cloud server in the server set to perform data migration on the target data;

and releasing the server set when the data migration is determined to be completed.

8. The method of claim 7, wherein the data migration system is built based on a cloud-native architecture, a set of servers including at least one sending server and at least one receiving server; the data configuration information includes at least one of the following information: directory information, target migration time, source address information of the source device, and destination address information of the target device.

9. The method of claim 8, wherein the elastically expanding the at least one cloud platform according to the data configuration information to obtain a server set comprises:

10. The method of claim 9, wherein the elastically expanding the at least one cloud platform according to the number of servers to obtain the set of servers comprises:

11. The method of claim 9, wherein the invoking a cloud server in the set of servers to perform data migration on the target data comprises:

12. A data migration apparatus, applied to a scheduling server in a data migration system, where the data migration system further includes at least one cloud platform, and the apparatus includes:

the capacity expansion unit is used for elastically expanding the capacity of the at least one cloud platform according to the data configuration information to obtain a server set, wherein the server set comprises at least two cloud servers;

13. The apparatus of claim 12, wherein the data migration system is built based on a cloud-native architecture, a set of servers including at least one sending server and at least one receiving server; the data configuration information includes at least one of the following information: directory information, target migration time, source address information of the source device, and destination address information of the target device.

14. The apparatus of claim 13, wherein the capacity expansion unit is to:

15. The apparatus of claim 14, wherein the capacity expansion unit is to:

16. The apparatus of claim 14, wherein the migration unit is to:

17. An electronic device comprising a processor and a memory, the memory storing computer readable instructions that, when executed by the processor, perform the method of any of claims 7-11.

18. A readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 7-11.