CN113138972A

CN113138972A - Data migration method and device, storage medium and electronic equipment

Info

Publication number: CN113138972A
Application number: CN202010053359.0A
Authority: CN
Inventors: 陈俊伟; 林金明; 王洋; 金薇薇; 仇路; 李鑫
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2020-01-17
Filing date: 2020-01-17
Publication date: 2021-07-20

Abstract

The present disclosure provides a data migration method and apparatus, a computer-readable storage medium, and an electronic device; relates to the technical field of data management. The data migration method comprises the following steps: receiving a data migration instruction; determining a data migration direction between each existing original cache data fragment and each newly deployed target cache data fragment in the cache data cluster; according to the data migration instruction, migrating the data to be migrated in the original cache data fragment to the target cache data fragment according to the data migration direction, and maintaining a migration state in the original cache data fragment; and sending the migration result recorded in the migration state to a migration management module. According to the method and the device, the capacity expansion of the cache system can be realized through data migration on the premise of no service halt, the management of the data state to be migrated is optimized in the migration process, and the migration can be started again from the interrupt part, so that the data migration process is more convenient and efficient.

Description

Data migration method and device, storage medium and electronic equipment

Technical Field

The present disclosure relates to the field of data management technologies, and in particular, to a data migration method, a data migration apparatus, an electronic device, and a computer-readable storage medium.

Background

In order to overcome the defect that a relational database is adopted to store data in a traditional development mode, a method for storing hot data by utilizing a cache is developed, and the high performance of the cache can provide data access with high throughput and low delay. However, with the increase of the number of mobile internet users, the single-machine cache gradually fails to meet the requirements of the users for storing and accessing mass data, and people begin to use the distributed cache system to store the data, so that the computing power of the server cluster can be utilized to meet mass concurrent requests of the users. For a distributed cache system, the most important capacity is capacity expansion capacity, and the distributed cache system can be implemented by deploying new storage space resources and migrating a part of data from an original storage space to a newly deployed storage space. Therefore, the capacity expansion process is essentially a process of data migration on part of data in the system.

The existing capacity expansion scheme for the distributed cache system mainly has the following three schemes: shutdown capacity expansion, mirror image smooth capacity expansion and slot position migration. Although the three schemes can realize the function of capacity expansion of the distributed cache system to a certain extent, a series of problems of complex operation, data mirroring, influence on data access of a service party, high coupling of a capacity expansion process and the service party, no universality for the existing distributed cache system and the like exist at the same time.

Therefore, it is necessary to provide a convenient and efficient data migration method with low coupling with a service system to implement the function of capacity expansion of the distributed cache system.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The embodiment of the disclosure provides a data migration method, a data migration apparatus, an electronic device, and a computer-readable storage medium, so that a capacity expansion function of a distributed cache system can be realized, and a series of problems that an operation is complex, data mirroring is required, data access by a service party is affected, a capacity expansion process is highly coupled with the service party, and the existing distributed cache system does not have universality in the capacity expansion process can be solved.

According to a first aspect of the present disclosure, there is provided a data migration method, including:

receiving a data migration instruction;

determining a data migration direction between each existing original cache data fragment and each newly deployed target cache data fragment in the cache data cluster;

according to the data migration instruction, migrating the data to be migrated in the original cache data fragment to the target cache data fragment according to the data migration direction, and maintaining a migration state in the original cache data fragment;

and sending the migration result recorded in the migration state to a management module.

In an exemplary embodiment of the present disclosure, the determining a data migration direction between existing original cache data fragments and newly deployed target cache data fragments in a cache data cluster includes:

determining the node position of each target cache data fragment in a consistent hash ring corresponding to the cache data cluster;

and determining the data migration direction according to the node position.

In an exemplary embodiment of the present disclosure, the data migration instruction includes a first migration step size and a second migration step size;

the migrating the data to be migrated in the original cache data fragment to the target cache data fragment according to the data migration direction, and maintaining the migration state in the original cache data fragment includes:

initializing a migration state file to maintain the migration state, wherein the migration state comprises a first copy offset and a second copy offset;

performing full migration on the data to be migrated in the original cache data fragment by taking the first migration step as a unit;

performing incremental migration on the newly added data to be migrated in the full migration process of the original cache data fragments by taking the second migration step as a unit until the second copy offset is consistent with the first copy offset;

the first copy offset is a copy offset of the original cache data fragment hot standby file, and the second copy offset is a copy offset maintained in the full migration and incremental migration processes.

In an exemplary embodiment of the present disclosure, the performing full migration on the data to be migrated in the original cache data fragment by using the first migration step as a unit includes:

taking the first migration step as a unit, reading data from each original cache data fragment in a data node number self-increment mode, and judging whether the read data is the data to be migrated;

and when the judgment result is yes, migrating the data to be migrated to the corresponding target cache data fragment according to the migration direction until the data node number corresponding to the read data is the maximum data node number of the original cache data fragment.

In an exemplary embodiment of the present disclosure, the performing incremental migration on the newly added to-be-migrated data of the original cache data segment in the full migration process by using the second migration step as a unit includes:

and taking the second migration step as a unit, starting from the second copy offset, and migrating the newly added to-be-migrated data in the full migration process of the original cache data fragment to the corresponding target cache data fragment.

In an exemplary embodiment of the present disclosure, the data to be migrated in the original cache data fragment is migrated in full;

the incremental migration of the newly added data to be migrated in the full migration process of the original cache data fragment includes:

sending a migration notification to the original cache data fragment, so that the original cache data fragment sends the information of the data to be migrated to the target cache data fragment, and the state of the data to be migrated is set to be in migration;

after receiving the information of the data to be migrated, the target cache data fragment caches the data to be migrated according to the information, and returns a successful migration message to the original cache data fragment;

and when the original cache data fragment receives the message of successful migration, deleting the data to be migrated and setting the migration to be completed.

In an exemplary embodiment of the present disclosure, the performing incremental migration on the newly added to-be-migrated data of the original cache data fragment in the full migration process until the second copy offset is consistent with the first copy offset includes:

performing incremental migration on the newly added data to be migrated in the full migration process of the original cache data fragments;

and when the difference value between the first copy offset and the second copy offset reaches a set threshold value, sending a read-only command to the original cache data fragment, and continuing the incremental migration until the second copy offset is consistent with the first copy offset.

According to a second aspect of the present disclosure, there is provided a data migration method, including:

in response to a data migration request, deploying one or more target cache data fragments;

sending a data migration instruction to a migration module so that the migration module migrates data to be migrated from original cache data fragments of a cache data cluster to the target cache data fragments;

and sending the information of the target cache data fragmentation to a routing agent module according to the migration result sent by the migration module so that the routing agent module updates a consistent hash ring corresponding to the cache data cluster and adds the target cache data fragmentation to a fragmentation routing rule.

In an exemplary embodiment of the present disclosure, a data migration request key is deployed in the data migration system;

in response to a data migration request, deploying one or more target cache data fragments, comprising:

and responding to the triggering operation of the data migration request key, and deploying one or more target cache data fragments.

According to a third aspect of the present disclosure, there is provided a data migration apparatus comprising:

the communication module is used for receiving a data migration instruction;

the direction determining module is used for determining the data migration direction between each existing original cache data fragment and each newly deployed target cache data fragment in the cache data cluster;

the migration module is used for migrating the data to be migrated in the original cache data fragment to the target cache data fragment according to the data migration direction and maintaining a migration state in the original cache data fragment;

and the result returning module is used for sending the migration result recorded in the migration state to the management module.

According to a fourth aspect of the present disclosure, there is provided a data migration apparatus comprising:

a response module for deploying one or more target cache data fragments in response to the data migration request;

the migration module is used for sending a data migration instruction to the migration module so as to enable the migration module to migrate data to be migrated from original cache data fragments of the cache data cluster to the target cache data fragments;

and the updating module is used for sending the information of the target cache data fragment to the routing agent module according to the migration result sent by the migration module so as to enable the routing agent module to update the consistent hash ring corresponding to the cache data cluster and add the target cache data fragment to the fragment routing rule.

According to a fifth aspect of the present disclosure, there is provided an electronic device comprising:

a processor;

a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the methods described above via execution of the executable instructions.

According to a sixth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of the above.

Exemplary embodiments of the present disclosure may have some or all of the following benefits:

in the data migration method provided in the exemplary embodiment of the present disclosure, after receiving a data migration instruction, a migration module first determines a data migration direction between each existing original cache data fragment and each newly deployed target cache data fragment in a cache data cluster; then, according to the received data migration instruction, migrating the data to be migrated in the original cache data fragment to a target cache data fragment according to the determined data migration direction, and maintaining the migration state in the original cache data fragment; and finally, after the migration is finished, the migration result recorded in the migration state is sent to a management module. On one hand, the data migration method provided by the embodiment of the disclosure maintains the migration state in the original cache data fragment, and when the migration is interrupted due to factors such as server restart and network link failure in the migration process, the interruption point in the migration process can be obtained according to the migration state, so that when the migration is performed again, all the operations are not performed again, and only the previous migration process needs to be continued according to the obtained interruption point, thereby improving the migration efficiency and fault tolerance. On the other hand, the data to be migrated can be managed conveniently according to the migration state maintained in the original cache data fragment, so that the coupling degree between the data migration process and the service system is reduced, and efficient data access can be performed in the migration process. Meanwhile, the data migration method has good adaptability and can be used in most distributed data cache systems.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.

FIG. 1 illustrates a schematic diagram of an exemplary system architecture to which a data migration method of an embodiment of the present disclosure may be applied;

FIG. 2 illustrates a schematic structural diagram of a computer system suitable for use with the electronic device used to implement embodiments of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a process of a shutdown migration method according to one embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow diagram of a process for smooth migration of images, according to one embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow diagram of a process for slot migration according to one embodiment of the present disclosure;

FIG. 6 schematically shows a flow chart of a process of a data migration method according to one embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow chart of a process of a data migration method according to one embodiment of the present disclosure;

FIG. 8 is a schematic diagram illustrating a distributed cache system architecture for an application scenario of one embodiment of the present disclosure;

FIG. 9 schematically illustrates a flow diagram of a data migration method of an application scenario of an embodiment of the present disclosure;

FIG. 10 schematically illustrates a flow diagram of a data migration method of an application scenario of an embodiment of the present disclosure;

FIG. 11 schematically illustrates a block diagram of a data migration method according to one embodiment of the present disclosure;

FIG. 12 schematically shows a block diagram of a data migration method according to one embodiment of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

Fig. 1 is a schematic diagram illustrating a system architecture of an exemplary application environment to which a data migration method and apparatus according to an embodiment of the present disclosure may be applied.

As shown in fig. 1, the system architecture 100 may include one or more of

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few. The

terminal devices

101, 102, 103 may be various electronic devices having a display screen, including but not limited to desktop computers, portable computers, smart phones, tablet computers, and the like. It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation. For example, server 105 may be a server cluster comprised of multiple servers, or the like.

The data migration method provided by the embodiment of the present disclosure may be executed by the server 105, and accordingly, the data migration apparatus may also be disposed in the server 105. The data migration method provided by the embodiment of the present disclosure may also be executed by the

terminal devices

101, 102, and 103 and the server 105 together, and accordingly, the data migration apparatus may be disposed in the

terminal devices

101, 102, and 103 and the server 105. This is not particularly limited in the present exemplary embodiment.

For example, in this exemplary embodiment, a user may send a data migration request through the server 105, and after receiving the data migration request, a cache system management website in the distributed cache system deploys one or more target cache data fragments in an idle server managed by the cache system management website in response to the data migration request, and sends a data migration instruction to the migration module; after receiving the migration instruction, the migration module determines a data migration direction between each existing original cache data fragment and each newly deployed target cache data fragment in the cache data cluster, migrates the data to be migrated in the original cache data fragment to the newly deployed target cache data fragment according to the determined data migration direction according to the received data migration instruction, and maintains a migration state in the original cache data fragment in the migration process; and after the migration operation of all the data to be migrated is finished, sending the migration result recorded in the migration state to the cache system management website. And the cache system management website sends the information of the target cache data fragment to the routing agent module according to the received migration result sent by the migration module, and the routing agent module updates a consistent hash ring corresponding to the cache data cluster according to the received information of the target cache data fragment and adds the target cache data fragment to the fragment routing rule.

FIG. 2 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present disclosure.

It should be noted that the computer system 200 of the electronic device shown in fig. 2 is only an example, and should not bring any limitation to the functions and the scope of the application of the embodiments of the present disclosure.

As shown in fig. 2, the computer system 200 includes a Central Processing Unit (CPU)201 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)202 or a program loaded from a storage section 208 into a Random Access Memory (RAM) 203. In the RAM 203, various programs and data necessary for system operation are also stored. The CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 204. An input/output (I/O) interface 205 is also connected to bus 204.

The following components are connected to the I/O interface 205: an input portion 206 including a keyboard, a mouse, and the like; an output section 207 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 208 including a hard disk and the like; and a communication section 209 including a network interface card such as a LAN card, a modem, or the like. The communication section 209 performs communication processing via a network such as the internet. A drive 210 is also connected to the I/O interface 205 as needed. A removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 210 as necessary, so that a computer program read out therefrom is mounted into the storage section 208 as necessary.

The technical solution of the embodiment of the present disclosure is explained in detail below:

the high-speed development of mobile internet services brings challenges of mass storage of data, high concurrent access, low delay and the like to background services. Conventional development models typically employ relational databases to store data and support access and modification operations on the stored data. However, when the system is exposed to high concurrent data access, problems such as increased delay, decreased throughput and the like can occur, and in severe cases, the database can be down, and the requirements of a business party cannot be met.

In consideration of characteristics of internet services, data is generally accessed more than modified, and the data has obvious hot spot characteristics. For the above reasons, in order to overcome the above disadvantages of using a relational database, a cache is used to store hot data, so as to provide high throughput and low latency data access by virtue of high performance of the cache. However, with the increasing of mobile internet users, the single-machine cache gradually fails to meet the storage and access requirements of mass data, and thus, the concept of a distributed cache system is provided. The distributed cache system is a data storage system which is established by segmenting hot spot data in a cross-server deployment mode, and can meet mass concurrent requests of users by utilizing the computing power of a server cluster.

For the distributed cache system described above, the most important capability is capacity expansion capability. Namely, when the existing cache system has a bottleneck of memory, a Central Processing Unit (CPU) or a network card resource, the processing capability of the whole distributed cache system is enhanced by adding a new data fragment to a new server and migrating the original cache data. To implement this data migration process, the inventors have tentatively proposed the following three ways:

the first method is shutdown capacity expansion, as shown in fig. 3, and the specific implementation steps are as follows:

step S310: deploying a new cache system with processing capacity stronger than that of the existing cache system;

step S320: carrying out shutdown operation on a service party;

step S330: migrating the data of the existing cache system to a new cache system;

step S340: and changing the data source configuration of the service party, pointing to a new cache system, and restarting the application.

However, in practice, the method has been found to have the following problems: firstly, in the process of capacity expansion, a service party is required to perform shutdown operation, and for service types with high availability requirements, the shutdown operation brings huge loss; secondly, because the method deploys a new cache system in the capacity expansion process, the hardware cost is increased; moreover, after the capacity expansion is completed, the service side also needs to change the data source configuration, so that the capacity expansion process is highly coupled with the service, and the implementation is relatively complex.

In order to overcome the problem that a service party needs to perform shutdown operation in the shutdown capacity expansion method and meet the capacity expansion requirement of a service with high availability requirement, the inventor proposes a second method, which is a mirror image smoothing capacity expansion method, as shown in fig. 4, and may be specifically divided into the following five stages:

step S410: a deployment stage, namely deploying a new cache system with stronger processing capacity than that of the existing cache system;

step S420: a double-write stage, namely, a business side carries out double-write operation on the existing cache system and the new cache system, and the read operation of the business side is still carried out through the existing cache system;

step S430: in the historical data migration stage, the data of the existing cache system is migrated to a new cache system through a migration tool;

step S440: in the cutting-reading stage, all the reading operations of the business side are switched to a new cache system;

step S450: and a line-off double-write stage, namely, under the condition of ensuring the data consistency of the two cache systems, stopping the write operation of the business party on the existing cache system, and simultaneously, taking the existing cache system off line, and the business party only carries out read-write operation on the new cache system.

Although the scheme does not need to perform shutdown operation in the capacity expansion process, and can meet the capacity expansion requirement of the service system with high availability requirement, in the operation process, the method is found to have some problems, specifically as follows: in the whole capacity expansion process, the business party needs to perform multiple operations such as double writing, cutting and reading, offline double writing and the like, and the coupling between the capacity expansion process and the business party is increased. Meanwhile, the whole capacity expansion process is relatively long and complex, and the complexity of capacity expansion operation and the probability of capacity expansion failure are increased.

In addition, for the redis distributed system, the inventor further proposes a third slot migration method, as shown in fig. 5, which may specifically be divided into the following steps:

step S510: deploying a new data fragment, and sending deployment information to other fragments in the cluster;

step S520: calculating a slot position responsible for each fragment in the cluster, and determining a slot position to be migrated of the data to be migrated in the new fragment;

step S530: informing the slot position to be migrated of each original fragment and the new fragment to be migrated so as to enable the original fragment to be ready for the data to be migrated;

step S540: informing the slot position to be migrated of the new fragment and the original fragment to be migrated, and making the new fragment ready for importing the data to be migrated;

step S550: sending a migration command to each original fragment, and acquiring a key value set of data to be migrated, wherein the migration command comprises a slot position at the position to be migrated and the number of the data to be migrated;

step S560: according to the obtained key values, sequentially migrating the data to be migrated corresponding to the key values from the original fragments to the new fragments;

step S570: judging whether the data to be migrated in the original fragment is completely migrated, if not, jumping to the step S550, otherwise, continuing to execute the step S580;

step S580: and broadcasting the allocation condition of the slots in the cluster to all the fragments in the cluster.

In practice, this method was found to have the following problems: first, this scheme is only suitable for cache systems that utilize slot assignment for fragmentation. At present, only the redis system adopts the method to perform data fragmentation, and most of distributed cache systems adopt a consistent hash algorithm, so that the universality of the slot migration method is poor; secondly, the distribution of the slot positions needs to be recalculated in the migration process of the method, and if the slot positions are not calculated reasonably, the stored data of different fragments are not uniform; in addition, in the migration process, if the data accessed by the service party is the data which has been migrated to the new fragment, the original fragment needs to be accessed first, the original fragment returns to the new fragment to which the data has been migrated by the service party, and the service party needs to access the new fragment again to acquire the data. That is, for data migrated to a new fragment in the migration process, the service party needs to obtain the data through two accesses, which increases the burden of the service party.

In order to solve the problems in the above method, in the present exemplary embodiment, the inventor further proposes a new technical solution, and the technical solution of the embodiments of the present disclosure is elaborated below:

the exemplary embodiment first provides a data migration method, which is applied to a migration module. The data migration method may be applied to the server 105, or may be applied to one or more of the

terminal devices

101, 102, and 103 and the server 105 at the same time, as shown in fig. 6, and specifically includes the following steps:

step S610: receiving a data migration instruction;

step S620: determining a data migration direction between each existing original cache data fragment and each newly deployed target cache data fragment in the cache data cluster;

step S630: according to the data migration instruction, migrating the data to be migrated in the original cache data fragment to the target cache data fragment according to the data migration direction, and maintaining a migration state in the original cache data fragment;

step S640: and sending the migration result recorded in the migration state to a management module.

In the data migration method provided by the disclosed exemplary embodiment, on one hand, the data migration method provided by the disclosed exemplary embodiment maintains a migration state in an original cache data fragment, and when a migration interruption is caused by factors such as server restart and network link failure in a migration process, an interruption point in the migration process can be obtained according to the migration state, so that when the migration is performed again, all the interruption points are not required to be performed again, only the previous migration process needs to be continued according to the obtained interruption point, and the migration efficiency and fault tolerance are improved. On the other hand, the data to be migrated can be managed conveniently according to the migration state maintained in the original cache data fragment, so that the coupling degree between the data migration process and the service system is reduced, and efficient data access can be performed in the migration process. Meanwhile, the data migration method has good adaptability and can be used in most distributed data cache systems.

Next, in another embodiment, the above steps are explained in more detail.

In step S610, a data migration instruction is received.

In this example embodiment, the data migration method described above may be used in a distributed cache system that employs a consistent hashing algorithm for data fragmentation. The distributed cache system may include a management module, a migration module, a routing agent module, and a cache data cluster. It should be noted that the above description of the distributed cache system architecture is only an exemplary illustration, and the protection scope of the exemplary embodiment is not limited thereto.

In this exemplary embodiment, the migration module receives a data migration instruction from the management module, where the data migration instruction includes information required by a data migration process. For example, the data migration instruction may include information such as a migration step involved in the migration process, a name, an IP address, and a port number of the cache data fragment, and a type of the cache data fragment. In addition, other data migration instructions that conform to the above definition also fall within the scope of the present exemplary embodiment.

In step S620, a data migration direction between each existing original cache data fragment and each newly deployed target cache data fragment in the cache data cluster is determined.

In this exemplary embodiment, all the cached data is stored in the cached data cluster, and the cached data cluster divides the cached data into a plurality of different cached data fragments according to a fragment routing rule, where the fragment routing rule is obtained by a routing agent module in the system according to a consistent hash algorithm.

In this exemplary embodiment, the cache data fragment may be divided into two types, one type is an original cache data fragment, and refers to an existing data fragment storing cache data in the cache data cluster. And the other is target cache data fragmentation, in order to meet the requirements of capacity expansion and the like, part of cache data in the original cache data fragmentation needs to be migrated to newly deployed cache data fragmentation, and the newly deployed cache data fragmentation is the target cache data fragmentation.

In the data migration method provided in this exemplary embodiment, before migrating the cache data, it is necessary to determine a data migration direction of the cache data to be migrated between each original cache data fragment and each target cache data fragment, and the process may be specifically implemented as follows: determining the node position of each target cache data fragment in a consistent hash ring corresponding to the cache data cluster; and determining the data migration direction from the original cache data fragment to the target cache data fragment according to the determined node position.

In the method provided by this exemplary embodiment, the determining a node position of each target cache data fragment in the consistent hash ring corresponding to the cache data cluster may specifically be implemented as follows: and carrying out Hash operation on the IP address and the port number of the target cache data fragment, and mapping each target cache data fragment into a corresponding node on the consistent Hash ring according to the operation result. It should be noted that the above scenario is only an exemplary illustration of the target cache data fragment and the corresponding relationship between nodes, and other methods for mapping the target cache data fragment to the corresponding node on the consistent hash ring also belong to the protection scope of the present exemplary embodiment.

In this exemplary embodiment, after obtaining the node distribution of each target cache data fragment on the consistent hash ring, according to the consistent hash algorithm, and according to the node distribution condition, the data migration direction from the original cache data fragment to the target cache data fragment may also be determined. The process can be specifically realized as follows: and adding the node corresponding to the target cache data fragment in the consistent hash ring, and migrating the cache data which is stored in the node corresponding to the original cache data fragment and is closest to the node corresponding to the newly deployed target cache data fragment in the clockwise direction to the newly deployed target cache data fragment, wherein the data migration direction is from the original cache data fragment corresponding to the node to the target cache data fragment.

The following explains the process of determining the data migration direction by taking 10 data from 1 to 10 as cache data, nodes a, g and z as original cache data fragments, and node n as a newly deployed target cache data fragment as an example: and calculating the hash values of the data and the nodes by using the ASCII code of the character string as a hash key value. It can be obtained that the hash values corresponding to the data 1 to 10 are in turn: 196, 200, 204, 208, 212, 216, 220, 224, 228, 192; the hash values of the nodes a, g and z corresponding to the original cache data fragmentation are as follows in sequence: 203, 209, 228; comparing nodes a, g and z corresponding to the original cache data fragments with the hash values of the data, storing the data into the node closest to the hash value in the clockwise direction, and further obtaining a corresponding consistent hash ring: node a contains 0, 1 and 2; the node g comprises 3 and 4; node z contains 5, 6, 7, 8 and 9. When a target cache data fragment is newly deployed in the distributed cache system, in this example, the hash value of a node n corresponding to the newly deployed target cache data fragment is obtained by calculation as 216, the node n is located between a node g and a node z, and a position corresponding to data 6, according to the principle of consistent hash, the closest node of data 5 and 6 in the clockwise direction is changed from the node z to the node n, and then the two data are migrated to the node n. Therefore, the direction from the original cache data fragment corresponding to the node z to the target cache data fragment corresponding to the node n is the determined data migration direction.

It should be noted that the foregoing scenario is only an exemplary illustration of a process for determining a data migration direction, and does not limit the present exemplary embodiment, for example, a newly deployed target cache data is fragmented into multiple pieces, or other methods for determining a data migration direction also belong to the protection scope of the present exemplary embodiment.

In the data migration method provided by the present example embodiment, a consistent hash algorithm is used to segment the cache data, and the algorithm has a significant advantage over the conventional modulo operation in that when nodes are increased or decreased, the affected data is very little, that is, in the case of capacity expansion and the like that the data needs to be migrated, only a small part of the data needs to be migrated, and thus, the efficiency of data migration and management can be improved.

Taking the 10 pieces of data from 1 to 10 as cache data, the nodes a, g, and z as original cache data fragments, and the node n as a newly deployed target cache data fragment as an example, the 10 pieces of data are distributed to 3 cache nodes in a conventional modulo manner, and the implementation thereof may be as follows: assuming that 3 nodes are sequentially nodes a, b and c, the 10 data are sequentially modulo by 3, and are put into the corresponding nodes according to the modulo result, and the result is as follows: node a contains 0, 3, 6 and 9; node b contains 1, 4 and 7; node c contains 2, 5 and 8. When a target cache data fragment is newly deployed in the distributed cache system, in this example, when the newly deployed target cache data fragment corresponds to the node d, the data distribution is changed to: node a contains 0, 4 and 8; node b contains 1, 5 and 9; node c contains 2 and 6; node d contains 3 and 7. It can be seen that in the process of node addition, 7 data of data 3, 4, 5, 6, 7, 8 and 9 are migrated. In the process of adding new nodes, only the data 5 and the data 6 are migrated by adopting the consistent hash algorithm. Therefore, the consistent hash algorithm is more suitable for data caching.

It should be noted that the above scenario is only an exemplary illustration, and the scope of protection of the exemplary embodiment is not limited thereto.

In step S630, according to the data migration instruction, migrating the data to be migrated in the original cache data segment to the target cache data segment according to the data migration direction, and maintaining a migration state in the original cache data segment.

In this exemplary embodiment, the data to be migrated is cache data that needs to be migrated from the original cache data fragment to the target cache data fragment. Taking the 10 pieces of data from 1 to 10 as cache data, the nodes a, g, and z as original cache data fragments, and the node n as a newly deployed target cache data fragment as an example, according to the principle of consistent hash, the closest node of the data 5 and 6 in the clockwise direction is changed from the node z to the node n, and then the two pieces of data need to be migrated to the node n in the data migration process, in this example, the data 5 and the data 6 are pieces of data to be migrated. It should be noted that the above scenario is only an exemplary illustration, and the scope of protection of the exemplary embodiment is not limited thereto.

In the data migration method provided in this exemplary embodiment, when a migration process starts, a migration state file is initialized in an original cache data fragment to maintain the migration state, where the migration state is information related to the data migration process. For example, the migration status file may include migration status information such as a migration stage, a number of a data node that has been migrated, and a copy offset. In addition, the migration state file is updated in real time in the data migration process. It should be noted that the above scenario is only an exemplary illustration, and the scope of protection of the exemplary embodiment is not limited thereto.

In this exemplary embodiment, according to the data migration instruction, migrating the data to be migrated in the original cache data fragment to the target cache data fragment according to the data migration direction determined in the step S620, specifically, the process may be as follows: firstly, initializing a migration state file to maintain a migration state, and acquiring a first migration step length and a second migration step length from the data migration instruction; then, taking the obtained first migration step as a unit, and carrying out full migration on the data to be migrated in the original cache data fragment; and performing incremental migration on newly added data to be migrated of the original cache data fragments in the full migration process by taking the obtained second migration step as a unit until the copy offset in the migration process maintained in the migration state file is consistent with the copy offset of the hot standby file of the original cache data fragments.

The above full migration refers to migrating all the data to be migrated in the original cache data fragment to the target cache data fragment according to the determined data migration direction, and the specific implementation may be as follows: setting a migration stage in a migration state file as full migration, copying an offset as a write-in position of a current original cache data fragment hot standby file, and setting a data node number corresponding to an original cache data fragment which is migrated to be 0; in a period of a first migration step length, reading cache data from original cache data fragments according to a mode of self-increment of a data node number, extracting a key value corresponding to the cache data, judging whether the cache data is to-be-migrated data or not, and if so, executing migration operation on the to-be-migrated data; and repeating the steps until the data node number of the obtained cache data reaches the maximum data node number in the original cache data fragment, indicating that the full migration is finished.

Since the method provided by this exemplary embodiment does not stop the service in the data migration process, because there may be a new buffer data in the full migration process, and there may also be data to be migrated in the new buffer data, in this method, after the full migration process is finished, it is also necessary to perform incremental migration on the new buffer data in the full migration process. Specifically, the implementation process can be as follows: and setting the migration stage maintained in the migration state file as incremental migration, circularly reading newly added cache data in the full migration process from the original cache data fragments by taking a second migration step as a unit according to the copy offset saved in the migration file, judging whether the cache data is to-be-migrated data, and if so, executing migration operation on the to-be-migrated data until the copy offset maintained in the migration state file is consistent with the copy offset of the original cache data fragment hot standby file.

In the incremental migration process, the copy offset is updated in the migration state file according to the process of data migration. When the copy offset maintained in the migration state file is close to the copy offset of the backup file in the original cache data fragment, sending a read-only command to the original cache data fragment, stopping the write operation of the original cache data fragment until the two copy offsets are consistent, and ending the incremental migration.

In the processes of the full migration and the incremental migration, when the obtained cache data is to-be-migrated data, a migration operation needs to be performed on the to-be-migrated data, which may be specifically implemented as follows: sending a migration notification of the data to be migrated to the original cache data fragment; after the original cache data fragment receives the migration notification, key and value information of the data to be migrated are extracted, the extracted information and other information required by migration are sent to a target cache data fragment, and the migration state of the data to be migrated is set to be in migration; after receiving the information, the target cache data fragment extracts the key and the value of the data to be migrated from the target cache data fragment, establishes cache data by using the extracted information, and sends a message of successful migration to the target cache data fragment; and after receiving the message of successful migration, the original cache data fragment deletes the data to be migrated and sets the data to be migrated to be in a migration completion state.

In the data migration process, the migration state file is initialized in the original cache data fragment, and the migration state in the data migration process is maintained in the migration state file. Therefore, in the data migration process, when the service side requests to obtain the cache data from the original cache data fragment, if the cache data requested to be accessed is the data to be migrated and the migration state is migration completion, the request is sent to the target cache data fragment, and when the migration state is migration or the cache data requested to be accessed is not the data to be migrated, the request of the service side is responded on the target cache data fragment. For the data to be migrated, the problem that the service party needs to initiate two requests in the slot position migration scheme is solved by the method for maintaining the data migration state in the original cache data fragment, and the burden of the service party is reduced.

It should be noted that, in order to explain the scenario of migrating the data to be migrated in the original cache data fragment to the target cache data fragment according to the data migration direction determined in step S620, the above is only an exemplary illustration, and the protection scope of the present exemplary embodiment is not limited thereto.

In step S640, the migration result recorded in the migration state is sent to a management module.

In this exemplary embodiment, after the data migration process is completed, the migration result needs to be sent to the management module, so that the management module sends the information of the target cache data fragment to the routing agent module according to the migration result, the routing agent module updates the consistent hash ring corresponding to the cache data cluster, and adds the target cache data fragment to the fragment routing rule.

In addition, in the data migration method provided in this exemplary embodiment, because the migration state is maintained, when a migration interruption occurs during the migration process due to a server restart, a network link failure, and the like, the migration is continued only at an interruption point maintained according to the migration state file without being resumed.

Correspondingly, the present exemplary embodiment further provides a data migration method applied to the management module, as shown in fig. 7, specifically including the following steps:

step S710: in response to a data migration request, deploying one or more target cache data fragments;

step S720: sending a data migration instruction to a migration module so that the migration module migrates data to be migrated from original cache data fragments of a cache data cluster to the target cache data fragments;

step S730: and sending the information of the target cache data fragmentation to a routing agent module according to the migration result sent by the migration module so that the routing agent module updates a consistent hash ring corresponding to the cache data cluster and adds the target cache data fragmentation to a fragmentation routing rule.

Next, in another embodiment, the above steps are explained in more detail.

In step S710, one or more target cache data slices are deployed in response to the data migration request.

In this example embodiment, when receiving a data migration request, a management module deploys one or more target cache data fragments on a machine having idle resources managed by the management module, so that the migration module migrates data to be migrated in an original cache data fragment in a cache data cluster to the target cache data fragment.

In this exemplary embodiment, preferably, a data migration request key may be further configured in the management module, and when a service needs to migrate cache data due to a requirement such as capacity expansion, a data migration process may be started through a triggering operation of the data migration request key. The whole data migration process is integrated into one key action to realize one-key simple operation, and the complexity and failure probability of capacity expansion operation are reduced.

In step S720, a data migration instruction is sent to the migration module, so that the migration module migrates the data to be migrated from the original cache data fragment of the cache data cluster to the target cache data fragment.

In this example embodiment, after deploying one or more target cache data fragments, a data migration instruction is sent to the migration module, so that the migration module migrates data to be migrated from the original cache data fragments of the cache data cluster to the target cache data fragments. The migration process has already been described in detail in the data migration method applied to the migration module, and therefore is not described herein again.

In step S730, according to the migration result sent by the migration module, the information of the target cache data fragment is sent to the routing agent module, so that the routing agent module updates the consistent hash ring corresponding to the cache data cluster, and adds the target cache data fragment to the fragment routing rule.

In this exemplary embodiment, the routing agent module is composed of a series of fragment routing nodes with the same function and independent from each other, and the service side accesses the distributed cache system through the routing agent module. The fragment routing node can provide routing service in two ways: one as a stand-alone proxy service and one integrated into the business's code as an SDK (Software Development Kit, an acronym for Software Development Kit). Each fragment routing node stores the information of each fragment in the cache data cluster, and routing access to the cache data fragments is realized. It should be noted that the above description of the routing agent module is only an exemplary illustration, and the protection scope of the exemplary embodiment is not limited thereto.

In this exemplary embodiment, after the data migration process is completed, the routing agent module updates the consistent hash ring corresponding to the cache data cluster, and adds the newly deployed target cache data segment to the segment routing rule.

The following further describes the data migration method in the present exemplary embodiment with reference to the specific scenarios in fig. 8 to 10.

Fig. 8 is an architecture diagram of a distributed cache system in this specific application scenario, and includes a service provider 810, a cache system management website 820, a migration service 830, a routing agent cluster 840, and a cache data cluster 850.

The service side initiates a migration request for data and other requests for access, modification, writing and the like in the data migration method.

The cache system management website is responsible for coordinating the whole data migration process, and specifically comprises the following steps: maintaining deployment information of the routing agent cluster and the cache data cluster; completing resource allocation and deployment of the newly deployed target cache data fragments; sending a migration command to a migration service in the migration process, and setting the state of the cache data cluster to be in the migration; after receiving a migration completion notification sent by a migration service, setting the state of the cache data cluster as migration completion, and notifying a routing agent cluster to update the fragmented routing rule; and controlling and coordinating the whole migration process, and integrating the process into a website button so that a business side clicks the button to complete the migration process.

The migration service mainly performs the following functions: receiving a migration command sent by a cache system management website, determining a data migration direction according to the deployment condition of cache data fragments in the migration command, and informing an original cache data fragment to perform data migration; and generating a subprocess for each migration direction, wherein the subprocess acquires all data of the original cache data fragmentation according to a certain step length, and judges the data to be migrated according to the routing fragmentation rule of the cache data cluster. For data to be migrated, the sub-process sends a migration command to an original cache data fragment, and the original cache data fragment receives the migration command and migrates the data to be migrated to a newly deployed target cache data fragment; data migration in each migration direction is completed through a full migration process and an incremental migration process; maintaining a migration state, and supporting the migration to continue according to an interruption point after the interruption of the migration process; and when the migration in all the migration directions is completed, informing the website that the data migration is completed.

The routing agent cluster is composed of a series of fragment routing nodes with the same function and independent mutually, and a service party accesses the cache system through the routing agent cluster. A fragment routing node may use two approaches: one as a separate proxy service and one integrated as an SDK into the code of the business party. Each fragment routing node stores the information of each fragment in the cache data cluster, and routing access to the cache data fragments is realized. The routing agent cluster mainly completes the following functions: routing the request of the service party to different cache data fragments according to the key value, thereby realizing the data fragmentation; and after the data migration is finished, adding the newly deployed target cache data fragments into a fragment routing rule, wherein the fragment routing rule adopts a consistent hash algorithm.

The cache data cluster stores all cache data, and divides the cache data into a plurality of cache data fragments according to the fragment routing rule. Each cache data slice stores different cache data, but provides the same function. When the distributed cache system has capacity expansion requirements, one or more target cache data fragments are newly deployed, and partial data in the original cache data fragments are migrated to the newly deployed target cache data fragments according to the fragment routing rule. The cache data cluster mainly completes the following functions: providing basic functions of a cache system, such as basic operations of searching, updating, inserting, deleting and the like, and functions of cold and hot data management, hot standby management and the like; providing a data migration command for an original cache data fragment, establishing tcp connection with a target cache data fragment in the migration process, and migrating data to be migrated to the target cache data fragment through the tcp connection; maintaining the migration state of each data in the migration process and providing an access mechanism for processing the data in the migration process; and allocating a self-added data node number to each data, wherein the migration service can pull all data of the cache data fragment through the node number.

Taking a scenario of expanding the distributed cache system shown in fig. 8 as an example, the following steps of the data migration process are further described with reference to the flowchart shown in fig. 9:

in step S910, the service party clicks the capacity expansion button on the cache system management website. In the step, the complex capacity expansion operation flow is integrated on one graphical button, so that one-key capacity expansion is realized, and the difficulty and the failure rate of capacity expansion operation are reduced.

In step S920, the cache system management website deploys one or more target cache data fragments from the managed idle resource machines.

In step S930, the cache system management website issues a data migration command to the migration service, where the migration command includes the following information: (1) the data migration step size is generally 10000 by default; (2) the name of each cache data fragment in the cache data cluster after the target cache data fragment is newly deployed is uniformly distributed by a cache system management website; (3) caching an IP and a port of each cache data fragment in the data cluster; (4) and the type of each cache data fragment is the type of the deployed fragment as the original cache data fragment, and the type of the newly deployed fragment is the type of the target cache data fragment.

In step S940, the information such as the name, IP address, port, and type of each cache data fragment is broadcast to the entire cache data cluster.

In step S950, a data migration direction is determined according to the type of each cache data slice in the cache data cluster and the characteristics of the consistent hash algorithm.

In the specific application scenario, the data migration direction is determined by the following method: determining the node position of each cache data fragment in a consistent hash ring corresponding to the cache data cluster; and determining the data migration direction from the original cache data fragment to the target cache data fragment according to the determined node position.

The process of determining the node position of each cache data fragment in the consistent hash ring corresponding to the cache data cluster includes: carrying out Hash operation on the IP address and the port number of each cache data fragment, and mapping each cache data fragment into a corresponding node on the consistent Hash ring according to the operation result; after the node distribution of each cache data fragment on the consistent hash ring is obtained, determining the data migration direction from the original cache data fragment to the target cache data fragment according to the consistent hash algorithm and the node distribution condition, wherein the specific implementation process is as follows: and adding the node corresponding to the target cache data fragment in the consistent hash ring, and migrating the cache data which is stored in the node corresponding to the original cache data fragment and is closest to the node corresponding to the newly deployed target cache data fragment in the clockwise direction to the newly deployed target cache data fragment, wherein the data migration direction is from the original cache data fragment corresponding to the node to the target cache data fragment.

In step S960, data migration in each data migration direction determined in step S950 is initiated.

In step S970, it is determined whether the migration is successful. When the data to be migrated in all the data migration directions are successfully migrated, if the judgment result is yes, jumping to step S980; otherwise, the cache system is notified that the migration of the management website fails, and the process jumps to step S930.

In step S980, the migration service notifies the cache system of the completion of the migration of the management website.

In step S990, the fragment routing rule is updated.

The data migration process in each data migration direction in step S960 may be divided into two flows, namely, full migration and incremental migration, and the data migration process in step S960 is described in more detail with reference to the flow shown in fig. 10 as follows:

in step S1010, migration is started, a migration status file is initialized, information such as a migration stage, a number of a data node that has been migrated, and a copy offset is maintained in the migration status file, and the migration status file is updated in each subsequent migration process.

In step S1020, the migration phase in the migration state file is set to be full migration, the number of the migrated data node is initially 0, and the copy offset is the write-in position of the hot standby file of the original cache data segment in the current migration, where the copy offset is composed of the hot standby number and the file write-in position.

In step S1030, full migration is initiated, which includes three steps S1031 to S1033:

in step S1031, in a period of one migration step, reading data from the original cache data fragment in a manner that the number of the data node is self-incremented, extracting a key value in the data, and determining whether the data needs to be migrated according to a routing fragment rule, if so, executing a migration operation for the data;

in step S1032, it is determined whether the full migration is completed. And if the data node number corresponding to the read data reaches the maximum data node number in the original cache data fragment, indicating that the full migration is finished. If the judgment result is no, jumping to step S1033, if the judgment result is yes, jumping to step S1040, and setting the migration stage in the migration state file as full migration end;

in step S1033, a migration command is transmitted, and the process jumps to step S1031.

In step S1040, the migration phase in the migration state file is set as incremental migration, and incremental migration is started. The incremental migration comprises three steps S1041 to S1043:

in step S1041, according to the copy offset saved in the migration file, reading the newly added data in the full migration process from the original cache segment in a certain migration step size cycle, and determining whether the data needs to be migrated according to the routing segment rule, if the data needs to be migrated, executing a migration operation for the data, and when the copy offset maintained in the migration service is close to the copy offset of the spare file in the original cache data segment, sending a read-only command to the old cache data segment;

in step S1042, it is determined whether the incremental migration is completed. And if the copy offset maintained in the migration service is consistent with the copy offset of the hot standby file in the original cache data fragment, indicating that the incremental migration is finished. If the judgment result is negative, jumping to step S1043, if the judgment result is positive, jumping to step S1050, and setting the migration stage in the migration state file as the full migration end;

in step S1043, a migration command is sent, and the process jumps to step S1041.

In this specific application scenario, the execution of the migration operation on the data mentioned in the full migration in step S1030 and the incremental migration in step S1040 is specifically implemented as follows: the migration service sends a migration command about the data to the original cache data fragment; after the original cache data fragment receives the migration command, extracting key and value information of the data, assembling a migration command, sending the migration command to the target cache data fragment, and setting the data state in migration; after the target cache data fragment receives the migration command, extracting keys and values in the command to establish cache data, and sending a message of successful data migration to the original cache data fragment; and after the original cache data fragment receives the message of successful data migration, deleting the data and setting the data to be in a migration completion state.

In the data migration process, when a related operation request of a service party for data is sent to an original cache data fragment, if the requested data is in a migration completion state, the request is sent to a target cache data fragment; otherwise, the requirement of the service party is responded on the original cache data fragment. For the data in migration, the problem that a business party needs to initiate two requests is avoided by the method for maintaining the data migration state in the cache data fragment, and the burden of the business party is reduced.

In step S1050, the migration status in the migration status file is set to migration complete.

In the specific application scenario, in the data migration process, the migration service maintains a migration state file for each data migration direction, and updates information such as a migration stage, a migrated node number, a copy offset and the like recorded in the migration state file according to the migration progress. Therefore, when the migration is interrupted due to factors such as server restart and network link failure in the migration process, the migration is not required to be performed again, and the migration is continued only by using the interruption point maintained according to the migration state file, so that the migration efficiency and the fault tolerance are improved.

In this specific application scenario, on one hand, a migration state is maintained in an original cache data fragment, and when a migration interruption occurs due to factors such as server restart and network link failure in the migration process, an interruption point in the migration process can be obtained according to the migration state, so that when the migration is performed again, all the operations need not be performed again, and only the previous migration process needs to be continued according to the obtained interruption point, thereby improving the migration efficiency and fault tolerance. On the other hand, the data to be migrated can be managed conveniently according to the migration state maintained in the original cache data fragment, so that the coupling degree between the data migration process and the service system is reduced, and efficient data access can be performed in the migration process. Meanwhile, the complex capacity expansion operation flow is integrated on one graphical button, so that one-key capacity expansion is realized, and the difficulty and the failure rate of capacity expansion operation are reduced.

It should be noted that although the various steps of the methods of the present disclosure are depicted in the drawings in a particular order, this does not require or imply that these steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.

Further, the present exemplary embodiment provides a data migration apparatus, which is applied to a migration module. Referring to fig. 11, the data migration apparatus 1100 may include a communication module 1110, a direction determination module 1120, a migration module 1130, and a result return module 1140. Wherein:

the communication module 1110 may be configured to receive data migration instructions;

the direction determining module 1120 may be configured to determine a data migration direction between each existing original cache data fragment and each newly deployed target cache data fragment in the cache data cluster;

the migration module 1130 may be configured to migrate, according to the data migration instruction, data to be migrated in the original cache data segment to the target cache data segment according to the data migration direction, and maintain a migration state in the original cache data segment;

the result returning module 1140 may be configured to send the migration result recorded in the migration status to the management module.

The specific details of each module or unit in the data migration apparatus have been described in detail in the corresponding data migration method, and therefore are not described herein again.

Correspondingly, in the present exemplary embodiment, a data migration apparatus is further provided, which is applied to the management module. Referring to fig. 12, the data migration apparatus 1200 may include a response module 1210, a migration module 1220, and an update module 1230. Wherein:

the response module 1210 may be configured to deploy one or more target cache data fragments in response to a data migration request;

the migration module 1220 may be configured to send a data migration instruction to the migration module, so that the migration module migrates data to be migrated from original cache data fragments of the cache data cluster to the target cache data fragments;

the updating module 1230 may be configured to send the information of the target cache data fragment to the routing agent module according to the migration result sent by the migration module, so that the routing agent module updates the consistent hash ring corresponding to the cache data cluster, and adds the target cache data fragment to the fragment routing rule.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method as described in the above embodiments. For example, the electronic device may implement the various steps shown in fig. 3-10, and so on.

It should be noted that the computer readable media shown in the present disclosure may be computer readable signal media or computer readable storage media or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of data migration, comprising:

receiving a data migration instruction;

2. The data migration method according to claim 1, wherein the determining a data migration direction between existing original cache data fragments and newly deployed target cache data fragments in the cache data cluster comprises:

and determining the data migration direction according to the node position.

3. The data migration method according to claim 1, wherein the data migration instruction includes a first migration step size and a second migration step size;

4. The data migration method according to claim 3, wherein the performing full migration on the data to be migrated in the original cache data segment by using the first migration step as a unit includes:

5. The data migration method according to claim 3, wherein the performing incremental migration on the newly added data to be migrated of the original cache data slice in the full migration process with the second migration step as a unit includes:

6. The data migration method according to claim 3, wherein the data to be migrated in the original cache data fragment is migrated in full;

7. The data migration method according to claim 3, wherein the incrementally migrating the newly added data to be migrated in the full migration process of the original cache data segment until the second copy offset is consistent with the first copy offset includes:

8. A method of data migration, comprising:

9. The data migration method according to claim 8, wherein a data migration request key is deployed in the data migration system;

10. A data migration apparatus, comprising:

the communication module is used for receiving a data migration instruction;

11. A data migration apparatus, comprising:

12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1-9.

13. An electronic device, comprising:

a processor;

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the method of any of claims 1-9 via execution of the executable instructions.