CN113312339B

CN113312339B - Data migration method and device, computer equipment and storage medium

Info

Publication number: CN113312339B
Application number: CN202110853923.1A
Authority: CN
Inventors: 谢亚晋
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-07-28
Filing date: 2021-07-28
Publication date: 2021-10-29
Anticipated expiration: 2041-07-28
Also published as: CN113312339A

Abstract

The application discloses a data migration method and device, computer equipment and a storage medium, which are applied to the technical field of Internet. The method comprises the following steps: a first virtual node cluster receives first data information reported by a first service node; if the first data information meets the data migration condition, the first virtual node cluster determines a target virtual node cluster according to the target virtual node cluster identifier; the first virtual node cluster determines a second service node from the target virtual node cluster; the first virtual node cluster sends a first data migration instruction to the first service node, wherein the first data migration instruction instructs the first service node to send the first data to the second service node. By the method, when data are inclined or the load of the server is overlarge, the load of all the servers does not need to be acquired and addressed, the servers capable of bearing the data can be determined through the cluster, and therefore the data migration efficiency is improved.

Description

Data migration method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of internet technologies, and in particular, to a method and an apparatus for data migration, a computer device, and a storage medium.

Background

With the development of internet services, more and more applications need to provide services through the internet, and with the continuous increase of services, the dependency between service types and services becomes complicated. Therefore, each service node in the distributed cache system may fail, and it is necessary to ensure that good services can still be provided to the outside when the number of service nodes in the system changes, so the good distributed cache system should satisfy the balance, so that all servers are utilized, and the occurrence of excessive load of individual servers is prevented.

At present, when cache data amounts stored on a plurality of servers are inconsistent, and data skew is caused due to the large data amount of a part of servers, a server capable of bearing data needs to be determined from other servers, and at this time, the processes of calculating loads of other servers and addressing are long, so that the efficiency of data migration is affected, and therefore, how to improve the efficiency of data migration becomes an urgent problem to be solved.

Disclosure of Invention

The embodiment of the application provides a data migration method and device, computer equipment and a storage medium, a cluster comprising a plurality of servers is processed by grouping the plurality of servers, so that when data tilt or server load is overlarge, the load of all the servers does not need to be acquired and addressed, the servers capable of bearing data can be determined through the cluster, and the data migration efficiency is improved.

In view of the above, a first aspect of the present application provides a method for data migration, including:

a first virtual node cluster receives first data information reported by a first service node, wherein the first data information carries a target virtual node cluster identifier and a first service node identifier, the target virtual node cluster identifier indicates a target virtual node cluster, the first service node identifier indicates the first service node, the first virtual node cluster is composed of a plurality of service nodes, and the target virtual node cluster is composed of a plurality of virtual node clusters;

if the first data information meets the data migration condition, the first virtual node cluster determines a target virtual node cluster according to the target virtual node cluster identifier;

the first virtual node cluster determines a second service node from the target virtual node cluster, wherein the second service node meets the data bearing condition;

the method comprises the steps that a first virtual node cluster sends a first data migration instruction to a first service node, wherein the first data migration instruction carries a target virtual node cluster identifier and a second service node identifier, the second service node identifier indicates a second service node, and the first data migration instruction indicates that the first service node sends first data to the second service node.

A second aspect of the present application provides a data migration apparatus, including:

the receiving module is used for receiving first data information reported by a first service node, wherein the first data information carries a target virtual node cluster identifier and a first service node identifier, the target virtual node cluster identifier indicates a target virtual node cluster, the first service node identifier indicates the first service node, the first virtual node cluster is composed of a plurality of service nodes, and the target virtual node cluster is composed of a plurality of virtual node clusters;

the determining module is used for determining a target virtual node cluster according to the target virtual node cluster identifier if the first data information meets the data migration condition;

the determining module is further configured to determine a second service node from the target virtual node cluster, where the second service node satisfies a data carrying condition;

the sending module is configured to send a first data migration instruction to the first service node, where the first data migration instruction carries a target virtual node cluster identifier and a second service node identifier, the second service node identifier indicates a second service node, and the second data migration instruction indicates the first service node to send the first data to the second service node.

In one possible implementation, the target virtual node cluster includes a first virtual node cluster, and the first data carries a first data identifier;

the target virtual node cluster identifier carried by the first data is determined based on the first data identifier;

a first virtual node cluster identifier carried by the first data is determined based on the target virtual node cluster and the first data identifier;

the first serving node identification carried by the first data is determined based on the first virtual node cluster and the first data identification.

In a possible implementation manner, the target virtual node cluster identifier carried by the first data is obtained by taking a remainder of a first hash value of the first data identifier, and the first hash value of the first data identifier is obtained by performing hash calculation on the first data identifier based on a hash function.

In a possible implementation manner, the first virtual node cluster identifier carried by the first data is obtained by taking a remainder of a second hash value of the first data identifier, and the second hash value of the first data identifier is obtained by performing hash calculation on a first hash value of the first data identifier based on a hash function.

In a possible implementation manner, the first service node identifier carried by the first data is obtained by taking a remainder of a third hash value of the first data identifier, and the third hash value of the first data identifier is obtained by performing hash calculation on a second hash value of the first data identifier based on a hash function.

In one possible embodiment, the first virtual node cluster includes a first service node and a second service node;

the first data information also carries a first virtual node cluster identifier;

a determining module, configured to determine, according to the first virtual node cluster identifier, a first virtual node cluster from the target virtual node cluster;

a second service node is determined from the first cluster of virtual nodes.

In one possible embodiment, the target virtual node cluster further comprises a second virtual node cluster;

the first virtual node cluster comprises a first service node;

the second virtual node cluster comprises a second service node;

the determining module is specifically used for reporting first data information to the target virtual node cluster;

receiving a second data migration instruction sent by the target virtual node cluster, wherein the second data migration instruction carries a target virtual node cluster identifier, a second service node identifier and a second virtual node cluster identifier, and the second virtual node cluster identifier indicates the second virtual node cluster;

and determining the second service node according to the second service node identifier.

the first virtual node cluster comprises a first service node and a third service node;

the second virtual node cluster comprises a second service node;

the receiving module is further configured to receive second data information reported by a third service node, where the second data information carries a target virtual node cluster identifier and a third service node identifier, and the third service node identifier indicates the third service node;

the determining module is further configured to determine that the second data information meets the data migration condition, and determine the target virtual node cluster according to the target virtual node cluster identifier;

the determining module is specifically used for reporting the first data information and the second data information to the target virtual node cluster;

determining a second service node according to the second service node identifier;

the first data migration instruction further carries a second virtual node cluster identifier, and the first data migration instruction further instructs the third service node to migrate the loaded second data to the second service node.

In one possible embodiment, the data migration condition is that the first data information is greater than a first load data threshold;

the data bearing condition is that the data which can be borne by the second service node is greater than a second load data threshold;

and after the first data is migrated to the second service node, replacing the first service node identifier carried by the first data with the second service node identifier.

A third aspect of the present application provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to perform the method of the above-described aspects.

According to the technical scheme, the embodiment of the application has the following advantages:

in an embodiment of the present application, a method for data migration is provided, where a first virtual node cluster receives first data information reported by a first service node, where the first data information carries a target virtual node cluster identifier and a first service node identifier, the target virtual node cluster identifier indicates a target virtual node cluster, the first service node identifier indicates a first service node, the first virtual node cluster is composed of a plurality of service nodes, and the target virtual node cluster is composed of a plurality of virtual node clusters. And if the first data information meets the data migration condition, the first virtual node cluster determines a target virtual node cluster according to the target virtual node cluster identifier, and determines a second service node from the target virtual node cluster, wherein the second service node meets the data bearing condition. And then the first virtual node cluster sends a first data migration instruction to the first service node, wherein the first data migration instruction carries a target virtual node cluster identifier and a second service node identifier, the second service node identifier indicates the second service node, and the first data migration instruction indicates the first service node to send the first data to the second service node. By adopting the method, the cluster comprising the plurality of servers is processed in a grouping way on the plurality of servers, so that when data tilt occurs or the load of the servers is overlarge, the load of all the servers does not need to be acquired and addressed, the servers capable of bearing the data can be determined through the cluster, and the efficiency of data migration is improved.

Drawings

Fig. 1 is a schematic frame diagram of a virtual hierarchical frame according to an embodiment of the present disclosure;

FIG. 2 is a schematic architectural diagram of a data migration system based on a virtual hierarchical framework according to an embodiment of the present application;

fig. 3 is a schematic diagram of an embodiment of a method for data migration according to an embodiment of the present application;

FIG. 4 is a diagram illustrating another embodiment of a method for data migration in an embodiment of the present application;

FIG. 5 is a diagram of an embodiment of a first service node and a second service node in the embodiment of the present application;

fig. 6 is a schematic flowchart of a data migration method according to an embodiment of the present application;

fig. 7 is another schematic flow chart of a method for data migration according to an embodiment of the present application;

fig. 8 is a schematic flowchart of a data migration method according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present application;

fig. 10 is a schematic diagram of an embodiment of a server in the embodiment of the present application.

Detailed Description

The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present application and in the drawings described above, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "corresponding" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

With the development of internet services, more and more applications need to provide services through the internet, and with the continuous increase of services, the dependency between service types and services becomes complicated. Therefore, each node of the distributed cache system may fail, and it is necessary to ensure that good services can still be provided to the outside when the number of service nodes of the system changes, so that the good distributed cache system should satisfy the balance, so that all servers are utilized, and the occurrence of excessive load of individual servers is prevented. At present, when cache data amounts stored on a plurality of servers are inconsistent, and data skew is caused due to the large data amount of a part of servers, a server capable of bearing data needs to be determined from other servers, and at this time, the processes of calculating loads of other servers and addressing are long, so that the efficiency of data migration is affected, and therefore, how to improve the efficiency of data migration becomes an urgent problem to be solved. Based on this, the embodiments of the present application provide a data migration method, when data is skewed or a server load is too large, the load of all servers does not need to be acquired and addressed, and a server capable of bearing data can be determined through a cluster, so that the efficiency of data migration is improved.

For ease of understanding, some terms or concepts related to the embodiments of the present application are explained first.

One, virtual layered framework

All service nodes are divided into a plurality of virtual node clusters, and each virtual node cluster comprises a plurality of service nodes. And then, continuously dividing the plurality of virtual node clusters into a plurality of virtual node clusters, wherein the next layer of virtual node cluster comprises a plurality of virtual node clusters, and so on, thereby forming a layered virtual framework. When data are read and written by the distributed cloud storage system, addressing is carried out through the virtual layered framework, and the distributed cloud storage system is positioned to the service node belonging to the data.

Second, service node

The service node is a storage device in the distributed cloud storage system, and the service node is specifically a server in the scheme.

Three, virtual node cluster

Based on the introduction of the virtual hierarchical framework, the virtual node cluster may include a plurality of service nodes, or may be a virtual node cluster at a next layer including a plurality of virtual node clusters.

Four, consistent Hash algorithm

The consistent hash algorithm is widely applied in a distributed system. Consistent hashing is a hashing algorithm, and when a server is removed or added, the mapping relation between an existing service request and a server processing the request can be changed as little as possible, so that the requirement of monotonicity is met as much as possible. Specifically, as the number of system nodes becomes smaller, the client needs to recalculate its hash value when requesting an object, and since the hash value has changed, it is likely that the server that holds the object cannot be found. Therefore, a good distributed cache system should satisfy the balance, and the hash result can be distributed to all servers as much as possible, so that all servers can be utilized, and the phenomenon that the load of individual servers is too large is prevented. Therefore, the whole hash value space can be organized into a virtual ring by adopting a consistent hash algorithm, hash calculation is carried out on each server by using a hash function, a network protocol (IP) address or a host name of the server can be selected as a keyword to carry out hash calculation to obtain a hash value, and each machine can determine the position of the machine on the hash ring according to the hash value.

In the embodiments of the present application, a new virtual hierarchical framework is newly introduced, in which all service nodes are divided into a plurality of virtual node clusters, and each virtual node cluster includes a plurality of service nodes. And then, continuously dividing the plurality of virtual node clusters into a plurality of virtual node clusters, wherein the next layer of virtual node cluster comprises a plurality of virtual node clusters, and so on, thereby forming a layered virtual framework. For better understanding of the present solution, the following describes a virtual hierarchical framework in this embodiment, because each service node has limited memory and Central Processing Unit (CPU) pressure, and in order to prevent a single point failure of the service node, a plurality of service nodes need to be deployed, and a total of 256 service nodes for data caching are taken as an example for explanation, please refer to fig. 1, fig. 1 is a schematic diagram of a virtual hierarchical framework provided in this embodiment of the present application, as shown in fig. 1, first, 256 service nodes are divided into 64 virtual node clusters, each virtual node cluster includes 4 service nodes, at this time, the 64 virtual node clusters include a virtual node cluster a21, a virtual node cluster a22, a virtual node cluster a23, a virtual node cluster a24, and the like, and the virtual node cluster a21 includes a service node a11, service node A12, service node A13, and service node A14, and virtual node cluster A24 includes service node A15, service node A16, service node A17, and service node A18. It should be understood that FIG. 1 does not depict all 64 virtual node clusters and the service nodes that each virtual node cluster includes, but is analogous to the example of virtual node cluster A21 in FIG. 1.

Based on this, based on that each of the 4 virtual node clusters is a group, the 64 virtual node clusters are continuously divided into one group, so that 16 next-layer virtual node clusters can be obtained, at this time, the 16 next-layer virtual node clusters include virtual node cluster a31, virtual node cluster a32, virtual node cluster a33, virtual node cluster a34, and the like, and the virtual node cluster a31 includes virtual node cluster a21, virtual node cluster a22, virtual node cluster a23, and virtual node cluster a 24. It should be understood that FIG. 1 does not depict all 16 next-level virtual node clusters and the virtual node clusters that each next-level virtual node cluster includes, but is analogous to the example of virtual node cluster A31 in FIG. 1.

Further, 16 next-layer virtual node clusters are further divided according to actual conditions, and so on, for example, based on that every 4 next-layer virtual node clusters are a group, 16 next-layer virtual node clusters are further divided into a group, so that 4 next-layer virtual node clusters can be obtained, the 4 next-layer virtual node clusters include virtual node cluster a41, virtual node cluster a42, virtual node cluster a43 and virtual node cluster a44, and virtual node cluster a41 includes virtual node cluster a31, virtual node cluster a32, virtual node cluster a33 and virtual node cluster a 34. It should be understood that FIG. 1 does not depict all of the virtual node clusters that each virtual node cluster of the next lower level comprises, but is analogous to the example of virtual node cluster A41 in FIG. 1. The virtual hierarchical framework described in this embodiment can be obtained through the steps illustrated in the foregoing example, and through the foregoing virtual hierarchical framework, when each data needs to be stored, a service node capable of storing the data can be determined according to an addressing rule described later.

Based on this, the system in the embodiment of the present application is described below. Referring to fig. 2, fig. 2 is a schematic diagram of an architecture of a data migration system based on a virtual hierarchical framework according to an embodiment of the present application, and as shown in fig. 2, the data migration system includes a first service node and a second service node, and the first service node and the second service node are connected through a network. Specifically, a first service node and a second service node report load data of the service node to a virtual node cluster to which the first service node and the second service node belong based on a preset period, the virtual node cluster to which the first service node and the second service node belong judges whether the service node exceeds a load capable of being carried through the load data, if the load data of the first service node determines that the load of the first service node is too high, the virtual node cluster to which the first service node belongs is determined, a second service node with a lower load or capable of carrying data (namely, a data carrying condition is met) is determined from the virtual node cluster, and then the virtual node cluster issues a data migration instruction to the first service node to indicate that part of data in the first service node is migrated to the second service node through a network, so that data migration is completed.

Specifically, the first service node is a service node bearing first-type service data, and then in the virtual node cluster to which the first service node belongs, if a plurality of service nodes meeting the data bearing condition are determined, the service nodes which are both bearing the first-type service data can be screened from the plurality of service nodes meeting the data bearing condition, and then the second service node is determined from the service nodes which are both bearing the first-type service data, so as to ensure the reliability of migration of the same-type service data. Secondly, when a plurality of service nodes all exceed the load capable of bearing in the first service node, the priority order of the bearing service among different service nodes can be considered, and the data borne by the service node with higher priority is migrated first, so as to ensure the service quality of the service with higher priority. It should be understood that how to determine the second service node in practical application may also be flexibly determined according to practical situations on the basis of meeting the data carrying condition, and all the situations are not exhaustive here.

It should be noted that the service node described in fig. 1 and fig. 2 may be a server or a server cluster composed of multiple servers, or a cloud computing center, and the like, which is not limited herein. The service node and the service node can communicate with each other through a wireless network, a wired network or a removable storage medium. Wherein the wireless network described above uses standard communication techniques and/or protocols. The wireless Network is typically the internet, but can be any Network including, but not limited to, bluetooth, Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, private, or any combination of virtual private networks. In some embodiments, custom or dedicated data communication techniques may be used in place of or in addition to the data communication techniques described above. The removable storage medium may be a Universal Serial Bus (USB) flash drive, a removable hard drive or other removable storage medium, and the like.

Although only two service nodes are shown in fig. 2, it should be understood that the example in fig. 2 is only for understanding the present solution, and the number of specific service nodes should be flexibly determined in combination with the actual situation.

Secondly, Cloud technology (Cloud technology) refers to a hosting technology for unifying series resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. The cloud technology is based on the general names of network technology, information technology, integration technology, management platform technology, application technology and the like applied in the cloud computing business model, can form a resource pool, is used as required, and is flexible and convenient. Cloud computing technology will become an important support. Background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture-like websites and more web portals. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.

A distributed cloud storage system (hereinafter, referred to as a storage system) refers to a storage system that integrates a large number of storage devices (storage devices are also referred to as storage nodes) of different types in a network through application software or application interfaces to cooperatively work by using functions such as cluster application, grid technology, and distributed storage file system, and provides a data storage function and a service access function to the outside.

At present, a storage method of a storage system is as follows: logical volumes are created, and when created, each logical volume is allocated physical storage space, which may be the disk composition of a certain storage device or of several storage devices. The client stores data on a certain logical volume, that is, the data is stored on a file system, the file system divides the data into a plurality of parts, each part is an object, the object not only contains the data but also contains additional information such as data identification (ID, ID entry), the file system writes each object into a physical storage space of the logical volume, and the file system records storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

The process of allocating physical storage space for the logical volume by the storage system specifically includes: physical storage space is divided in advance into stripes according to a group of capacity measures of objects stored in a logical volume (the measures often have a large margin with respect to the capacity of the actual objects to be stored) and Redundant Array of Independent Disks (RAID), and one logical volume can be understood as one stripe, thereby allocating physical storage space to the logical volume.

Further, the blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, and an encryption algorithm. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer.

The block chain underlying platform can comprise processing modules such as user management, basic service, intelligent contract and operation monitoring. The user management module is responsible for identity information management of all blockchain participants, and comprises public and private key generation maintenance (account management), key management, user real identity and blockchain address corresponding relation maintenance (authority management) and the like, and under the authorization condition, the user management module supervises and audits the transaction condition of certain real identities and provides rule configuration (wind control audit) of risk control; the basic service module is deployed on all block chain node equipment and used for verifying the validity of the service request, recording the service request to storage after consensus on the valid request is completed, for a new service request, the basic service firstly performs interface adaptation analysis and authentication processing (interface adaptation), then encrypts service information (consensus management) through a consensus algorithm, transmits the service information to a shared account (network communication) completely and consistently after encryption, and performs recording and storage; the intelligent contract module is responsible for registering and issuing contracts, triggering the contracts and executing the contracts, developers can define contract logics through a certain programming language, issue the contract logics to a block chain (contract registration), call keys or other event triggering and executing according to the logics of contract clauses, complete the contract logics and simultaneously provide the function of upgrading and canceling the contracts; the operation monitoring module is mainly responsible for deployment, configuration modification, contract setting, cloud adaptation in the product release process and visual output of real-time states in product operation, such as: alarm, monitoring network conditions, monitoring node equipment health status, and the like.

The platform product service layer provides basic capability and an implementation framework of typical application, and developers can complete block chain implementation of business logic based on the basic capability and the characteristics of the superposed business. The application service layer provides the application service based on the block chain scheme for the business participants to use.

With reference to the above description, based on the virtual hierarchical framework shown in fig. 1, a virtual node cluster a31 is taken as a target virtual node cluster, a virtual node cluster a21 is taken as a first virtual node cluster, and a service node a11 is taken as a first service node to describe the data migration method in the present application as an example, please refer to fig. 3, where fig. 3 is a schematic diagram of an embodiment of the data migration method in the embodiment of the present application, as shown in fig. 3, an embodiment of the data migration method in the embodiment of the present application includes:

101. the method comprises the steps that a first virtual node cluster receives first data information reported by a first service node, wherein the first data information carries a target virtual node cluster identifier and a first service node identifier, the target virtual node cluster identifier indicates a target virtual node cluster, the first service node identifier indicates the first service node, the first virtual node cluster is composed of a plurality of service nodes, and the target virtual node cluster is composed of a plurality of virtual node clusters.

In this embodiment, as can be seen from the example of fig. 1, the target virtual node cluster is composed of a plurality of virtual node clusters, and the first virtual node cluster is composed of a plurality of service nodes, and the first virtual node cluster includes the first service node. Based on this, the first virtual node cluster receives first data information reported by the first service node, where the first data information carries a target virtual node cluster identifier and a first service node identifier, and the target virtual node cluster identifier indicates the target virtual node cluster, that is, it is described that the first service node stored in the first data information also belongs to the target virtual node cluster. Each service node has a service node identifier in one-to-one correspondence, so that the first data information stored in the first service node carries the first service node identifier, and the first data information is determined to be from the first service node by indicating the characteristic of the first service node through the first service node identifier.

Exemplarily, referring to fig. 4, the first service node is identified as "11", and fig. 4 is a schematic diagram of another embodiment of a method for data migration in an embodiment of the present application, as shown in fig. 4, B31 indicates a target virtual node cluster, B21 indicates a first virtual node cluster, and B11 indicates a first service node. Based on this, the first serving node B11 reports the first data information to the first virtual node cluster B21, and the first virtual node cluster B21 determines that the first data information is reported by the first serving node B11 through the first serving node identifier "11" carried by the first data information. It should be understood that the example of fig. 4 is only used for understanding the purpose of the first data information carrying the aforementioned identifier and the function of the aforementioned identifier, and should not be construed as a limitation of the present solution.

Further, the first virtual node cluster determines whether the first data information satisfies the data migration condition, and if so, it indicates that the first data information carried by the first service node exceeds the load capacity of the first service node, that is, the load of the first service node is too high, which causes data skew, and the data carried by the first service node needs to be migrated, so step 102 needs to be performed here. If not, it indicates that the first data information carried by the first service node is within the load capacity of the first service node, and at this time, the subsequent steps are not executed.

102. And if the first data information meets the data migration condition, the first virtual node cluster determines a target virtual node cluster according to the target virtual node cluster identifier.

In this embodiment, in step 101, the first virtual node cluster further determines whether first data information reported by the first service node meets the data migration condition, and the first data information carries the target virtual node cluster, so that the first virtual node cluster determines the target virtual node cluster according to the target virtual node cluster identifier.

Illustratively, referring again to fig. 4, with the target virtual node cluster identified as "31" and the first service node identified as "11", B31 indicates the target virtual node cluster, B21 indicates the first virtual node cluster, and B11 indicates the first service node. Based on this, the first virtual node cluster B21 determines that the first data information is reported by the first serving node B11 through the first serving node identifier "11" carried by the first data information. Further, the first virtual node cluster B21 can also determine that the first serving node also belongs to the target virtual node cluster B31 and also determine that itself (the first virtual node cluster B21) belongs to the target virtual node cluster B31 through the target virtual node cluster identifier "31" carried by the first data information.

103. The first virtual node cluster determines a second service node from the target virtual node cluster, wherein the second service node meets the data bearing condition.

In this embodiment, the first virtual node cluster determines, from the target virtual node cluster, a second service node that satisfies the data bearer condition. The data carrying condition is satisfied, that is, the load capacity of the second service node in the specification can also carry certain data, so that part of the data on the first service node can be carried.

Further, the second service node may belong to the first virtual node cluster together with the first service node, or belong to another virtual node cluster in the target virtual node cluster. For example, referring to fig. 5, fig. 5 is a schematic diagram of an embodiment of a first service node and a second service node in the embodiment of the present application, as shown in (a) of fig. 5, C31 indicates a target virtual node cluster, C21 indicates a first virtual node cluster, C11 indicates a first service node, and C14 indicates a second service node, in which case, the first service node C11 and the second service node C14 both belong to the first virtual node cluster C21, and the first virtual node cluster C21 belongs to the target virtual node cluster C31, so that the first service node C11 and the second service node C14 also belong to the target virtual node cluster C31. Second, as shown in the (B) diagram in fig. 5, C31 indicates a target virtual node cluster, C21 indicates a first virtual node cluster, C24 indicates a second virtual node cluster, C11 indicates a first service node, and C15 indicates a second service node, when the first service node C11 belongs to the first virtual node cluster C21 and the second service node C15 belongs to the second virtual node cluster C24, but both the first virtual node cluster C21 and the second virtual node cluster C24 belong to the target virtual node cluster C31, so that the first service node C11 and the second service node C15 also belong to the target virtual node cluster C31. It should be understood that the example of fig. 5 is only used for understanding the first serving node and the second serving node, and should not be construed as a limitation of the present solution.

104. The method comprises the steps that a first virtual node cluster sends a first data migration instruction to a first service node, wherein the first data migration instruction carries a target virtual node cluster identifier and a second service node identifier, the second service node identifier indicates a second service node, and the first data migration instruction indicates that the first service node sends first data to the second service node.

In this embodiment, the first virtual node cluster sends the first data migration instruction to the first service node, because the first data migration instruction carries the target virtual node cluster identifier and the second service node identifier, and the second service node identifier indicates the second service node, the first service node can determine the second service node based on the second service node identifier carried by the first data migration instruction, and send the first data to the second service node through the network, where the first data is part of the first data information in step 101, and specifically how much the first data is, and needs to be flexibly determined according to actual load conditions of the first service node and the second service node, and this is not limited.

In the embodiment of the application, a data migration method is provided, and in the above manner, a cluster including a plurality of servers is processed by grouping the plurality of servers, so that when data is skewed or the load of the servers is too large, the load of all the servers does not need to be acquired and addressed, and the servers capable of bearing data can be determined through the cluster, so that the efficiency of data migration is improved.

Optionally, on the basis of the embodiment corresponding to fig. 3, in an optional embodiment of the data migration method provided in the embodiment of the present application, the target virtual node cluster includes a first virtual node cluster, and the first data carries the first data identifier;

In this embodiment, the target virtual node cluster includes a first virtual node cluster, and the first data carries the first data identifier. Specifically, the first data identifier is a unique Identity (ID) of the first data, and each data carries a unique data identifier.

Based on this, the target virtual node cluster identifier carried by the first data is determined based on the first data identifier, specifically, a first calculation result of the first data identifier is obtained by calculating the first data identifier, the target virtual node cluster is determined from a plurality of virtual node clusters on the same layer as the target virtual node cluster based on the first calculation result of the first data identifier, and the first calculation result of the first data identifier is determined as the target virtual node cluster identifier. Further, a first virtual node cluster identifier carried by the first data is determined based on the target virtual node cluster and the first data identifier, specifically, a second calculation result of the first data identifier is obtained by calculating a first calculation result of the first data identifier, the first virtual node cluster is determined from a plurality of virtual node clusters included in the target virtual node cluster based on the second calculation result of the first data identifier, and the second calculation result of the first data identifier is determined as the first virtual node cluster identifier. Still further, the first service node identifier carried by the first data is determined based on the first virtual node cluster and the first data identifier, specifically, a third calculation result of the first data identifier is obtained by calculating a second calculation result of the first data identifier, the first service node is determined from the service nodes included in the first virtual node cluster based on the third calculation result of the first data identifier, the third calculation result of the first data identifier is determined as the first service node identifier, and at this time, the first data is stored in the first service node.

Based on the virtual hierarchical frame shown in fig. 1, when the first data is stored, the first data is addressed through the virtual hierarchical frame based on the first data identifier, so that the first service node where the first data can be stored is accurately located, and the reliability of the storage of the first data is ensured. And the first data carries the virtual node cluster identifier corresponding to each layer of virtual node cluster, and when the first data is subjected to read-write operation and data migration subsequently, the first data cannot be positioned to other virtual node cluster identifiers, so that the reliability of data migration and data read-write is improved.

Specifically, based on the virtual hierarchical framework shown in fig. 1, each layer of virtual framework has a preset Hash function, please refer to fig. 1 again, where virtual node cluster a41 to virtual node cluster a44 correspond to preset Hash function 1 (Hash 1), and the virtual node cluster id of virtual node cluster a41 is 10, which means that virtual node cluster a41 is the first virtual node cluster in the first layer of virtual framework, the virtual node cluster id of virtual node cluster a42 is 11, which means that virtual node cluster a42 is the second virtual node cluster in the first layer of virtual framework, and so on, the virtual node cluster id of virtual node cluster a43 is 12, and the virtual node cluster id of virtual node cluster a44 is 14. Similarly, virtual node cluster a31 through virtual node cluster a34 correspond to the preset Hash function 2 (Hash 2), and the virtual node cluster id of virtual node cluster a31 is 20, the virtual node cluster id of virtual node cluster a32 is 21, the virtual node cluster id of virtual node cluster a33 is 22, and the virtual node cluster id of virtual node cluster a44 is 24. Virtual node cluster a21 through virtual node cluster a24 correspond to preset Hash function 3 (Hash 3), and the virtual node cluster identification of virtual node cluster a21 is 30, the virtual node cluster identification of virtual node cluster a22 is 31, the virtual node cluster identification of virtual node cluster a23 is 32, and the virtual node cluster identification of virtual node cluster a24 is 34. Service node a11 through service node a14 correspond to a preset Hash function 4 (Hash 4), and the service node identification of service node a11 is 10, the service node identification of service node a12 is 11, the service node identification of service node a13 is 12, and the service node identification of service node a14 is 13. Then for the first data, a first serving node for storing the first data may be found by the first data identification and based on hashes 1 through hashes 4. The process of how addressing and determining the corresponding identities is based on Hash 1-Hash 3 will be described in detail below.

Optionally, on the basis of the embodiment corresponding to fig. 3, in an optional embodiment of the data migration method provided in the embodiment of the present application, the target virtual node cluster identifier carried by the first data is obtained by taking a remainder for the first hash value of the first data identifier, and the first hash value of the first data identifier is obtained by performing hash calculation on the first data identifier based on a hash function.

In this embodiment, a first hash value of the first data identifier is obtained by performing hash calculation on the first data identifier based on a hash function, and then a remainder is taken from the first hash value of the first data identifier, where the obtained value is a target virtual node cluster identifier carried by the first data. Specifically, each layer of virtual frame corresponds to one preset hash function, so that hash calculation needs to be performed on the first data identifier based on the preset hash function corresponding to the virtual frame layer to which the target virtual node cluster belongs, so as to obtain a first hash value of the first data identifier, and the remainder is specifically taken as the remainder for the number of virtual node clusters of the virtual frame layer to which the target virtual node cluster belongs.

To further understand how to obtain the target virtual node cluster identifier carried by the first data, please refer to fig. 1 again, and take the example that the virtual node cluster a41 through the virtual node cluster a44 correspond to the preset Hash function 1 (Hash 1), and the first data identifier is "X". Based on this, for the first data, the first data identifier X is calculated through the virtual node cluster a41 to the Hash1 corresponding to the virtual node cluster a44, so as to obtain the Hash value Hash1(X) = X1 of the first data identifier, since 4 virtual node clusters are shared in the virtual node cluster a41 to the virtual node cluster a44, the Hash value X1 of the first data identifier is made to take the remainder of 4, if the Hash value X1 of the first data identifier is 4, the Hash value X1 of the first data identifier is made to take the remainder of 4, so as to obtain X1%4=0, at this time, it is determined that the first virtual node cluster in the first layer of virtual framework is the virtual node cluster to which the first service node of the first data storage belongs, and it is known from the foregoing embodiment that the virtual node cluster a41 to the virtual node cluster a44 that the virtual node cluster a is the virtual node cluster a41 (i.e., the virtual node cluster a 68510), the path of the first data storage is now located to virtual node cluster a41 of the first tier framework, and the first data now carries virtual node cluster identification "10" of virtual node cluster a 41.

Further, since virtual node cluster a41 includes virtual node cluster a31 through virtual node cluster a34, further calculations are needed to determine a target virtual node cluster from virtual node cluster a 41. Based on the foregoing example, it is explained that the virtual node cluster a31 to the virtual node cluster a34 correspond to the preset Hash function 2 (Hash 2) as an example. Based on this, for the first data, after determining the virtual node cluster identifier "10" carrying the virtual node cluster a41, the target virtual node cluster described in this embodiment needs to be determined in the virtual node cluster a41, so that the obtained Hash value X1 of the first data identifier is calculated by passing through the Hash2 corresponding to the virtual node cluster a31 to the virtual node cluster a34, so as to obtain a first Hash value Hash1(X1) = X2 of the first data identifier, since there are 4 virtual node clusters in the virtual node cluster a31 to the virtual node cluster a34, the first Hash value X2 of the first data identifier is used to obtain a remainder for 4, if the first Hash value X2 of the first data identifier is 4, the remainder is used to take the first Hash value X2 of the first data identifier for 4, so as to obtain X2%4=0, at this time, it is determined that the first virtual node cluster in the second layer virtual frame is the virtual node cluster a storing the first virtual node cluster data belonging to the virtual node cluster a34, as can be seen from the foregoing embodiments, the virtual node cluster a31 (virtual node cluster id is 20) is set to 0 after the virtual node cluster id in the virtual node cluster a31 to the virtual node cluster a34, at this time, the path of the first data storage is located to the virtual node cluster a31 of the second layer virtual frame, that is, the virtual node cluster a31 is the target virtual node cluster, and at this time, the first data carries the virtual node cluster id "20" of the target virtual node cluster a 31.

Optionally, on the basis of the embodiment corresponding to fig. 3, in an optional embodiment of the data migration method provided in the embodiment of the present application, the first virtual node cluster identifier carried by the first data is obtained by taking a remainder of the second hash value of the first data identifier, and the second hash value of the first data identifier is obtained by performing hash calculation on the first hash value of the first data identifier based on a hash function.

In this embodiment, a second hash value of the first data identifier is obtained by performing hash calculation on the first hash value of the first data identifier based on a hash function, and then a remainder is taken for the second hash value of the first data identifier, where the obtained value is the first virtual node cluster identifier carried by the first data. Specifically, each layer of virtual frame corresponds to one preset hash function, so that the first hash value of the first data identifier obtained in the foregoing embodiment needs to be subjected to hash calculation based on the preset hash function corresponding to the virtual frame layer to which the first virtual node cluster belongs, so as to obtain the second hash value of the first data identifier, and the remainder is specifically taken as the remainder of the number of the virtual node clusters of the virtual frame layer to which the first virtual node cluster belongs.

To further understand how to obtain the target virtual node cluster identifier carried by the first data, please refer to fig. 1 again, based on the foregoing embodiment, after the virtual node cluster a31 is taken as the target virtual node cluster, since the virtual node cluster a31 includes the virtual node cluster a21 to the virtual node cluster a24, further calculation is required to determine the first virtual node cluster from the virtual node cluster a 31. The explanation is given taking as an example that the virtual node cluster a21 through the virtual node cluster a24 correspond to the preset Hash function 3 (Hash 3). Based on this, for the first data, after determining the virtual node cluster identifier "20" carrying the virtual node cluster a31, the first virtual node cluster described in this embodiment needs to be determined in the virtual node cluster a31, so that the obtained first Hash value X2 of the first data identifier is calculated by passing through the Hash3 corresponding to the virtual node cluster a21 to the virtual node cluster a24, so as to obtain the second Hash value Hash1(X2) = X3 of the first data identifier, since there are 4 virtual node clusters in the virtual node cluster a21 to the virtual node cluster a24, the remainder is obtained by adding the second Hash value X3 of the first data identifier to 4, if the second Hash value X3 of the first data identifier is 4, the remainder is obtained by adding the second Hash value X3 of the first data identifier to 4, so as to obtain X3%4=0, at this time, we determine that the first virtual node cluster in the first virtual node cluster a three-layer virtual node storage framework belongs to the virtual node storage node of the first virtual node cluster a virtual node storage framework, as can be seen from the foregoing embodiments, the virtual node cluster a21 (virtual node cluster id is 30) is set to 0 after the virtual node cluster id in the virtual node cluster a21 to the virtual node cluster a24, at this time, the path of the first data storage is located to the virtual node cluster a21 of the third layer virtual frame, that is, the virtual node cluster a21 is the first virtual node cluster, and at this time, the first data carries the virtual node cluster id "30" of the first virtual node cluster a 21.

Optionally, on the basis of the embodiment corresponding to fig. 3, in an optional embodiment of the data migration method provided in the embodiment of the present application, the first service node identifier carried by the first data is obtained by taking a remainder for the third hash value of the first data identifier, and the third hash value of the first data identifier is obtained by performing hash calculation on the second hash value of the first data identifier based on a hash function.

In this embodiment, the second hash value of the first data identifier is subjected to hash calculation based on a hash function to obtain a third hash value of the first data identifier, and then a remainder is taken for the third hash value of the first data identifier, where the obtained value is the first service node identifier carried by the first data. Specifically, each layer of virtual frame corresponds to one preset hash function, so that the second hash value of the first data identifier obtained in the foregoing embodiment needs to be subjected to hash calculation based on the preset hash function corresponding to the virtual frame layer to which the first service node belongs, so as to obtain a third hash value of the first data identifier, and the remainder is specifically obtained by taking the remainder for the number of service nodes of the virtual frame layer to which the first service node belongs.

To further understand how to obtain the identifier of the target virtual node cluster carried by the first data, please refer to fig. 1 again, and based on the foregoing embodiment, after the virtual node cluster a21 is the first virtual node cluster, since the virtual node cluster a21 includes the service node a11 to the service node a14, further calculation is required to determine the first service node from the virtual node cluster a 21. The service node a11 to the service node a14 are illustrated as corresponding to the preset Hash function 4 (Hash 4) as an example. Based on this, for the first data, after determining the virtual node cluster identifier "30" carrying the virtual node cluster a21, the first service node described in this embodiment needs to be determined in the virtual node cluster a21, so that the obtained second Hash value X3 of the first data identifier is calculated through the Hash4 corresponding to the service node a11 to the service node a14, and the third Hash value Hash1(X3) = X4 of the first data identifier is obtained, since the service node a11 to the service node a14 have 4 service nodes in total, the third Hash value X4 of the first data identifier is left over 4, if X4%4=0 is obtained, at this time, we determine that the first service node in the fourth-tier virtual framework is the first service node in the first data storage, as can be known from the foregoing embodiment, the service node a11 to the service node a14 are set as the service node a11 after being set as the service node a 630 (3640), at this time, the path of the first data storage is located to the service node a11 of the fourth layer virtual frame, that is, the service node a11 is the first service node, and at this time, the first data carries the service node identifier "40" of the first service node a 11.

In the embodiment of the application, a method for storing first data in a first service node is provided, and in the above manner, each data has a unique data identifier, so that based on the data identifier of each data, based on a preset hash function corresponding to each layer in a virtual layered frame, a corresponding hash value is obtained through hash calculation to determine the service node capable of storing the data, and therefore the feasibility of the scheme is improved. Secondly, in the process of determining the service node to which the data belongs, the data can carry the virtual node cluster identifier of the virtual node cluster to which the service node belongs and the node identifier of the service node, so that the data can be conveniently read and written, the efficiency of determining the data storage address is improved, and the efficiency of data migration can be improved.

Optionally, on the basis of the embodiment corresponding to fig. 3, in an optional embodiment of the method for data migration provided in the embodiment of the present application, the first virtual node cluster includes a first service node and a second service node;

the determining, by the first virtual node cluster, the second service node from the target virtual node cluster specifically includes:

the first virtual node cluster determines a first virtual node cluster from the target virtual node cluster according to the first virtual node cluster identifier;

the first virtual node cluster determines a second service node from the first virtual node cluster.

In this embodiment, the first virtual node cluster includes a first service node and a second service node, and the first data information further carries a first virtual node cluster identifier. Specifically, the first service node and the second service node belong to the same virtual node cluster, for example, referring to fig. 1 again, if the service node a11 is the first service node and the virtual node cluster a21 is the first virtual node cluster, then the second service node may be any one of the service node a12 to the service node a14 belonging to the first virtual node cluster a21 that satisfies the data bearer condition, which is not limited herein.

The first virtual node cluster includes a first service node and a second service node, and the first data information also carries a first virtual node cluster identifier, so that the first virtual node cluster specifically determines that the first data information belongs to the first data node cluster according to the first virtual node cluster identifier carried by the first data information. For easy understanding, please refer to fig. 6, fig. 6 is a flowchart illustrating a data migration method according to an embodiment of the present application, where, as shown in fig. 6, D31 indicates a target virtual node cluster, D21 indicates a first virtual node cluster, D11 indicates a first service node, and D14 indicates a second service node. Based on this, the service nodes D11 to D14 belonging to the first virtual node cluster D21 report the carried data information, and the data information carried by the service nodes D11 to D14 is similar to the first data information, and both carry the target virtual node cluster identifier and the first virtual node cluster identifier, so that the first virtual node cluster D21 determines that a service node capable of satisfying the data carrying condition is the second service node D14 according to the data information carried by the service nodes D11 to D14, and then directly issues the first data migration instruction to the first service node D11, so that the first service node D11 sends the first data to the second service node D14, and at this time, the data migration can be rapidly completed in the layered architecture on the same layer.

In the embodiment of the application, a data migration method is provided, and by adopting the above manner, data migration is completed between service nodes belonging to the same first virtual node cluster, and the data carried by the service nodes can be adjusted more quickly, so that the efficiency of data migration is improved, and the problem of data inclination is adjusted quickly. Secondly, the first virtual node cluster is determined through the first virtual node cluster identifier, a second service node which belongs to the first virtual node cluster together with the first service node can be found out more quickly, and feasibility of data migration can be improved on the basis of improving efficiency of data migration.

Optionally, on the basis of the embodiment corresponding to fig. 3, in an optional embodiment of the data migration method provided in the embodiment of the present application, the target virtual node cluster further includes a second virtual node cluster;

the first virtual node cluster comprises a first service node;

the second virtual node cluster comprises a second service node;

the first virtual node cluster reports first data information to the target virtual node cluster;

the first virtual node cluster receives a second data migration instruction sent by the target virtual node cluster, wherein the second data migration instruction carries a target virtual node cluster identifier, a second service node identifier and a second virtual node cluster identifier, and the second virtual node cluster identifier indicates the second virtual node cluster;

and the first virtual node cluster determines a second service node according to the second service node identifier.

In this embodiment, the target virtual node cluster further includes a second virtual node cluster, and the first virtual node cluster includes a first service node, and the second virtual node cluster includes a second service node. Specifically, the first service node and the second service node belong to different virtual node clusters under the target virtual node cluster, for example, please refer to fig. 1 again, if the service node a11 is the first service node, the virtual node cluster a21 is the first virtual node cluster, and the virtual node cluster a24 is the second virtual node cluster, then the second service node may be any one of the service nodes a15 to a service node a18 belonging to the second virtual node cluster a24, which satisfies the data carrying condition, and is not limited herein.

Since the first virtual node cluster includes the first service node and the second virtual node cluster includes the second service node, after the first virtual node cluster determines that the first data information belongs to the first data node cluster according to the first virtual node cluster identifier carried by the first data information, the first virtual node cluster reports the data information carried by each service node belonging to the first virtual node cluster, and determines that none of the service nodes in the first virtual node cluster can meet the data carrying condition.

In this case, please refer to fig. 7 for easy understanding, fig. 7 is another flowchart illustrating a method for data migration according to an embodiment of the present application, and as shown in fig. 7, E31 indicates a target virtual node cluster, E21 indicates a first virtual node cluster, E11 indicates a first service node, E24 indicates a second virtual node cluster, and E18 indicates a second service node. Based on this, since the first virtual node cluster E21 belongs to the target virtual node cluster E31, after the first virtual node cluster E21 reports the first data information to the target virtual node cluster E31, and the target virtual node cluster E31 determines that the first service node E11 belongs to the target data node cluster according to the target virtual node cluster identifier carried by the first data information, since the target virtual node cluster E31 includes the virtual node clusters E21 to E24, the data information carried by the target virtual node cluster E31 is reported, the data information carried by the first virtual node cluster E21 is the data information carried by the service nodes E11 to E14, and similarly, the data information carried by the second virtual node cluster E24 is the data information carried by the service nodes E15 to E18, and the data information carried by each service node is similar to the first data information, other virtual node clusters are not described in detail herein.

Based on this, the data information borne by the virtual node cluster E21 to the virtual node cluster E24 all carry the target virtual node cluster identifier, the target virtual node cluster E31 determines, according to the data information, a service node that can satisfy the data bearing condition from the service nodes included in the virtual node cluster E22, the virtual node cluster E23, and the virtual node cluster E24 as a second service node, that is, determines the service node E18 in the virtual node cluster E24 as a second service node, and then the target virtual node cluster E31 issues a second data migration instruction to the first virtual node cluster E21, where the second data migration instruction carries the target virtual node cluster identifier, the second service node cluster identifier, and the second virtual cluster identifier indicates the second virtual node cluster. Therefore, the first virtual node cluster E21 determines the second service node E18 according to the second service node identifier, and issues a first data migration instruction carrying the second virtual node cluster identifier to the first service node E11, so that the first service node E11 sends the first data to the second service node E18 in the second virtual node cluster E24, and at this time, data migration can be completed in a layered architecture of different layers quickly.

In the embodiment of the application, another data migration method is provided, and in the above manner, data migration is completed between service nodes that do not belong to the first virtual node cluster, and data carried by the service nodes can be quickly adjusted by scheduling the upper layer virtual node cluster of the first virtual node cluster, so that the efficiency of data migration is improved, the problem of data inclination is quickly adjusted, and the flexibility of data migration under different conditions is also improved.

the second virtual node cluster comprises a second service node;

the method for data migration further comprises the following steps:

the first virtual node cluster receives second data information reported by a third service node, wherein the second data information carries a target virtual node cluster identifier and a third service node identifier, and the third service node identifier indicates the third service node;

the first virtual node cluster determines that the second data information meets the data migration condition, and determines a target virtual node cluster according to the target virtual node cluster identifier;

the first virtual node cluster reports the first data information and the second data information to the target virtual node cluster;

the first virtual node cluster determines a second service node according to the second service node identifier;

In this embodiment, the target virtual node cluster further includes a second virtual node cluster, the first virtual node cluster includes a first service node and a third service node, and the second virtual node cluster includes a second service node. Specifically, the first service node and the second service node belong to different virtual node clusters under the target virtual node cluster respectively, and the target virtual node cluster includes the first service node and the third service node. For example, referring to fig. 1 again, if the service node a11 is a first service node, the virtual node cluster a21 is a first virtual node cluster, and the virtual node cluster a24 is a second virtual node cluster, then the second service node may be any one of the service nodes a15 to a service node a18 belonging to the second virtual node cluster a24 and satisfying the data bearer condition, and the third service node may be any one of the service nodes a12 to a service node a14 belonging to the first virtual node cluster a21, which is not limited herein.

Based on this, the first virtual node cluster receives second data information reported by the third service node, where the second data information carries a target virtual node cluster identifier and a third service node identifier, and the third service node identifier indicates the third service node, and the specific second data information is similar to the first data information, and is not described here again. Further, the first virtual node cluster determines whether the second data information meets a data migration condition, and if so, it indicates that the second data information carried by the third service node exceeds the load capacity of the third service node, that is, the load of the third service node is too high, which causes data skew, and data carried by the third service node needs to be migrated.

Further, please refer to fig. 8, fig. 8 is another flowchart of the method for data migration according to the embodiment of the present application, as shown in fig. 8, F31 indicates a target virtual node cluster, F21 indicates a first virtual node cluster, F24 indicates a second virtual node cluster, F11 indicates a first service node, F12 indicates a third service node, and F18 indicates a second service node. Based on this, the first virtual node cluster F21 reports the first data information and the second data information to the target virtual node cluster F31, and the target virtual node cluster F31 determines according to the target virtual node cluster identifier carried by the first data information and the second data information that the first service node F11 and the third service node F12 belong to the target data node cluster, and since the target virtual node cluster F31 includes the virtual node clusters F21 to F24, the data information carried by the virtual node cluster F31 is reported.

Secondly, the data information borne by the virtual node cluster F21 to the virtual node cluster F24 all carry a target virtual node cluster identifier, the target virtual node cluster F31 determines, according to the data information, a service node capable of meeting the data bearing condition as a second service node from the service nodes included in the virtual node cluster F22, the virtual node cluster F23 and the virtual node cluster F24, that is, determines the service node F18 in the virtual node cluster F24 as a second service node, and then the target virtual node cluster F31 issues a second data migration instruction to the first virtual node cluster F21, where the second data migration instruction carries the target virtual node cluster identifier, the second service node identifier and the second virtual node cluster identifier, and the second virtual node cluster identifier indicates the second virtual node cluster. Therefore, the first virtual node cluster F21 determines the second service node F18 according to the second service node identifier, and issues a first data migration instruction to the first service node F11 and the third service node F12, and since the first data migration instruction also carries the second virtual node cluster identifier, the first service node F11 sends the first data to the second service node F18 in the second virtual node cluster F24, and the third service node F12 sends the second data to the second service node F18 in the second virtual node cluster F24, where the second data belongs to the aforementioned second data information.

In the embodiment of the application, a data migration method is provided, and in the above manner, when a plurality of service nodes belonging to a first virtual node cluster all exceed bearable load capacity, data migration on the plurality of service nodes is completed by scheduling a virtual node cluster on a layer above the first virtual node cluster, so that efficiency of data migration is further improved.

Optionally, on the basis of the embodiment corresponding to fig. 3, in an optional embodiment of the data migration method provided in the embodiment of the present application, the data migration condition is that the first data information is greater than the first load data threshold;

In this embodiment, the data migration condition is specifically that the first data information is greater than a first load data threshold, and the first load data threshold is determined through multiple experiments and data. Whether the first data information carried by the first service node exceeds the load capacity of the first service node can be determined through the first load data threshold.

Secondly, the data carrying condition is that the data that can be carried by the second service node is greater than a second load data threshold, which is similar to the first load data threshold, the second load data threshold is determined through multiple experiments and data, the second load data threshold may be the same as or different from the first load data threshold, and it needs to be determined flexibly according to the actual situation. Whether the data which can be borne by the second service node is smaller than the load capacity of the second service node can be judged through the second load data threshold value, so that the situation that the second service node is overloaded after the data migration is completed is avoided.

And thirdly, after the first data is migrated to the second service node, replacing the first service node identifier carried by the first data with the second service node identifier. As can be seen from the foregoing embodiment, when the first data is stored in the first service node, the first data will carry the first service node identifier of the first service node, and after the first data is migrated to the second service node, the first data will not carry the first service node identifier, but carry the second service node identifier of the second service node. Further, if the first service node and the second service node both belong to the first virtual node cluster, the first data still carries the first virtual node cluster identifier. If the first service node belongs to the first virtual node cluster and the second service node belongs to the second virtual node cluster, the first data still does not carry the identifier of the first virtual node cluster, but carries the identifier of the second virtual node cluster, and so on, the replacement of the service node stored by the first data can cause the replacement of the carried identifier, and the replacement of the specific identifier needs to be determined according to the actual condition of data migration.

In the embodiment of the application, a data migration method is provided, and in the above manner, after data migration is completed, a service node identifier of migrated data (i.e., first data) is corrected, so that data addressing accuracy during data reading and writing is ensured, and reliability of data migration is improved.

The scheme provided by the embodiment of the present application is introduced mainly in terms of methods, and the method for data migration provided by the embodiment of the present application can be applied to the following application scenarios:

the first type of application scenario is a game, and the service types of the game are more complex compared with other applications, such as mail service, world chat service, group service, copy service, map service and the like, so that when the game scenario is applied, service nodes serving the same type are generally divided into the same virtual node cluster, and therefore when data carried by the service nodes in the same service type exceed loads, the service nodes capable of carrying migrated data can be determined in the service nodes of the same type through the data migration method provided by the scheme, and the service quality of the service of the same type is ensured. Secondly, different game features are different, so that a way of specifically layering virtual node clusters is different, and if the method is specifically applied to a Massively Multiplayer Online Role Playing Game (MMORPG), since the MMORPG is usually divided into a plurality of regional servings, the division of the virtual node clusters can be performed not by service types but by different regional servings, so that one virtual node cluster serves one regional servings, each virtual node cluster can carry different services in the same regional servings, and therefore when data carried by service nodes in the same service type exceeds load, a service node capable of carrying migration data can be determined in the service nodes in the same regional servings by the data migration method provided by the scheme, and the service node capable of carrying migration data can be a service node carrying different service data, thereby increasing the flexibility of data migration.

The second type of application scenario is a communication type application, the service of the communication type application can be an instant messaging application, the instant messaging application can be real-time communication among multiple people, and the real-time communication content can include text, image, voice, video and file data transmission. Therefore, when the method is applied to communication applications, the service nodes for reading and writing similar data can be divided into the same virtual node cluster, for example, the service nodes for reading and writing characters belong to the same virtual node cluster, and the service nodes for reading and writing images belong to the same virtual node cluster. If the data borne by the service nodes for reading and writing the characters exceeds the load, the data migration method provided by the scheme can determine the service nodes capable of bearing the migrated data from other service nodes for reading and writing the characters, so that the reliability of data migration on the same-kind data is ensured.

It is to be understood that the foregoing application scenarios are only presented for further understanding of the present solution, and in practical applications, specific application scenarios of the method for data migration provided in the embodiments of the present application include, but are not limited to, several of the above examples, and specific application scenarios that are available here are not exhaustive.

Referring to fig. 9, fig. 9 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present application, and as shown in fig. 9, the data migration apparatus 900 includes:

a receiving module 901, configured to receive first data information reported by a first service node, where the first data information carries a target virtual node cluster identifier and a first service node identifier, the target virtual node cluster identifier indicates a target virtual node cluster, the first service node identifier indicates the first service node, the first virtual node cluster is composed of multiple service nodes, and the target virtual node cluster is composed of multiple virtual node clusters;

a determining module 902, configured to determine, if the first data information satisfies the data migration condition, a target virtual node cluster according to the target virtual node cluster identifier;

a determining module 902, further configured to determine a second service node from the target virtual node cluster, where the second service node meets the data carrying condition;

a sending module 903, configured to send a first data migration instruction to a first service node, where the first data migration instruction carries a target virtual node cluster identifier and a second service node identifier, the second service node identifier indicates a second service node, and the second data migration instruction indicates the first service node to send first data to the second service node.

Optionally, on the basis of the embodiment corresponding to fig. 9, in another embodiment of the data migration apparatus 900 provided in this embodiment of the present application, the target virtual node cluster includes a first virtual node cluster, and the first data carries the first data identifier;

Optionally, on the basis of the embodiment corresponding to fig. 9, in another embodiment of the data migration apparatus 900 provided in this embodiment of the present application, the target virtual node cluster identifier carried by the first data is obtained by taking a remainder of the first hash value of the first data identifier, and the first hash value of the first data identifier is obtained by performing hash calculation on the first data identifier based on a hash function.

Optionally, on the basis of the embodiment corresponding to fig. 9, in another embodiment of the data migration apparatus 900 provided in this embodiment of the present application, the first virtual node cluster identifier carried by the first data is obtained by taking a remainder of the second hash value of the first data identifier, and the second hash value of the first data identifier is obtained by performing hash calculation on the first hash value of the first data identifier based on a hash function.

Optionally, on the basis of the embodiment corresponding to fig. 9, in another embodiment of the data migration apparatus 900 provided in this embodiment of the present application, the first service node identifier carried by the first data is obtained by taking a remainder of a third hash value of the first data identifier, and the third hash value of the first data identifier is obtained by performing hash calculation on a second hash value of the first data identifier based on a hash function.

Optionally, on the basis of the embodiment corresponding to fig. 9, in another embodiment of the data migration apparatus 900 provided in this embodiment of the present application, the first virtual node cluster includes a first service node and a second service node;

a determining module 902, configured to determine, according to the first virtual node cluster identifier, a first virtual node cluster from the target virtual node clusters;

a second service node is determined from the first cluster of virtual nodes.

Optionally, on the basis of the embodiment corresponding to fig. 9, in another embodiment of the data migration apparatus 900 provided in the embodiment of the present application, the target virtual node cluster further includes a second virtual node cluster;

the first virtual node cluster comprises a first service node;

the second virtual node cluster comprises a second service node;

a determining module 902, configured to report first data information to a target virtual node cluster;

the second virtual node cluster comprises a second service node;

the receiving module 901 is further configured to receive second data information reported by a third service node, where the second data information carries a target virtual node cluster identifier and a third service node identifier, and the third service node identifier indicates the third service node;

the determining module 902 is further configured to determine that the second data information meets the data migration condition, and determine the target virtual node cluster according to the target virtual node cluster identifier;

a determining module 902, configured to report the first data information and the second data information to the target virtual node cluster;

Optionally, on the basis of the embodiment corresponding to fig. 9, in another embodiment of the data migration apparatus 900 provided in this embodiment of the present application, the data migration condition is that the first data information is greater than the first load data threshold;

Referring to fig. 10, fig. 10 is a schematic diagram of an embodiment of a server in the embodiment of the present application, and as shown in the drawing, the server 1000 may generate a large difference due to different configurations or performances, and may include one or more Central Processing Units (CPUs) 1022 (e.g., one or more processors) and a memory 1032, and one or more storage media 1030 (e.g., one or more mass storage devices) storing an application program 1042 or data 1044. Memory 1032 and storage medium 1030 may be, among other things, transient or persistent storage. The program stored on the storage medium 1030 may include one or more modules (not shown), each of which may include a series of instruction operations for the server. Still further, a central processor 1022 may be disposed in communication with the storage medium 1030, and configured to execute a series of instruction operations in the storage medium 1030 on the server 1000.

The server 1000 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input-output interfaces 1058, and/or anOne or more operating systems 1041, e.g. Windows Server^TM，Mac OS X^TM，Unix^TM，Linux^TM，FrEBSD^TMAnd so on.

The steps performed by the server in the above embodiment may be based on the server structure shown in fig. 10.

The server includes a CPU 1022 for executing the embodiment shown in fig. 3 and the corresponding embodiments in fig. 3.

Also provided in an embodiment of the present application is a computer-readable storage medium, which stores a computer program, and when the computer program runs on a computer, the computer program causes the computer to execute the steps performed by the first virtual node cluster of the server in the method described in the foregoing embodiment shown in fig. 3 and the methods described in fig. 3.

Also provided in an embodiment of the present application is a computer program product including a program, which when run on a computer, causes the computer to perform the steps performed by the first virtual node cluster of the server in the method described in the embodiment of fig. 3.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. A method of data migration, comprising:

a first virtual node cluster receives first data information reported by a first service node, wherein the first data information carries a target virtual node cluster identifier and a first service node identifier, the target virtual node cluster identifier indicates a target virtual node cluster, the first service node identifier indicates the first service node, the first virtual node cluster is composed of a plurality of service nodes, the target virtual node cluster is composed of a plurality of virtual node clusters, the target virtual node cluster comprises the first virtual node cluster, the first data further carries a first data identifier, the target virtual node cluster identifier is determined based on the first data identifier, and the first virtual node cluster identifier carried by the first data is determined based on the target virtual node cluster and the first data identifier, the first service node identification is determined based on the first virtual node cluster and the first data identification;

if the first data information meets the data migration condition, the first virtual node cluster determines the target virtual node cluster according to the target virtual node cluster identifier;

the first virtual node cluster determines a second service node from the target virtual node cluster, wherein the second service node meets a data bearing condition;

the first virtual node cluster sends a first data migration instruction to the first service node, wherein the first data migration instruction carries the target virtual node cluster identifier and a second service node identifier, the second service node identifier indicates the second service node, and the first data migration instruction indicates that the first service node sends first data to the second service node.

2. The method according to claim 1, wherein the target virtual node cluster identifier carried by the first data is obtained by taking a remainder of a first hash value of the first data identifier, and the first hash value of the first data identifier is obtained by performing hash calculation on the first data identifier based on a hash function.

3. The method according to claim 2, wherein a first virtual node cluster identifier carried by the first data is obtained by taking a remainder for a second hash value of the first data identifier, and the second hash value of the first data identifier is obtained by performing a hash calculation on a first hash value of the first data identifier based on a hash function.

4. The method according to claim 3, wherein a first service node identifier carried by the first data is obtained by taking a remainder for a third hash value of the first data identifier, and the third hash value of the first data identifier is obtained by performing a hash calculation on a second hash value of the first data identifier based on a hash function.

5. The method of claim 1, wherein the first cluster of virtual nodes comprises the first service node and the second service node;

the first data information also carries the first virtual node cluster identifier;

the first virtual node cluster determining a second service node from the target virtual node cluster, including:

the first virtual node cluster determines the first virtual node cluster from the target virtual node cluster according to the first virtual node cluster identifier;

the first virtual node cluster determines the second service node from the first virtual node cluster.

6. The method of claim 1, wherein the target virtual node cluster further comprises a second virtual node cluster;

the first cluster of virtual nodes comprises the first service node;

the second cluster of virtual nodes comprises the second service node;

the first virtual node cluster reports the first data information to the target virtual node cluster;

the first virtual node cluster receives a second data migration instruction sent by the target virtual node cluster, wherein the second data migration instruction carries the target virtual node cluster identifier, the second service node identifier and a second virtual node cluster identifier;

and the first virtual node cluster determines the second service node according to the second service node identifier.

7. The method of claim 1, wherein the target virtual node cluster further comprises a second virtual node cluster;

the first virtual node cluster comprises the first service node and a third service node;

the second cluster of virtual nodes comprises the second service node;

the method further comprises the following steps:

the first virtual node cluster receives second data information reported by the third service node, wherein the second data information carries the target virtual node cluster identifier and a third service node identifier, and the third service node identifier indicates the third service node;

the first virtual node cluster determines that the second data information meets the data migration condition, and determines the target virtual node cluster according to the target virtual node cluster identifier;

the first virtual node cluster receives a second data migration instruction sent by the target virtual node cluster, wherein the second data migration instruction carries the target virtual node cluster identifier, the second service node identifier and a second virtual node cluster identifier, and the second virtual node cluster identifier indicates the second virtual node cluster;

the first virtual node cluster determines the second service node according to the second service node identifier;

the first data migration instruction further carries the second virtual node cluster identifier, and the first data migration instruction further instructs the third service node to migrate the loaded second data to the second service node.

8. The method of claim 1, wherein the data migration condition is that the first data information is greater than a first load data threshold;

9. A data migration apparatus, characterized in that the data migration apparatus comprises:

a receiving module, configured to receive first data information reported by a first service node, where the first data information carries a target virtual node cluster identifier and a first service node identifier, the target virtual node cluster identifier indicates a target virtual node cluster, the first service node identifier indicates the first service node, the first virtual node cluster is composed of multiple service nodes, the target virtual node cluster is composed of multiple virtual node clusters, the target virtual node cluster includes the first virtual node cluster, the first data further carries a first data identifier, the target virtual node cluster identifier is determined based on the first data identifier, and the first virtual node cluster identifier carried by the first data is determined based on the target virtual node cluster and the first data identifier, the first service node identification is determined based on the first virtual node cluster and the first data identification;

a determining module, configured to determine the target virtual node cluster according to the target virtual node cluster identifier if the first data information satisfies a data migration condition;

a sending module, configured to send a first data migration instruction to the first service node, where the first data migration instruction carries the target virtual node cluster identifier and a second service node identifier, the second service node identifier indicates the second service node, and the first data migration instruction indicates that the first service node sends first data to the second service node.

10. A computer device, comprising: a memory, a transceiver, a processor, and a bus system;

wherein the memory is used for storing programs;

the processor is configured to execute a program in the memory to implement the method of any one of claims 1 to 8;

the bus system is used for connecting the memory and the processor so as to enable the memory and the processor to communicate.

11. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any of claims 1 to 8.