CN113791736A

CN113791736A - Data migration method, network card device, server and data migration system

Info

Publication number: CN113791736A
Application number: CN202111077950.0A
Authority: CN
Inventors: 胡睿; 廖恬瑜
Original assignee: Jingdong Technology Information Technology Co Ltd
Current assignee: Jingdong Technology Information Technology Co Ltd
Priority date: 2021-09-15
Filing date: 2021-09-15
Publication date: 2021-12-14

Abstract

The embodiment of the application provides a data migration method, a network card device, a server and a data migration system, and is applied to the technical field of computers. According to the method, data to be migrated is divided into N fragmented data groups, a target fragmented data group comprises a plurality of fragmented data, when a data migration request corresponding to first target fragmented data in the target fragmented data group is identified, storage space in a second memory is applied for M fragmented data in the target fragmented data group, so that the M fragmented data in the target fragmented data group are merged, a data migration operation is executed on the merged target fragmented data group, and the M fragmented data in the target fragmented data group is migrated to the network storage device from the second memory or the network storage device from the second memory, so that time for completing migration of the data to be migrated by a second processor is saved, and the performance of the second processor is improved.

Description

Data migration method, network card device, server and data migration system

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data migration method, a network card device, a server, and a data migration system.

Background

As the complexity of service processing of an application program on a server is increasingly increased, in order to improve the performance of a network subsystem in the service processing process, an intelligent network card may be deployed in the server, and part of functions of a Central Processing Unit (CPU) in the server are transferred to the intelligent network card, so as to reduce the load of the CPU in the server.

At present, data writing or reading between a network storage device and a memory arranged in a server can be realized through an intelligent network card. However, the data writing and reading operations implemented by the current intelligent network card cause the processing time of the CPU in the intelligent network card to be long, and further cause the problem of poor processing performance of the CPU in the intelligent network card.

Disclosure of Invention

The embodiment of the application provides a data migration method, a network card device, a server and a data migration system, and when data writing and reading operations are realized through an intelligent network card, the processing time of a CPU in the intelligent network card is reduced, and the processing performance of the CPU in the intelligent network card is improved.

In a first aspect, an embodiment of the present application provides a data migration method, which is applied to a network card device, where the network card device is deployed in a server, the server further includes a first processor and a first memory, and the network card device includes a second memory; the method comprises the following steps: when a data migration instruction aiming at data to be migrated and sent by a first processor is received, dividing the data to be migrated into N fragmented data groups; a target fragmented data group in the N fragmented data groups comprises M fragmented data, wherein N is a positive integer, and M is a positive integer larger than 1; generating a data migration request corresponding to each fragmented data in the target fragmented data group; when a data migration request corresponding to first target fragmented data in a target fragmented data group is identified, applying for a storage space in a second memory for M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data; the first target fragment data is one fragment data in the target fragment data group; executing one data migration operation, and migrating the M piece data in the target piece data group from the first storage unit to the second storage unit; when the data migration operation is a data write operation, the first storage unit is a second memory, and the second storage unit is a network storage device; when the data migration operation is a data reading operation, the first storage unit is a network storage device, and the second storage unit is a second memory; m pieces of fragment data in the target fragment data group are sequentially migrated between the first memory and the second memory.

Optionally, the data migration request corresponding to the first target fragmented data includes a first identifier, and when the data migration request corresponding to the first target fragmented data in the target fragmented data group is identified, according to the data migration request corresponding to the first target fragmented data, a storage space in the second memory is applied for M fragmented data in the target fragmented data group, including: and when a data migration request carrying the first identifier in the target fragmented data group is identified, applying for a storage space in a second memory for the M fragmented data in the target fragmented data group according to the data migration request carrying the first identifier.

Optionally, the data migration operation is a data write operation; before executing a data migration operation and migrating the M pieces of fragmented data in the target fragmented data group from the first storage unit to the second storage unit, the method further includes: according to data migration requests corresponding to M pieces of fragment data in a target fragment data group, sequentially migrating the M pieces of fragment data from a first memory to a second memory; determining whether the M pieces of fragment data are completely migrated from the first memory to the second memory; executing a data migration operation once, and migrating the M piece data in the target piece data group from the first storage unit to the second storage unit, including: and when the M pieces of fragment data are completely migrated from the first memory to the second memory, executing one data migration operation, and migrating the M pieces of fragment data in the target piece of fragment data from the second memory to the network storage device.

Optionally, the last fragmented data in the target fragmented data group migrated from the first memory to the second memory is second target fragmented data, and the data migration request corresponding to the second target fragmented data includes a second identifier; determining whether all the M pieces of fragmented data have been migrated from the first memory to the second memory, including: and determining whether second target fragmented data corresponding to the data migration request carrying the second identifier in the target fragmented data group is migrated from the first memory to the second memory.

Optionally, the data migration operation is a data read operation, and executes a data migration operation to migrate M pieces of fragmented data in the target fragmented data group from the first storage unit to the second storage unit, where the data migration operation includes: determining whether all data migration requests corresponding to M fragmented data in the target fragmented data group are generated; when all the data are generated, executing one-time data migration operation, and migrating the M fragment data in the target fragment data group from the network storage device to a second memory; after executing a data migration operation and migrating the M fragmented data in the target fragmented data group from the first storage unit to the second storage unit, the method further includes: and sequentially migrating the M pieces of fragment data from the second memory to the first memory according to the data migration requests corresponding to the M pieces of fragment data in the target fragment data group.

Optionally, the last data migration request generated in the target fragmented data group includes a third identifier; determining whether data migration requests corresponding to M sharded data in the target sharded data group are all generated, including:

and determining whether a data migration request carrying the third identifier in the target fragmented data group is generated.

Optionally, N is 1.

In a second aspect, an embodiment of the present application provides a network card device, where the network card device is deployed in a server, the server further includes a first processor and a first memory, and the network card device includes a second processor, a logic chip, and a second memory; the logic chip is used for dividing the data to be migrated into N fragmented data groups when receiving a data migration instruction aiming at the data to be migrated and sent by the first processor; generating a data migration request corresponding to each fragmented data in the target fragmented data group; a target fragmented data group in the N fragmented data groups comprises M fragmented data, wherein N is a positive integer, and M is a positive integer larger than 1; the second processor is used for applying for a storage space in a second memory for the M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data when the data migration request corresponding to the first target fragmented data in the target fragmented data group is identified; executing one data migration operation, and migrating the M piece data in the target piece data group from the first storage unit to the second storage unit; the first target fragment data is one fragment data in the target fragment data group; when the data migration operation is a data write operation, the first storage unit is a second memory, and the second storage unit is a network storage device; when the data migration operation is a data reading operation, the first storage unit is a network storage device, and the second storage unit is a second memory; m pieces of fragment data in the target fragment data group are sequentially migrated between the first memory and the second memory.

Optionally, the data migration request corresponding to the first target fragmented data includes a first identifier; and the second processor is specifically configured to, when a data migration request carrying the first identifier in the target fragmented data group is identified, apply for a storage space in the second memory for the M fragmented data in the target fragmented data group according to the data migration request carrying the first identifier.

Optionally, the data migration operation is a data write operation; the logic chip is further used for sequentially migrating the M pieces of fragment data from the first memory to the second memory according to the data migration requests corresponding to the M pieces of fragment data in the target fragment data group; the second processor is further used for determining whether all the M pieces of fragment data are migrated from the first memory to the second memory; and when the M pieces of fragment data are completely migrated from the first memory to the second memory, executing one data migration operation, and migrating the M pieces of fragment data in the target piece of fragment data from the second memory to the network storage device.

Optionally, the last fragmented data in the target fragmented data group migrated from the first memory to the second memory is second target fragmented data, and the data migration request corresponding to the second target fragmented data includes a second identifier; the second processor is specifically configured to determine whether second target fragmented data corresponding to the data migration request carrying the second identifier in the target fragmented data group has been migrated from the first memory to the second memory.

Optionally, the data migration operation is a data read operation; the second processor is specifically configured to determine whether all data migration requests corresponding to M fragmented data in the target fragmented data group have been generated; when all the fragment data groups are generated, executing one-time data migration operation, and migrating the M fragment data in the target fragment data group from the network storage device to a second memory; and the logic chip is further used for sequentially migrating the M pieces of fragment data from the second memory to the first memory according to the data migration requests corresponding to the M pieces of fragment data in the target fragment data group.

Optionally, the last data migration request generated in the target fragmented data group includes a third identifier; and the second processor is specifically configured to determine whether a data migration request carrying the third identifier in the target fragmented data group has been generated.

Optionally, N is 1.

In a third aspect, an embodiment of the present application provides a server, which includes a first processor, a first memory, and the network card device.

In a fourth aspect, an embodiment of the present application provides a data migration system, which includes a network storage device and the server described above, where the network storage device is in communication connection with the server.

In the embodiment of the application, the data to be migrated is divided into N fragmented data groups, and a target fragmented data group in the N fragmented data groups includes a plurality of fragmented data, so that the number N of the fragmented data groups is smaller than the number of all fragmented data in the data to be migrated; and generating a data migration request corresponding to each fragmented data in the target fragmented data group, and applying for a storage space in a second memory for M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data when identifying the data migration request corresponding to the first target fragmented data in the target fragmented data group, so as to implement merging of M fragmented data in the target fragmented data group in the second memory, then performing a data migration operation on the merged target fragmented data group once, and migrating the M fragmented data in the target fragmented data group from the second memory to a network storage device or from the network storage device to the second memory. Therefore, for N fragmented data groups in the data to be migrated, N times of data migration operations need to be executed, and all fragmented data included in the data to be migrated is written into or read from the network storage device in N times, so that the distribution processing procedure of the second processor and the number of times of calling the tcp protocol stack are reduced, the time required for the second processor to complete migration of the data to be migrated is saved, and the processing performance of the second processor is improved.

Drawings

FIG. 1 is an interaction diagram of a server in the related art implementing data write operations for data to be migrated;

FIG. 2 is an interaction diagram of a server in the related art implementing a data read operation of data to be migrated;

fig. 3 is a schematic diagram of a hardware structure of a server equipped with a network card device according to an embodiment of the present application;

fig. 4 is a flowchart of a data migration method according to an embodiment of the present application;

FIG. 5 is a flowchart of a data write operation of data to be migrated according to an embodiment of the present application;

fig. 6 is a specific interaction diagram of a server implementing data write operation of data to be migrated according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a virtio data structure provided by an embodiment of the present application;

FIG. 8 is a flowchart of a data read operation of data to be migrated according to an embodiment of the present application;

fig. 9 is a specific interaction diagram of a server implementing data read operation of data to be migrated according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of a data migration system according to an embodiment of the present application.

Detailed Description

In the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same or similar items having substantially the same function and action. For example, the first chip and the second chip are only used for distinguishing different chips, and the sequence order thereof is not limited. Those skilled in the art will appreciate that the terms "first," "second," etc. do not denote any order or quantity, nor do the terms "first," "second," etc. denote any order or importance.

It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.

In the embodiments of the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or multiple.

In the using process of the server, a large amount of data is often generated, and in order to reduce the storage pressure of the server, the data to be migrated stored in a first memory in the server can be backed up to the network storage device, and the data to be migrated in the first memory can be cleared; and if the data to be migrated in the network storage equipment needs to be used subsequently, reading the data to be migrated from the network storage equipment and storing the data to be migrated in a first memory in the server.

In order to reduce the load of the CPU in the server for processing network transmission, a network card device may be deployed in the server, and data may be written in or read out between the network storage device and a first memory provided in the server based on the smart network card. The network card device can also be called an intelligent network card, the intelligent network card can transfer network protocol analysis processing and frequent I/O (write in/read out) work consuming the CPU processing capacity of the server to the intelligent network card, and the processing capacity of the server application program is greatly improved on the premise of not changing the implementation of the server application program.

In the related art, a CPU in a server may be referred to as a first processor, a memory in the server may be referred to as a first memory, a CPU, a memory and a logic chip may also be disposed in a network card device disposed in the server, the CPU in the network card device may be referred to as a second processor, and the memory in the network card device may be referred to as a second memory. The second processor comprises a logic driving module and an upper layer software module, wherein the logic driving module refers to a hardware logic driver which is positioned at the bottom layer in the second processor and interacts with the logic chip, and the upper layer software module refers to an upper layer application program which is positioned in an operating system framework and is used for interacting with the network storage device.

As shown in fig. 1, when the network card device needs to migrate data to be migrated in the first memory of the server to the network storage device, the first processor first executes S101, loads a virtio driver, and converts the data to be migrated stored in the first memory into a virtio data structure; then, the first processor executes S102 and sends a data write instruction to the logic chip; after the logic chip receives a data write instruction sent by the first processor, the logic chip executes S103, reads a descriptor chain in the virtio data structure from the first memory, and generates a data write request corresponding to each piece of fragment data included in the data to be migrated according to each descriptor in the descriptor chain; then, the logic chip executes S104, and sequentially writes the data write requests corresponding to each piece of fragmented data into a second memory; the logic driving module in the second processor executes step S105, sequentially reads data writing requests from the second memory, and applies for a storage space subsequently stored in the second memory for the fragmented data corresponding to the data writing requests according to the data writing requests; then, the logic driving module executes step S106 to send a notification message to the logic chip that the fragmented data needs to be migrated from the first memory to the second memory; then, the logic chip executes S107 to migrate the fragmented data from the first memory to the second memory; after the logic chip migrates the fragmented data from the first memory to the second memory, the logic chip executes S108 and sends a notification message that the fragmented data has been migrated to the second memory to the logic driver module; the edit driving module executes S109 and forwards the notification message that the fragmented data has been migrated to the second memory to the upper layer software module in the second processor.

The upper layer software module in the second processor executes S110, and distributes the write operation request of the fragmented data (i.e. the write operation request written into the network storage device) to different processing modules according to the position of the fragmented data which needs to be stored in the network storage device subsequently; the upper layer software module executes the step S111, calls a transmission control protocol (tcp) protocol stack of the linux kernel, and migrates the fragment data in the second memory to the network storage device; after receiving the fragment data, the network storage device returns a response message to the upper layer software module to prompt that the network storage device has received the fragment data, and then the upper layer software module executes S112 to send the response message returned by the network storage device to the logic driver module.

When the logic driving module determines that a response message that the last fragmented data in the data to be migrated has been written into the network storage device is received, the logic driving module executes S113 and sends a notification message that all fragmented data included in the data to be migrated has been migrated to the network storage device to the logic chip; the logic chip executes S114, and writes a notification message that all the fragmented data included in the data to be migrated has been migrated to the network storage device into the first memory; when the first processor reads the notification message that all the fragmented data in the first memory have been migrated to the network storage device, the first processor executes S115 to release the resources, and the entire data write operation of the data to be migrated is completed.

It should be noted that, the processes of S104 to S112 all aim at one piece of fragmented data in the data to be migrated, that is, when the data to be migrated includes X pieces of fragmented data, S104 to S112 need to be executed sequentially X times, and when the logic driving module receives a response message corresponding to the xth piece of fragmented data in the data to be migrated, S113 and subsequent operations are executed.

As shown in fig. 2, when the network card device needs to migrate data to be migrated in the network storage device to the first memory of the server, the first processor first executes S201, loads a virtio driver, and generates a virtio data structure in the first memory; then, the first processor executes S202, and sends a data read instruction to the logic chip; after the logic chip receives the data reading instruction sent by the first processor, the logic chip executes S203, reads the descriptor chain in the virtio data structure from the first memory, and generates a data reading request corresponding to each piece of fragment data included in the data to be migrated according to each descriptor in the descriptor chain; then, the logic chip executes 204 to write the data read request corresponding to each piece of fragmented data into the second memory; executing S205 by a logic driving module in the second processor, reading a data reading request from the second memory, and applying for a storage space subsequently stored in the second memory for the corresponding fragmented data according to the data reading request; then, the logic driver module executes S206 to send a notification message to the upper software module in the second processor that the fragmented data needs to be migrated from the network storage device to the second memory.

After receiving the notification message that the fragmented data sent by the logic driver module needs to be migrated to the second memory, the upper layer software module in the second processor executes S207, and distributes a read operation request of the fragmented data (i.e., a read operation request read from the network storage device) to different processing modules according to the storage location of the fragmented data in the network storage device; the upper layer software module executes S208, calls a tcp protocol stack of the linux kernel, and migrates the fragmented data in the network storage device to the second memory; after the network storage device sends the fragment data to the server, a response message is returned to the upper layer software module to prompt that the network storage device has sent the fragment data, and then the upper layer software module executes S209 to send the response message returned by the network storage device to the logic driving module; then, the logic driving module executes S210, and sends a notification message that the fragmented data needs to be migrated from the second memory to the first memory to the logic chip; then, the logic chip executes S211 to migrate the fragmented data from the second memory to the first memory; after the logic chip migrates the fragmented data from the second memory to the first memory, the logic chip executes S212 and sends a notification message that the fragmented data has been migrated to the first memory to the logic driver module.

When the logic drive module determines that a notification message that the last piece of data in the data to be migrated has been migrated to the first memory is received, the logic drive module executes S213 and sends a notification message that all piece of data included in the data to be migrated has been migrated to the first memory to the logic chip; the logic chip executes step S214, and writes a notification message that all the fragmented data included in the data to be migrated has been migrated to the first memory into the first memory; when the first processor reads the notification message that all the fragmented data in the first memory have been migrated to the first memory, the first processor executes S215 to release the resources, and the whole data reading operation of the data to be migrated is completed.

It should be noted that, the processes of S204 to S212 are all for one piece of fragmented data in the data to be migrated, that is, when the data to be migrated includes X pieces of fragmented data, S204 to S212 need to be sequentially executed X times, and when the logic driving module determines that the notification message that the xth piece of fragmented data in the data to be migrated has been migrated to the first memory is received, S213 and subsequent operations are executed.

Therefore, it can be seen that, for data write operation and data read operation, each piece of fragmented data in the data to be migrated needs to execute a distribution processing procedure of data write operation once and call a tcp protocol stack of kernel state once, and when the data to be migrated includes X pieces of fragmented data, if all pieces of fragmented data in the data to be migrated need to be migrated, the distribution processing procedure of data write operation X times and call the tcp protocol stack of kernel state X times need to be executed. Because each distribution processing procedure of the second processor and each procedure of calling the tcp protocol stack consume the time of the second processor, the processing time required by the second processor is long due to the multiple distribution processing procedures and the multiple calling of the tcp protocol stack, and the processing performance of the second processor is poor.

Based on this, an embodiment of the present application provides a data migration method, where data to be migrated is divided into N fragmented data groups, and a target fragmented data group in the N fragmented data groups includes multiple fragmented data, so that the number N of the fragmented data groups is smaller than the number X of all fragmented data in the data to be migrated; and generating a data migration request corresponding to each fragmented data in the target fragmented data group, and applying for a storage space in a second memory for M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data when identifying the data migration request corresponding to the first target fragmented data in the target fragmented data group, so as to implement merging of M fragmented data in the target fragmented data group in the second memory, then performing a data migration operation on the merged target fragmented data group once, and migrating the M fragmented data in the target fragmented data group from the second memory to a network storage device or from the network storage device to the second memory. Therefore, for N fragmented data groups in the data to be migrated, N times of data migration operations need to be executed, and all fragmented data included in the data to be migrated is written into or read from the network storage device in N times, so that the distribution processing procedure of the second processor and the number of times of calling the tcp protocol stack are reduced, the time required for the second processor to complete migration of the data to be migrated is saved, and the processing performance of the second processor is improved.

When the number N of the fragmented data groups divided by all the fragmented data in the data to be migrated is smaller, the times of distributing the processing procedure and calling the tcp protocol stack by the second processor are smaller, the time required by the second processor to complete migration of the data to be migrated is shorter, and thus the processing performance of the second processor is further improved.

The data migration method in the embodiment of the application can be applied to a network card device, the network card device can be deployed in a server, a CPU in the server is called a first processor, a memory in the server is called a first memory, the CPU, the memory and a logic chip are also deployed in the network card device deployed in the server, the CPU in the network card device is called a second processor, and the memory in the network card device is called a second memory.

Referring to fig. 3, a schematic diagram of a hardware structure of a server installed with a network card device according to an embodiment of the present application is provided. As shown in fig. 3, the server 300 includes a first processor 31, a first memory 32, and a network card device 33, and the network card device 33 includes a second processor 331, a logic chip 332, and a second memory 333. The first memory 32 may be a Double Data Rate (DDR), the second memory 333 may also be a DDR, and the logic chip 332 may be a Field Programmable Gate Array (FPGA).

In this embodiment of the application, data write operation for migrating data to be migrated in the first memory 32 to the network storage device through the network card device 33, and data read operation for migrating data to be migrated in the network storage device to the first memory 32 all need to participate in the first processor 31, the second processor 331, and the logic chip 332 together, and the second processor 331 may be divided into a logic driver module and an upper software module according to a system architecture thereof, where the logic driver module refers to a hardware logic driver located at a bottom layer in the second processor and interacting with the logic chip 332, and the upper software module refers to an upper application located in an operating system architecture and used for interacting with the network storage device.

Referring to fig. 4, a flowchart of a data migration method provided in the embodiment of the present application is applied to a network card device, and specifically may include the following steps:

s401, when a data migration instruction aiming at data to be migrated and sent by a first processor is received, dividing the data to be migrated into N fragmented data groups; the target fragmented data group of the N fragmented data groups includes M fragmented data, where N is a positive integer and M is a positive integer greater than 1.

When the data to be migrated stored in the first memory needs to be backed up to the network storage device, a trigger operation may be performed on the server side, so that the first processor generates a data migration instruction for the data to be migrated, where the data migration instruction is a data write instruction. The first processor sends a data write instruction for the data to be migrated to a logic chip of the network card device, and when the logic chip receives the data write instruction for the data to be migrated sent by the first processor, the data to be migrated is divided into N fragmented data groups, wherein N is a positive integer.

The data to be migrated is divided into at least one fragmented data group, where the fragmented data group is written into the network storage device together, and the data to be migrated includes N fragmented data groups, where each fragmented data group includes at least one fragmented data, and at least part of the fragmented data groups includes multiple fragmented data. A fragmented data group comprising a plurality of fragmented data is referred to as a target fragmented data group, where the target fragmented data group comprises M fragmented data, and M is a positive integer greater than 1.

When the data to be migrated stored in the network storage device needs to be migrated to the first memory, a trigger operation may be performed on the server side, so that the first processor generates a data migration instruction for the data to be migrated, where the data migration instruction is a data read instruction. The first processor sends a data reading instruction aiming at the data to be migrated to a logic chip of the network card device, and when the logic chip receives the data reading instruction aiming at the data to be migrated sent by the first processor, the data to be migrated is divided into N fragmented data groups, wherein N is a positive integer.

The data to be migrated is divided into at least one fragmented data group, so that the data to be migrated includes N fragmented data groups, each fragmented data group includes at least one fragmented data, and at least part of the fragmented data groups includes a plurality of fragmented data. A fragmented data group comprising a plurality of fragmented data is referred to as a target fragmented data group, where the target fragmented data group comprises M fragmented data, and M is a positive integer greater than 1.

Because the target fragmented data group comprises a plurality of fragmented data, the number N of fragmented data groups comprised by the data to be migrated is smaller than the number X of all fragmented data comprised by the data to be migrated.

For example, N is 1, the data to be migrated includes 10 pieces of fragmented data, and all the 10 pieces of fragmented data are divided into one fragmented data group, so that the data to be migrated includes 1 target fragmented data group, and the target fragmented data group includes 10 pieces of fragmented data.

Of course, it can be understood that the number N of the fragmented data groups included in the data to be migrated is not limited to 1, and may also be 2, 3, or the like. For example, the data to be migrated includes 10 pieces of fragmented data, the first 5 pieces of fragmented data are divided into a first fragmented data group, the last 5 pieces of fragmented data are divided into a second fragmented data group, both the first fragmented data group and the second fragmented data group may be referred to as target fragmented data groups, at this time, X is equal to 10, N is equal to 2, and M is equal to 5.

S402, generating a data migration request corresponding to each piece of piece data in the target piece data group.

In the embodiment of the application, the logic chip generates, for each target fragmented data group, a data migration request corresponding to each fragmented data in the target fragmented data group. That is, in the target fragmented data group, each fragmented data corresponds to one data migration request.

When the data migration instruction sent by the first processor is a data writing instruction, the data migration request is a data writing request; when the data migration instruction sent by the first processor is a data reading instruction, the data migration request is a data reading request.

S403, when a data migration request corresponding to first target fragmented data in a target fragmented data group is identified, applying for a storage space in a second memory for M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data; the first target fragment data is one fragment data in the target fragment data group.

In this embodiment of the application, after the logic chip generates the data migration request corresponding to each fragmented data in the target fragmented data group, when the logic driver module in the second processor identifies the data migration request corresponding to the first target fragmented data in the target fragmented data group, the logic driver module applies for the storage space in the second memory for the M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data group, so that the storage addresses of the M fragmented data in the target fragmented data group stored in the second memory are consecutive.

Specifically, the data migration request corresponding to the first target fragmented data includes length information of M fragmented data in the corresponding target fragmented data group, and the logic driver module actually applies for a storage space in the second memory for the M fragmented data in the target fragmented data group according to the length information in the data migration request corresponding to the first target fragmented data. The length information represents the size of the occupied space of the M pieces of sliced data in the target sliced data group.

The first target fragment data is any one fragment data in the target fragment data group. For example, when the data migration request is a data write request, the first target fragmented data may refer to a first fragmented data in the target fragmented data group that needs to be migrated from the first memory to the second memory; when the data migration request is a data read request, the first target fragmented data may refer to a first fragmented data in the target fragmented data group that needs to be migrated from the second memory to the first memory.

For example, the data to be migrated includes 10 pieces of fragmented data, and the 10 pieces of fragmented data are all divided into 1 piece of fragmented data group, that is, the data to be migrated includes 1 target piece of fragmented data group, and the target piece of fragmented data group includes 10 pieces of fragmented data, and when the logic driver module in the second processor identifies a data migration request corresponding to the first target piece of fragmented data in the target piece of fragmented data group, the logic driver module applies for a storage space in the second memory for the 10 pieces of fragmented data in the target piece of fragmented data group according to the size of an occupied space of the 10 pieces of fragmented data in the target piece of fragmented data group. Or, the data to be migrated includes 10 fragmented data, the 10 fragmented data are all divided into 2 fragmented data groups, the 2 fragmented data groups are target fragmented data groups, that is, the data to be migrated includes 2 target fragmented data groups, a first target fragmented data group includes 5 fragmented data, a second target fragmented data group also includes 5 fragmented data, when the logic driving module identifies a data migration request corresponding to first target fragmented data in the first target fragmented data group, the logic driving module applies for a storage space in a second memory for the 5 fragmented data in the first target fragmented data group according to an occupied space size of the 5 fragmented data in the first target fragmented data group; when the logic driving module identifies a data migration request corresponding to first target fragmented data in a second target fragmented data group, the logic driving module applies for a storage space in a second memory for the 5 fragmented data in the second target fragmented data group according to the size of an occupied space of the 5 fragmented data in the second target fragmented data group.

In an optional embodiment, when the logic chip generates a data migration request corresponding to each fragmented data in the target fragmented data group, a first identifier may be added to the data migration request corresponding to the first target fragmented data in the target fragmented data group, that is, the data migration request corresponding to the first target fragmented data in the target fragmented data group includes the first identifier. Therefore, when the logic driving module in the second processor identifies the data migration request carrying the first identifier in the target fragmented data group, it may be determined that the data migration request corresponding to the first target fragmented data in the target fragmented data group is identified, and then, the logic driving module applies for a storage space in the second memory for the M fragmented data in the target fragmented data group according to the data migration request carrying the first identifier.

S404, executing a data migration operation once, and migrating the M pieces of fragment data in the target fragment data group from the first storage unit to the second storage unit.

In this embodiment of the present application, after the logic driver module applies for a storage space of the second memory for M fragmented data in the target fragmented data group, the upper layer software module in the second processor may execute 1 time of data migration operation on the target fragmented data group, and migrate the M fragmented data in the target fragmented data group together from the first storage unit to the second storage unit. One of the first storage unit and the second storage unit is a second memory, and the other one is a network storage device.

Therefore, when the data to be migrated includes N fragmented data groups, the upper layer software module in the second processor performs N times of data migration operations, so as to migrate X fragmented data included in the data to be migrated from the first storage unit to the second storage unit N times.

For example, the data to be migrated includes 10 pieces of fragmented data, and the 10 pieces of fragmented data are all divided into 1 piece of fragmented data group, that is, the data to be migrated includes 1 target piece of fragmented data group, and the target piece of fragmented data group includes 10 pieces of fragmented data, and the upper layer software module executes 1 time of data migration operation, and can migrate the 10 pieces of fragmented data included in the data to be migrated from the first storage unit to the second storage unit at one time; or, the data to be migrated includes 10 pieces of fragmented data, the 10 pieces of fragmented data are all divided into 2 pieces of fragmented data groups, the 2 pieces of fragmented data groups are all target pieces of fragmented data groups, that is, the data to be migrated includes 2 target pieces of fragmented data groups, a first target piece of fragmented data group includes 5 pieces of fragmented data, a second target piece of fragmented data group also includes 5 pieces of fragmented data, the upper layer software module executes the data migration operation for the 1 st time, migrates 5 pieces of fragmented data in the first target piece of fragmented data group together from the first storage unit to the second storage unit, the upper layer software module executes the data migration operation for the 2 nd time, and migrates 5 pieces of fragmented data in the second target piece of fragmented data group together from the first storage unit to the second storage unit.

Since the storage addresses of the M fragmented data in the target fragmented data group stored in the second memory are consecutive, when the M fragmented data in the target fragmented data group is stored in the second memory, the M fragmented data in the target fragmented data group can be merged, and therefore, the merged target fragmented data group only needs to perform one data migration operation to migrate the M fragmented data in the target fragmented data group from the second memory to the network storage device or from the network storage device to the second memory.

The data migration operation of the data to be migrated is data writing operation or data reading operation. The data writing operation refers to that the data to be migrated stored in the first memory is migrated to the second memory and then is migrated from the second memory to the network storage device, that is, the data to be migrated in the first memory of the server is copied to the network storage device, at this moment, the first storage unit is the second memory, and the second storage unit is the network storage device; the data reading operation refers to that the data to be migrated stored in the network storage device is migrated to the second memory first and then is migrated from the second memory to the first memory, that is, the data to be migrated in the network storage device is copied to the first memory of the server, at this time, the first storage unit is the network storage device, and the second storage unit is the second memory. The network storage device may be a cloud disk server.

That is to say, when the data migration request is a data write request and the data migration operation is a data write operation, the upper layer software module in the second processor executes N times of data migration operations, reads out X pieces of fragment data included in the data to be migrated from the second memory in N times, and writes the X pieces of fragment data into the network storage device; when the data migration request is a data read request and the data migration operation is a data read operation, the upper layer software module in the second processor executes the data migration operation for N times, reads out and writes X pieces of fragment data included in the data to be migrated from the network storage device into the second memory in N times, wherein N is smaller than X.

In addition, when the data to be migrated is stored in the first memory, the storage addresses of the respective fragmented data in the data to be migrated in the first memory are often discontinuous, so that the M fragmented data in the target fragmented data group are sequentially migrated between the first memory and the second memory, that is, only one fragmented data in the target fragmented data group is migrated between the first memory and the second memory at a time. When the target fragmented data group includes M fragmented data, it is necessary to execute M migration operations between the first memory and the second memory, and migrate M fragmented data groups included in the target fragmented data group from the first memory to the second memory, or migrate M fragmented data groups included in the target fragmented data group from the second memory to the first memory.

In addition, the data to be migrated comprises N fragmented data groups, wherein one part of the fragmented data groups is a target fragmented data group comprising a plurality of fragmented data, and the other part of the fragmented data groups is a fragmented data group comprising 1 fragmented data; or, each of the N sliced data groups is a target sliced data group including a plurality of sliced data.

It should be noted that, in the above execution process, the data migration process of a target fragmented data group including multiple fragmented data is mainly described, and the steps from S402 to S404 are executed for each target fragmented data group; for a fragmented data group that only includes one fragmented data, the actual migration process is similar to the data migration process shown in fig. 1 or fig. 2, and will not be described again here.

The following describes a data migration process in the embodiment of the present application, with respect to data write operation and data read operation, respectively.

A first scenario, regarding to a data write operation of data to be migrated, with reference to fig. 5, is a flowchart of the data write operation of the data to be migrated provided in the embodiment of the present application, and specifically includes the following steps:

s501, when a data migration instruction aiming at data to be migrated and sent by a first processor is received, dividing the data to be migrated into N fragmented data groups; the target fragmented data group of the N fragmented data groups includes M fragmented data, where N is a positive integer and M is a positive integer greater than 1.

And S502, generating a data migration request corresponding to each piece of piece data in the target piece data group.

At this time, the data migration instruction is a data write instruction, the data migration request is a data write request, and the specific execution processes of S501 and S502 are similar to the processes of S401 and S402, and are not described herein again to avoid repetition.

S503, when a data migration request corresponding to a first target fragmented data in a target fragmented data group is identified, applying for a storage space in a second memory for M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data; the first target fragment data is one fragment data in the target fragment data group.

The specific implementation process of this step is similar to the process of S403, and is not described herein again to avoid repetition.

S504, according to the data migration requests corresponding to the M pieces of fragment data in the target fragment data group, sequentially migrating the M pieces of fragment data from the first memory to the second memory.

In this embodiment of the application, after the logic driver module applies for the M fragmented data in the target fragmented data group to the storage space of the second memory, the logic chip will respectively and sequentially migrate the M fragmented data in the target fragmented data group from the first memory to the second memory in sequence according to the data migration requests corresponding to the M fragmented data in the target fragmented data group.

When the data to be migrated includes N fragmented data groups, for a target fragmented data group including M fragmented data, the logic chip may sequentially migrate M fragmented data in the target fragmented data group from the first memory to the second memory M times.

For example, when the data to be migrated includes 1 target fragmented data group, where the target fragmented data group includes 10 fragmented data, the logic chip sequentially migrates 10 fragmented data groups in the target fragmented data group from the first memory to the second memory for 10 times; or the data to be migrated includes 2 target fragmented data groups, the first target fragmented data group includes 5 fragmented data, and the second target fragmented data group also includes 5 fragmented data, so that the logic chip may sequentially migrate 5 fragmented data groups in the first target fragmented data group from the first memory to the second memory in 5 times, and then sequentially migrate 5 fragmented data groups in the second target fragmented data group from the first memory to the second memory in 5 times.

S505, it is determined whether all the M pieces of fragment data have been migrated from the first memory to the second memory.

After sequentially migrating the M fragmented data in the target fragmented data group from the first memory to the second memory, the logic drive module may determine whether all the M fragmented data in the target fragmented data group have been migrated from the first memory to the second memory.

In an optional implementation manner, the last fragmented data in the target fragmented data group migrated from the first memory to the second memory is the second target fragmented data, and the logic chip may add a second identifier to the data migration request corresponding to the second target fragmented data in the target fragmented data group when generating the data migration request corresponding to each fragmented data in the target fragmented data group, that is, the data migration request corresponding to the second target fragmented data includes the second identifier. Therefore, the logic driving module in the second processor determines whether all M pieces of fragment data in the target fragment data group have been migrated from the first memory to the second memory by determining whether second target fragment data corresponding to the data migration request carrying the second identifier in the target fragment data group has been migrated from the first memory to the second memory.

When the logic driving module identifies that second target fragmented data corresponding to the data migration request carrying the second identifier has been migrated to the second memory, it may be determined that all M fragmented data in the target fragmented data group have been migrated from the first memory to the second memory; when the logic driving module does not recognize that the second target fragmented data corresponding to the data migration request carrying the second identifier has been migrated to the second memory, it may be determined that all M fragmented data in the target fragmented data group have not been migrated from the first memory to the second memory.

For example, when the data to be migrated includes 1 target fragmented data group, where the target fragmented data group includes 10 fragmented data, a second identifier may be added to a data migration request corresponding to the 10 th fragmented data in the target fragmented data group, and when the logic driver module identifies that a second target fragmented data (i.e., the 10 th fragmented data) corresponding to the data migration request carrying the second identifier has been migrated to the second memory, it may be determined that all fragmented data in the target fragmented data group has been migrated from the first memory to the second memory; or, the data to be migrated includes 2 target fragmented data groups, where the first target fragmented data group includes 5 fragmented data, the second target fragmented data group also includes 5 fragmented data, a second identifier may be added to a data migration request corresponding to the 5 th fragmented data in the first target fragmented data group, and a second identifier is added to a data migration request corresponding to the 5 th fragmented data in the second target fragmented data group, when the logic driving module recognizes that the second target fragmented data corresponding to the data migration request carrying the second identifier in the first target fragmented data group (i.e., the 5 th fragmented data in the first target fragmented data group) has been migrated to the second memory, it may be determined that all fragmented data in the first target fragmented data group stored in the first memory has been migrated to the second memory, and when the logic driving module recognizes that the data migration request pair carrying the second identifier in the second target fragmented data group has been migrated to the second memory When the corresponding second target fragmented data (i.e. the 5 th fragmented data in the second target fragmented data group) has been migrated to the second memory, it may be determined that all fragmented data in the second target fragmented data group stored in the first memory has been completely migrated to the second memory.

S506, when all the M pieces of fragment data are migrated from the first memory to the second memory, executing one data migration operation, and migrating the M pieces of fragment data in the target piece of fragment data from the second memory to the network storage device.

In this embodiment of the application, when the logic driver module in the second processor recognizes that all M pieces of fragmented data in the target fragmented data group have been migrated from the first memory to the second memory, the upper layer software module in the second processor may execute one data migration operation to migrate M pieces of fragmented data in the target fragmented data group together from the second memory to the network storage device.

When the logic driving module identifies a data migration request corresponding to a first target fragmented data in a target fragmented data group, it applies for a storage space in a second memory for M fragmented data in the target fragmented data group, so that after the M fragmented data in the target fragmented data group are sequentially migrated from the first memory to the second memory, the storage addresses of the M fragmented data in the target fragmented data group in the second memory are consecutive, thereby merging the M fragmented data in the target fragmented data group in the second memory, that is, splicing the M fragmented data in the target fragmented data group in the second memory.

Therefore, the upper software module in the second processor executes a data migration operation once for one merged target fragmented data group, and migrates M fragmented data in the target fragmented data group together from the second memory to the network storage device.

Fig. 6 is a specific interaction diagram for a server to implement data write operation of data to be migrated according to an embodiment of the present application, which may specifically include the following steps:

s601, the first processor loads a virtio driver and converts the data to be migrated stored in the first memory into a virtio data structure.

The virtual drive program refers to a drive program for driving a network card device, the drive program is run on the first processor, and after the virtual drive program is loaded, the first processor converts the data format of the data to be migrated, which is originally stored in the first memory, to obtain a virtual data structure, and the virtual data structure is still stored in the first memory.

As shown in fig. 7, the virtio data structure includes three parts, which are an available ring (pointer), a descriptor chain (descriptor), and stored individual fragment data (data buffer).

The field in the flags in the available ring indicates whether the data migration process is interrupted, the field in the idx in the available ring is used for indicating which of the currently processed pointers (rings) is in the data migration process, and the ring [ n ] in the available ring is used for storing the pointer, and the pointer points to the chain head of the descriptor in the descriptor chain.

The descriptor chain comprises a plurality of descriptors, wherein an addr field in each descriptor represents the storage address of the corresponding fragment data in the first memory, and the addr field represents the address of the fragment data which needs to be read from the first memory subsequently for data write operation; the len field in each descriptor indicates the length information of the fragment data corresponding to the len field, the flags field in each descriptor indicates whether a next descriptor exists, and the next field in each descriptor points to the next descriptor.

The data buffer represents each piece of fragment data in the data to be migrated which is actually stored in the first memory, an addr field of each descriptor in the descriptor chain points to one piece of fragment data in the data to be migrated, and the piece of fragment data corresponding to all the descriptors in the descriptor chain together form complete data to be migrated.

Because the storage addresses of the fragmented data included in the data to be migrated stored in the first memory are not continuous, each fragmented data corresponds to a first storage address in the first memory, and the first storage address corresponding to each fragmented data can be conveniently found from the virtio data structure by converting the data to be migrated stored in the first memory into the virtio data structure.

S602, after the first processor converts the data to be migrated stored in the first memory into a virtio data structure, the first processor sends a data write instruction for the data to be migrated to the logic chip.

S603, when the logic chip receives a data write instruction sent by the first processor, the logic chip divides the data to be migrated into N fragmented data groups, generates a data write request corresponding to each fragmented data in the target fragmented data group, adds a first identifier in the data write request of the first target fragmented data in the target fragmented data group, and adds a second identifier in the data write request of the second target fragmented data.

In the embodiment of the application, a logic chip divides data to be migrated into N fragmented data groups according to a preset rule, then reads a descriptor chain in a virtio data structure from a first memory in a Direct Memory Access (DMA) manner, and generates a data write request corresponding to each fragmented data in a target fragmented data group according to an addr field of each descriptor in the descriptor chain, where the data write request includes a first storage address of the corresponding fragmented data in the first memory, and in addition, a first identifier is added to the data write request of the first target fragmented data in the target fragmented data group, and a second identifier is added to the data write request of the second target fragmented data in the target fragmented data group.

Optionally, the first identifier may be configured to indicate first target fragmented data that needs to be migrated from the first memory to the second memory in the target fragmented data group, and the second identifier is configured to indicate second target fragmented data that is migrated from the first memory to the second memory in the target fragmented data group and is last in the target fragmented data group.

S604, the logic chip writes the data write request corresponding to each piece of piece data in the target piece of piece data group into the second memory.

S605, when the logic driving module in the second processor reads the data writing request with the first identifier from the second memory, applying for a storage space in the second memory for the M fragmented data in the target fragmented data group.

The logic driving module reads data writing requests corresponding to all the fragmented data in the target fragmented data group from the second memory, when the data writing requests with the first identification are read from the second memory, the data writing requests corresponding to the first target fragmented data in the target fragmented data group are identified, and the logic driving module applies for storage space of the second memory for the M fragmented data in the target fragmented data group according to the data writing requests with the first identification.

S606, the logic driver module sends a notification message to the logic chip that the M pieces of fragmented data in the target fragmented data group need to be migrated from the first memory to the second memory.

The notification message may include a second storage address of the storage space allocated in the second memory, where the second storage address refers to a first address of the second memory where the M pieces of fragmented data in the target fragmented data group are stored.

S607, the logic chip sequentially migrates each piece of partitioned data in the target partitioned data group from the first memory to the second memory.

S608, the logic chip sends a notification message that each piece of fragmented data in the target fragmented data group has been migrated to the second memory to the logic driver module.

And S609, when the logic driving module identifies that the second target fragmented data corresponding to the data writing request with the second identifier has been migrated to the second memory, sending a notification message that M fragmented data in the target fragmented data group need to be migrated from the second memory to the network storage device.

In fact, each time the logic chip migrates one piece of fragment data, a notification message that the piece of fragment data has been migrated to the second memory is sent to the logic driver module. Therefore, when the logic driver module reads the data write request with the second identifier in S605 and receives that the fragmented data corresponding to the data write request with the second identifier sent by the logic chip has been migrated to the second memory, the logic driver module determines that all the fragmented data in the target fragmented data group has been migrated from the first memory to the second memory, and then the logic driver module sends a notification message to the upper software module that all the fragmented data in the target fragmented data group needs to be migrated from the second memory to the network storage device.

Since the logic driving module allocates all the fragmented data in the target fragmented data group to the storage space of the second memory at one time according to the data write request with the first identifier, when the M fragmented data in the target fragmented data group are sequentially migrated from the first memory to the storage space of the second memory, the storage addresses of the fragmented data in the target fragmented data group stored in the second memory are consecutive, that is, after the first fragmented data in the target fragmented data group is stored in the second memory, the second fragmented data in the target fragmented data group is stored from the storage position after the first fragmented data, the third fragmented data in the target fragmented data group is stored from the storage position after the second fragmented data, and so on. Therefore, after all the fragmented data in the target fragmented data group is migrated from the first memory to the second memory, merging of all the fragmented data in the target fragmented data group is achieved in the second memory.

And S610, the upper software module in the second processor distributes the write operation of all the fragmented data in the target fragmented data group to different processing modules.

The upper layer software module distributes the write operation requests (i.e. the write operation requests for writing the fragmented data into the network storage device) of all the fragmented data (i.e. M fragmented data) in the target fragmented data group to different processing modules according to different positions where the corresponding fragmented data are subsequently stored in the network storage device.

S611, the upper layer software module migrates all the fragmented data in the target fragmented data group in the second memory to the network storage device together.

And the upper layer software module calls a tcp protocol stack of the linux kernel to transfer all the fragment data in the target fragment data group in the second memory to the network storage device together. When the target fragmented data group includes a plurality of fragmented data, S610 and S611 may be executed once, and all fragmented data included in the target fragmented data group is migrated from the second memory to the network storage device at one time, so as to reduce the distribution processing procedure of the upper layer software module and the number of times of calling the tcp protocol stack.

Actually, after the upper layer software module reads all the fragment data in the target fragment data group from the second memory, all the fragment data in the target fragment data group is encapsulated into one message, that is, all the fragment data in the target fragment data group is included in the message, and then the message is sent to the network storage device by the upper layer software module.

When all the fragmented data in the target fragmented data group is stored into the network storage device, the storage addresses of all the fragmented data in the target fragmented data group in the network storage device are consecutive.

And S612, when the upper layer software module receives the response message returned by the network storage device, the upper layer software module sends the response message returned by the network storage device to the logic driving module.

The response message is used to prompt that all the fragmented data in the target fragmented data group has been migrated to the network storage device.

S613, when the logic driver module receives a response message that the last fragmented data group in the data to be migrated has been written into the network storage device, the logic driver module sends a notification message to the logic chip that all fragmented data included in the data to be migrated has been migrated to the network storage device.

And S614, the logic chip writes the notification message that all the fragmented data included in the data to be migrated has been migrated to the network storage device into the first memory.

S615, when the first processor reads the notification message that all the fragment data in the first memory has been migrated to the network storage device, the first processor releases the resource.

The resource may refer to data to be migrated originally stored in the first memory, that is, the first processor clears the data to be migrated originally stored in the first memory to release a storage space of the first memory, and/or the first processor clears a virtio data structure in the first memory.

Therefore, it can be seen that, in the process of data writing operation, the upper layer software module and the data interacted between the upper layer software module and the network storage device in this embodiment are all the fragment data in the target fragment data group (i.e., M fragment data, where M is a positive integer greater than 1), whereas in the related art shown in fig. 1, the upper layer software module and the data interacted between the upper layer software module and the network storage device are all one fragment data.

For example, in the related art shown in fig. 1, the upper layer software module distributes the write operation request corresponding to each piece of fragmented data to different processing modules, and the upper layer software module writes one piece of fragmented data into the network storage device at a time; the upper layer software module in this embodiment of the present application distributes the write operation requests of all the fragmented data in the target fragmented data group to different processing modules, and the upper layer software module writes all the fragmented data in the target fragmented data group into the network storage device at a time.

When the number of the fragmented data written into the network storage device increases each time, the number of times that all the fragmented data in the data to be migrated are written into the network storage device is reduced, so that the number of times that the upper layer software module processes the data to be migrated is reduced, the time required by the upper layer software module to process the data to be migrated is reduced, and the processing performance of the second processor is improved. In addition, in the embodiment of the present application, the merging process of all the fragmented data in the target fragmented data group needs to be implemented by a logic chip and a logic driver module, and the processing logic of the upper layer software module does not need to be changed.

In a second scenario, referring to fig. 8, for a data reading operation of data to be migrated, a flowchart of the data reading operation of the data to be migrated provided in the embodiment of the present application may specifically include the following steps:

s801, when a data migration instruction aiming at data to be migrated and sent by a first processor is received, dividing the data to be migrated into N fragmented data groups; the target fragmented data group of the N fragmented data groups includes M fragmented data, where N is a positive integer and M is a positive integer greater than 1.

S802, generating a data migration request corresponding to each piece of piece data in the target piece data group.

At this time, the data migration instruction is a data read instruction, the data migration request is a data read request, and the specific execution processes of S801 and S802 are similar to those of S401 and S402 described above, and are not described herein again to avoid repetition.

S803, when a data migration request corresponding to a first target fragmented data in the target fragmented data group is identified, applying for a storage space in a second memory for M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data; the first target fragment data is one fragment data in the target fragment data group.

S804, whether the data migration requests corresponding to the M pieces of fragmented data in the target fragmented data group are all generated is determined.

In this embodiment of the application, after the logic chip generates the data migration request corresponding to each fragmented data in the target fragmented data group, the logic driver module may identify whether all the data migration requests corresponding to M fragmented data in the target fragmented data group have been generated.

In an optional embodiment, the last data migration request generated in the target fragmented data group includes a third identifier, and the logic chip may add the third identifier to the last data migration request generated in the target fragmented data group when generating the data migration request corresponding to each fragmented data in the target fragmented data group. Therefore, the logic driving module determines whether all data migration requests corresponding to all fragmented data in the target fragmented data group have been generated by detecting whether a data migration request carrying the third identifier is identified, that is, determining whether the data migration request carrying the third identifier in the target fragmented data group has been generated.

Optionally, when generating each data migration request in the target fragmented data group, the data migration requests of each fragmented data may be sequentially generated in sequence according to the actual merging order of the M fragmented data in the target fragmented data group. For example, the data migration request of the first fragmented data in the target fragmented data group is generated first, the data migration request of the second fragmented data in the target fragmented data group is generated second, and so on, the mth data migration request of the mth fragmented data in the target fragmented data group is generated. At this time, the third identifier is added to the data migration request of the mth shard data.

When the logic driving module identifies a data migration request carrying a third identifier, determining that all data migration requests corresponding to all fragmented data in the target fragmented data group are generated; and when the logic driving module does not recognize the data migration request carrying the third identifier, determining that the data migration requests corresponding to all the fragmented data in the target fragmented data group are not completely generated.

And S805, when all the shard data groups are generated, executing one data migration operation, and migrating the M shard data in the target shard data group to a second memory from the network storage device.

In this embodiment of the application, when the logic driver module in the second processor recognizes that the data migration requests corresponding to all the fragmented data in the target fragmented data group are all generated, the upper layer software module in the second processor executes a data migration operation once to migrate all the fragmented data in the target fragmented data group stored in the network storage device to the second memory together.

When the logic driving module identifies the data migration request corresponding to the first target fragmented data in the target fragmented data group, it applies for a storage space in the second memory for the M fragmented data in the target fragmented data group, so that the upper layer software module can read the M fragmented data in the target fragmented data group from the network storage device at one time and write the M fragmented data in the target fragmented data group into the second memory together, thereby implementing merging of all fragmented data in the target fragmented data group in the second memory.

Therefore, the upper layer software module in the second processor executes a data migration operation once for each target fragmented data group to be merged, and migrates all fragmented data in the target fragmented data group stored in the network storage device to the second memory together.

And S806, sequentially migrating the M pieces of fragment data from the second memory to the first memory according to the data migration requests corresponding to the M pieces of fragment data in the target fragment data group.

In this embodiment of the application, after the upper layer software module migrates all the fragmented data in the target fragmented data group stored in the network storage device to the second memory, the logic chip sequentially migrates M fragmented data in the target fragmented data group stored in the second memory to the first memory according to the data migration request corresponding to each fragmented data in the target fragmented data group.

Fig. 9 is a specific interaction diagram for a server to implement a data reading operation of data to be migrated according to an embodiment of the present application, which may specifically include the following steps:

s901, the first processor loads a virtio driver, and generates a virtio data structure in the first memory.

When data to be migrated needs to be read from the network storage device, the first processor loads a virtio driver, allocates a first storage address for each piece of data to be migrated when the piece of data needs to be stored in the first memory subsequently, and generates a virtio data structure.

The virtio data structure during a data read operation is different from the virtio data structure for a data write operation in that there is no fragmentation data in the virtio data structure for a while during a data read operation.

S902, the first processor sends a data reading instruction aiming at the data to be migrated to the logic chip.

S903, when the logic chip receives a data read instruction sent by the first processor, divides the data to be migrated into N fragmented data groups, generates a data read request corresponding to each fragmented data in the target fragmented data group, adds a first identifier to the data read request of the first target fragmented data in the target fragmented data group, and adds a third identifier to the last generated data read request.

And S904, the logic chip writes the data read request corresponding to each piece of fragment data in the target fragment data group into the second memory.

S905, when the logic driving module reads a data reading request with the first identifier from the second memory, applying for a storage space in the second memory for the M fragmented data in the target fragmented data group.

S906, when the logic driving module reads a data reading request with a third identifier from the second memory, a notification message that M pieces of fragment data in the target fragment data group need to be migrated from the network storage device to the second memory is sent to the upper layer software module.

When the logic driving module reads a data reading request with a third identifier from the second memory, it is determined that all data reading requests corresponding to all fragmented data in the target fragmented data group are completely generated, and at this time, the logic driving module sends a notification message to the upper software module that all fragmented data in the target fragmented data group need to be migrated from the network storage device to the second memory.

And S907, the upper layer software module distributes the read operation of all the fragment data in the target fragment data group to different processing modules.

The upper layer software module distributes the read operation requests of all the fragmented data in the target fragmented data group (i.e. the read operation requests for reading fragmented data from the network storage device) to different processing modules according to different storage positions of the corresponding fragmented data in the network storage device.

And S908, the upper software module migrates all the fragment data in the target fragment data group in the network storage device to the second memory.

And the upper layer software module calls a tcp protocol stack of the linux kernel to transfer all the fragment data in the target fragment data group in the network storage device to the second memory together. When the target fragmented data group includes a plurality of fragmented data, S907 and S908 may be executed once, and all fragmented data included in the target fragmented data group is migrated from the network storage device to the second memory at one time, so as to reduce the distribution processing procedure of the upper layer software module in the second processor and the number of times of calling the tcp protocol stack.

In fact, the network storage device encapsulates all the fragmented data in the target fragmented data group into one message, that is, all the fragmented data in the target fragmented data group is included in the message, and then, the network storage device sends the message to the upper layer software module.

When all the fragmented data in the target fragmented data group are stored in the network storage device, the storage addresses of all the fragmented data in the target fragmented data group in the network storage device are continuous, so that all the fragmented data in the target fragmented data group can be read out at one time; when the storage space allocated in the second memory can contain all the fragmented data in the target fragmented data group, the upper layer software module can write all the fragmented data in the target fragmented data group read from the network storage device into the storage space of the second memory at one time.

And S909, the upper layer software module sends a response message returned by the network storage device to the logic drive module.

The response message is used to prompt the network storage device that all the fragmented data in the target fragmented data group has been sent to the upper layer software module.

S910, the logic driver module sends a notification message to the logic chip, where each piece of fragment data in the target fragment data group needs to be migrated from the second memory to the first memory.

And S911, the logic chip migrates each piece of fragment data in the target piece of fragment data group from the second memory to the first memory in sequence.

S912, the logic chip sends a notification message that each piece data in the target piece data group has been migrated to the first memory to the logic drive module

S913, when the logic driver module determines that the notification message that the last piece of data in the data to be migrated has been migrated to the first memory is received, the logic driver module sends, to the logic chip, a notification message that all piece of data included in the data to be migrated has been migrated to the first memory.

S914, the logic chip writes a notification message that all the fragmented data included in the data to be migrated has been migrated to the first memory into the first memory.

S915, when the first processor reads the notification message that all the fragment data in the first memory has been migrated to the first memory, the first processor releases the resource.

The resource may refer to a virtio data structure in the first memory, and the releasing the resource refers to clearing the virtio data structure in the first memory.

Therefore, it can be seen that, in the process of data reading operation, the upper layer software module and the data interacted between the upper layer software module and the network storage device in this embodiment are all the fragment data in the target fragment data group (i.e., M fragment data, where M is a positive integer greater than 1), whereas in the related art shown in fig. 2, the upper layer software module and the data interacted between the upper layer software module and the network storage device are all one fragment data.

For example, in the related art shown in fig. 2, the upper layer software module distributes the read operation request corresponding to each piece of fragmented data to different processing modules, and the upper layer software module migrates one piece of fragmented data from the network storage device to the second memory at a time; the upper layer software module in this embodiment of the present application distributes read operation requests of all fragmented data in the target fragmented data group to different processing modules, and the upper layer software module migrates all fragmented data in the target fragmented data group together from the network storage device to the second memory each time.

When the number of the fragmented data read from the network storage device increases each time, the number of times of reading all the fragmented data in the data to be migrated from the network storage device is reduced, so that the number of times of processing the data to be migrated by the upper software module is reduced, the time required by the upper software module to process the data to be migrated is reduced, and the processing performance of the second processor is improved. In addition, in the embodiment of the present application, the merging process of all the fragmented data in the target fragmented data group needs to be implemented by a logic chip and a logic driver module, and the processing logic of the upper layer software module does not need to be changed.

After testing, as a result of comparing performance test data in actual products of the present application and related technologies, table one is a performance test data table of a second processor in the network card device in the related technologies, table two is a performance test data table of a second processor in the network card device in the embodiment of the present application, and the performance test data table shown in table two is to divide data to be migrated into one target fragmented data group, that is, to execute one data migration operation, and to migrate all fragmented data included in the data to be migrated, which is stored in the network storage device, to a second memory at one time.

Watch 1

Watch two

It can be seen that the number of read/write times per second (I/O per second, IOPS) and BW (bandwidth) in the embodiment of the present application are both greater than the IOPS and BW values in the related art, so that the throughput rate of the embodiment of the present application is significantly improved compared with the related art; moreover, latency in the embodiments of the present application is smaller than latency in the related art, that is, latency in the embodiments of the present application is significantly reduced compared to the related art. Therefore, the data migration method provided by the embodiment of the application can improve the performance of the second processor compared with the related art.

The embodiment of the application further provides a network card device, wherein the network card device is deployed in a server, the server further comprises a first processor and a first memory, and the network card device comprises a second processor, a logic chip and a second memory; the logic chip is used for dividing the data to be migrated into N fragmented data groups when receiving a data migration instruction aiming at the data to be migrated and sent by the first processor; generating a data migration request corresponding to each fragmented data in the target fragmented data group; a target fragmented data group in the N fragmented data groups comprises M fragmented data, wherein N is a positive integer, and M is a positive integer larger than 1; the second processor is used for applying for a storage space in a second memory for the M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data when the data migration request corresponding to the first target fragmented data in the target fragmented data group is identified; executing one data migration operation, and migrating the M piece data in the target piece data group from the first storage unit to the second storage unit; the first target fragment data is one fragment data in the target fragment data group; when the data migration operation is a data write operation, the first storage unit is a second memory, and the second storage unit is a network storage device; when the data migration operation is a data reading operation, the first storage unit is a network storage device, and the second storage unit is a second memory; m pieces of fragment data in the target fragment data group are sequentially migrated between the first memory and the second memory.

Optionally, N is 1.

The network card device of the embodiment of the present application can be correspondingly used to execute the steps executed in the foregoing method embodiments, and the implementation principle and technical effect are similar, which are not described herein again.

The embodiment of the application also provides a server, which comprises a first processor, a first memory and the network card device.

As shown in fig. 10, the data migration system includes a network storage device 400 and a server 300, where the network storage device 400 is in communication connection with the server 300, and specifically, the network storage device 400 is in communication connection with a network card device 33 in the server 300.

For the specific structure of the network card device and the process of implementing data migration, reference may be made to the descriptions corresponding to fig. 3 to 9, which are not described herein again.

Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above embodiments are provided to explain the purpose, technical solutions and advantages of the present application in further detail, and it should be understood that the above embodiments are merely illustrative of the present application and are not intended to limit the scope of the present application, and any modifications, equivalent substitutions, improvements and the like made on the basis of the technical solutions of the present application should be included in the scope of the present application.

Claims

1. A data migration method is characterized in that the method is applied to a network card device, the network card device is deployed in a server, the server further comprises a first processor and a first memory, and the network card device comprises a second memory; the method comprises the following steps:

when a data migration instruction aiming at data to be migrated and sent by the first processor is received, dividing the data to be migrated into N fragmented data groups; a target fragmented data group in the N fragmented data groups comprises M fragmented data, wherein N is a positive integer and M is a positive integer larger than 1;

generating a data migration request corresponding to each fragmented data in the target fragmented data group;

when a data migration request corresponding to first target fragmented data in the target fragmented data group is identified, applying for a storage space in the second memory for the M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data; the first target fragment data is one fragment data in the target fragment data group;

executing one data migration operation, and migrating the M pieces of fragmented data in the target fragmented data group from a first storage unit to a second storage unit;

when the data migration operation is a data write operation, the first storage unit is the second memory, and the second storage unit is a network storage device; when the data migration operation is a data read operation, the first storage unit is the network storage device, and the second storage unit is the second memory; the M pieces of fragmented data in the target fragmented data group are sequentially migrated between the first memory and the second memory.

2. The method according to claim 1, wherein the data migration request corresponding to the first target fragmented data includes a first identifier, and applying for a storage space in the second memory for the M fragmented data in the target fragmented data group according to the data migration request corresponding to the first target fragmented data when the data migration request corresponding to the first target fragmented data in the target fragmented data group is identified includes:

when a data migration request carrying the first identifier in the target fragmented data group is identified, applying for a storage space in the second memory for the M fragmented data in the target fragmented data group according to the data migration request carrying the first identifier.

3. The method of claim 1, wherein the data migration operation is a data write operation; before the performing a data migration operation and migrating the M pieces of sharded data in the target sharded data group from the first storage unit to the second storage unit, the method further includes:

sequentially migrating the M pieces of fragmented data from the first memory to the second memory according to data migration requests corresponding to the M pieces of fragmented data in the target fragmented data group;

determining whether all the M pieces of fragmented data are migrated from the first memory to the second memory;

the executing a data migration operation to migrate the M pieces of fragmented data in the target fragmented data group from the first storage unit to the second storage unit includes:

when all the M pieces of fragment data are migrated from the first memory to the second memory, executing a data migration operation once, and migrating the M pieces of fragment data in the target piece of fragment data from the second memory to the network storage device.

4. The method according to claim 3, wherein the last fragmented data in the target fragmented data group migrated from the first memory to the second memory is a second target fragmented data, and a data migration request corresponding to the second target fragmented data includes a second identifier;

the determining whether all of the M pieces of fragmented data have been migrated from the first memory to the second memory includes:

determining whether the second target fragmented data corresponding to the data migration request carrying the second identifier in the target fragmented data group has been migrated from the first memory to the second memory.

5. The method according to claim 1, wherein the data migration operation is a data read operation, and the performing a data migration operation to migrate the M pieces of fragmented data in the target fragmented data group from a first storage unit to a second storage unit includes:

determining whether all data migration requests corresponding to the M fragmented data in the target fragmented data group are generated;

when all the fragmented data groups are generated, executing one data migration operation, and migrating the M fragmented data in the target fragmented data group from the network storage device to the second memory;

after the executing of the one-time data migration operation and the migrating the M pieces of sharded data in the target sharded data group from the first storage unit to the second storage unit, the method further includes:

and sequentially migrating the M pieces of fragment data from the second memory to the first memory according to data migration requests corresponding to the M pieces of fragment data in the target fragment data group.

6. The method of claim 5, wherein a last data migration request generated in the target fragmented data group includes a third identifier; the determining whether the data migration requests corresponding to the M sharded data in the target sharded data group are all generated includes:

and determining whether the data migration request carrying the third identifier in the target fragmented data group is generated.

7. The method of any one of claims 1 to 6, wherein N is 1.

8. The network card device is characterized in that the network card device is deployed in a server, the server further comprises a first processor and a first memory, and the network card device comprises a second processor, a logic chip and a second memory;

the logic chip is configured to, when a data migration instruction for data to be migrated that is sent by the first processor is received, divide the data to be migrated into N fragmented data groups, where a target fragmented data group in the N fragmented data groups includes M fragmented data, N is a positive integer, and M is a positive integer greater than 1; generating a data migration request corresponding to each fragmented data in the target fragmented data group;

the second processor is configured to apply for a storage space in the second memory for the M fragmented data in the target fragmented data group according to a data migration request corresponding to the first target fragmented data when a data migration request corresponding to the first target fragmented data in the target fragmented data group is identified; executing a data migration operation once, and migrating the M pieces of fragmented data in the target fragmented data group from a first storage unit to a second storage unit;

the first target fragment data is one fragment data in the target fragment data group; when the data migration operation is a data write operation, the first storage unit is the second memory, and the second storage unit is a network storage device; when the data migration operation is a data read operation, the first storage unit is the network storage device, and the second storage unit is the second memory; the M pieces of fragmented data in the target fragmented data group are sequentially migrated between the first memory and the second memory.

9. The network card device of claim 8, wherein the data migration request corresponding to the first target sharded data includes a first identifier;

the second processor is specifically configured to, when recognizing that the data migration request carrying the first identifier in the target fragmented data group is received, apply for a storage space in the second memory for the M fragmented data in the target fragmented data group according to the data migration request carrying the first identifier.

10. The network card device of claim 8, wherein the data migration operation is a data write operation;

the logic chip is further configured to sequentially migrate the M fragmented data from the first memory to the second memory according to data migration requests corresponding to the M fragmented data in the target fragmented data group;

the second processor is further configured to determine whether all the M pieces of fragmented data have been migrated from the first memory to the second memory; and when all the M pieces of fragment data are migrated from the first memory to the second memory, executing a data migration operation once, and migrating the M pieces of fragment data in the target piece of fragment data from the second memory to the network storage device.

11. The network card device according to claim 10, wherein the last fragmented data in the target fragmented data group migrated from the first memory to the second memory is second target fragmented data, and a data migration request corresponding to the second target fragmented data includes a second identifier;

the second processor is specifically configured to determine whether the second target fragmented data corresponding to the data migration request carrying the second identifier in the target fragmented data group has been migrated from the first memory to the second memory.

12. The network card device of claim 8, wherein the data migration operation is a data read operation;

the second processor is specifically configured to determine whether all data migration requests corresponding to the M fragmented data in the target fragmented data group have been generated; when all the fragmented data groups are generated, executing one data migration operation, and migrating the M fragmented data in the target fragmented data group from the network storage device to the second memory;

the logic chip is further configured to sequentially migrate the M fragmented data from the second memory to the first memory according to data migration requests corresponding to the M fragmented data in the target fragmented data group.

13. The network card device of claim 12, wherein the last data migration request generated in the target fragmented data group includes a third identifier;

the second processor is specifically configured to determine whether a data migration request carrying the third identifier in the target fragmented data group is generated.

14. The network card device of any one of claims 8-13, wherein N is 1.

15. A server, comprising a first processor, a first memory, and the network card device according to any one of claims 8 to 14.

16. A data migration system comprising a network storage device and a server according to claim 15, said network storage device being communicatively coupled to said server.