CN114465770A - Data processing method and related device - Google Patents
Data processing method and related device Download PDFInfo
- Publication number
- CN114465770A CN114465770A CN202111643559.2A CN202111643559A CN114465770A CN 114465770 A CN114465770 A CN 114465770A CN 202111643559 A CN202111643559 A CN 202111643559A CN 114465770 A CN114465770 A CN 114465770A
- Authority
- CN
- China
- Prior art keywords
- target
- indication information
- data processing
- storage node
- network card
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 54
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000013507 mapping Methods 0.000 claims abstract description 21
- 238000004590 computer program Methods 0.000 claims description 22
- 238000004364 calculation method Methods 0.000 claims description 21
- 238000004806 packaging method and process Methods 0.000 claims description 5
- 239000002699 waste material Substances 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 14
- 230000006870 function Effects 0.000 description 9
- 238000007906 compression Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 230000006835 compression Effects 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000004075 alteration Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000011232 storage material Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
- H04L63/123—Applying verification of the received information received data contents, e.g. message integrity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer And Data Communications (AREA)
Abstract
The application relates to the technical field of storage, in particular to a data processing method and a related device, which are used for improving the data processing efficiency and reducing the waste of computing resources, and the method comprises the following steps: after the intelligent network card receives the data processing request, if the to-be-processed indication information contains a specified protocol field, extracting target indication information from the to-be-processed indication information based on a target protocol format; obtaining a target hash value of the target indication information based on a preset hash algorithm, and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node; and sending the target indication information to the target storage node based on the routing information of the target storage node. Therefore, the user mode and the kernel mode do not need to be switched, the data processing efficiency is improved, the waste of computing resources is reduced, and the performance of high bandwidth and low time delay is provided by utilizing the computing power of the intelligent network card.
Description
Technical Field
The present application relates to the field of storage technologies, and in particular, to a data processing method and a related apparatus.
Background
Small Computer System Interface (iSCSI) is a common block storage access protocol. In order to support the block storage service based on the iSCSI protocol, an independent physical server is usually deployed, where an iSCSI gateway service and a block storage client are deployed, the iSCSI gateway service is used to support a remote mount service of a block device, and the block storage client is used to forward a data processing request to a storage node.
However, since the bottom layer protocol of the iSCSI gateway service operates in the kernel mode and the upper layer protocol and the middle layer protocol operate in the user mode, data transmission needs to pass through the user mode and the kernel mode, which results in low data Processing efficiency and low bandwidth utilization rate.
Disclosure of Invention
The application provides a data processing method and device, which are used for improving data processing efficiency and reducing waste of computing resources.
In a first aspect, an embodiment of the present application provides a data processing method applied to an intelligent network card, including:
receiving a data processing request, wherein the data processing request carries indication information to be processed;
if the to-be-processed indicating information contains a specified protocol field, extracting target indicating information from the to-be-processed indicating information based on a target protocol format;
obtaining a target hash value of the target indication information based on a preset hash algorithm, and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node;
and sending the target indication information to the target storage node based on the routing information of the target storage node so as to enable the target storage node to perform data processing based on the target indication information.
As a possible implementation manner, the indication information to be processed includes a destination address;
after the receiving the data processing request, before determining that the to-be-processed indication information includes the specified protocol field, the method further includes:
determining a target network card queue corresponding to the destination address based on a mapping relation between the address and the network card queue, and storing the data processing request in the target network card queue;
and acquiring the data processing request from the target network card queue.
As a possible implementation manner, the obtaining a target hash value of the target indication information based on a preset hash algorithm includes:
splitting the target indication information according to a preset block storage size to obtain at least one data block;
and performing hash calculation on the at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
As a possible implementation manner, after extracting the target indication information from the indication information to be processed based on the target protocol format, the method further includes at least one of the following operations:
encrypting the target indication information by adopting a preset encryption algorithm to obtain processed indication information; alternatively, the first and second electrodes may be,
and compressing the target indication information by adopting a preset encryption algorithm to obtain the processed indication information.
As a possible implementation manner, the obtaining a target hash value of the target indication information based on a preset hash algorithm includes:
and performing hash calculation on the processed indication information by adopting a preset hash algorithm to obtain the processed indication information, and taking the target hash value as the target hash value of the target indication information.
As a possible implementation manner, the sending the target indication information to the target storage node based on the routing information of the target storage node includes:
based on a specified protocol format, packaging the target indication information to obtain packaged indication information;
and sending the encapsulated indication information to the target storage node based on the routing information of the target storage node.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including:
the device comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving a data processing request which carries indication information to be processed;
the protocol processing unit is used for extracting target indication information from the indication information to be processed based on a target protocol format if the indication information to be processed contains a specified protocol field;
the route processing unit is used for obtaining a target hash value of the target indication information based on a preset hash algorithm and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node;
and the sending unit is used for sending the target indication information to the target storage node based on the routing information of the target storage node so as to enable the target storage node to perform data processing based on the target indication information.
As a possible implementation manner, the indication information to be processed includes a destination address, and the protocol processing unit is further configured to:
determining a target network card queue corresponding to the destination address based on a mapping relation between the address and the network card queue, and storing the data processing request in the target network card queue;
and acquiring the data processing request from the target network card queue.
As a possible implementation manner, when the target hash value of the target indication information is obtained based on a preset hash algorithm, the route processing unit is specifically configured to:
splitting the target indication information according to a preset block storage size to obtain at least one data block;
and performing hash calculation on the at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
As a possible implementation manner, after extracting the target indication information from the indication information to be processed based on the target protocol format, the route processing unit is further configured to perform at least one of the following operations:
encrypting the target indication information by adopting a preset encryption algorithm to obtain processed indication information; alternatively, the first and second electrodes may be,
and compressing the target indication information by adopting a preset encryption algorithm to obtain the processed indication information.
As a possible implementation manner, when the target hash value of the target indication information is obtained based on a preset hash algorithm, the route processing unit is specifically configured to:
and performing hash calculation on the processed indication information by adopting a preset hash algorithm to obtain the processed indication information, and taking the target hash value as the target hash value of the target indication information.
As a possible implementation manner, when the target indication information is sent to the target storage node based on the routing information of the target storage node, the sending unit is specifically configured to:
based on a specified protocol format, packaging the target indication information to obtain packaged indication information;
and sending the encapsulated indication information to the target storage node based on the routing information of the target storage node.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, where the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the data processing method.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program, when the computer program runs on an electronic device, the computer program is configured to enable the electronic device to execute the steps of the data processing method.
In a fifth aspect, the present application provides a computer program product, which includes a computer program, where the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the steps of the data processing method.
To sum up, in the embodiment of the present application, after the intelligent network card receives the data processing request, if the to-be-processed indication information includes the specified protocol field, the target indication information is extracted from the to-be-processed indication information based on the target protocol format; obtaining a target hash value of the target indication information based on a preset hash algorithm, and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node; and sending the target indication information to the target storage node based on the routing information of the target storage node so that the target storage node performs data processing based on the target indication information.
Therefore, the iSCSI gateway service and the block storage client are unloaded to the intelligent network card, on one hand, user mode and kernel mode switching is not needed, so that the data processing efficiency is improved, the waste of computing resources is reduced, and on the other hand, the performance of high bandwidth and low time delay is provided by utilizing the computing power of the intelligent network card.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic diagram of an application scenario provided in an embodiment of the present application;
fig. 2 is a schematic structural diagram of indication information to be processed provided in an embodiment of the present application;
FIG. 3 is a flowchart illustrating a data processing method provided in an embodiment of the present application
FIG. 4 is a logic diagram of a data processing method provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of a data processing apparatus provided in an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments, but not all embodiments, of the technical solutions of the present application. All other embodiments obtained by a person skilled in the art without any inventive step based on the embodiments described in the present application are within the scope of the protection of the present application.
In order to support the block storage service based on the iSCSI protocol, in the related art, it is generally necessary to deploy an independent physical server, where an iSCSI gateway service and a block storage client are deployed in the physical server, the iSCSI gateway service is used to support a block device remote mount service, and the block storage client is used to forward a data processing request to a storage node.
However, since the bottom layer protocol of the iSCSI gateway service operates in the kernel mode and the upper layer protocol and the middle layer protocol operate in the user mode, data transmission needs to pass through the user mode and the kernel mode, which results in low data processing efficiency and low bandwidth utilization rate.
In order to improve data processing efficiency and reduce waste of computing resources, in the embodiment of the application, after the intelligent network card receives a data processing request, if the to-be-processed indication information contains a specified protocol field, extracting target indication information from the to-be-processed indication information based on a target protocol format; obtaining a target hash value of the target indication information based on a preset hash algorithm, and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node; and sending the target indication information to the target storage node based on the routing information of the target storage node so that the target storage node performs data processing based on the target indication information.
Therefore, the iSCSI gateway service and the block storage client are unloaded to the intelligent network card, on one hand, user mode and kernel mode switching is not needed, so that the data processing efficiency is improved, the waste of computing resources is reduced, and on the other hand, the performance of high bandwidth and low time delay is provided by utilizing the computing power of the intelligent network card.
The preferred embodiments of the present application will be described in conjunction with the drawings of the specification, it should be understood that the preferred embodiments described herein are for purposes of illustration and explanation only and are not intended to limit the present application, and features of the embodiments and examples of the present application may be combined with each other without conflict.
Fig. 1 is a schematic diagram of a possible application scenario applicable to the embodiment of the present application. The application scenario at least includes a terminal device 101, a gateway node 102, and a storage system 103. The number of the terminal devices 101 is not particularly limited in the present application.
The terminal device 101 is configured to send a data processing request to the gateway node 102 in response to a data processing operation triggered by a user. The terminal device 101 may be a mobile phone, a computer, an intelligent voice interaction device, a vehicle-mounted terminal, etc., but is not limited thereto.
An intelligent network card 104 is deployed in the gateway node 102. The intelligent network card 104 is configured to receive a data processing request from a terminal device, acquire target indication information based on to-be-processed indication information included in the data processing request, and forward the target indication information to the storage system 103. The smart network card 104 is a smart terminal for providing a remote block storage service.
Two functional modules may be integrated into the intelligent network card 104: an iSCSI module and a block storage client. The iSCSI module is used for processing SCSI instructions and for remotely mounting iSCSI disks, and a network security detection subsystem can be integrated in the iSCSI module. The block storage client is used for receiving the iSCSI instruction, storing the iSCSI instruction according to blocks, and forwarding the iSCSI instruction to the storage cluster through private routing information, and an encryption subsystem, a decryption subsystem, a compression subsystem and the like can be integrated in the block storage client.
The storage system 103 may include one or more storage nodes such as a storage node a and a storage node b, and when a certain storage node receives the target indication information from the intelligent network card 104, data processing is performed according to the target indication information.
In some embodiments, the storage system may adopt a distributed cloud storage system, cloud storage (cloud storage) is a new concept that is extended and developed on the cloud computing concept, and a distributed cloud storage system (hereinafter, referred to as a storage system) refers to a storage system that integrates a large number of storage nodes of different types in a network through functions of cluster application, a grid technology, a distributed storage file system, and the like to cooperatively work through application software or an application interface, and that provides a data storage function and a service access function to the outside.
At present, a storage method of a storage system is as follows: logical volumes are created, and when created, each logical volume is allocated physical storage space, which may be the disk composition of a certain storage device or of several storage devices. The client stores data on a certain logical volume, that is, the data is stored on a file system, the file system divides the data into a plurality of parts, each part is an object, the object not only contains the data but also contains additional information such as data Identification (ID), the file system writes each object into a physical storage space of the logical volume, and the file system records storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.
The process of allocating physical storage space for the logical volume by the storage system specifically includes: physical storage space is divided in advance into stripes according to a group of capacity measures of objects stored in a logical volume (the measures often have a large margin with respect to the capacity of the actual objects to be stored) and Redundant Array of Independent Disks (RAID), and one logical volume can be understood as one stripe, thereby allocating physical storage space to the logical volume.
Fig. 2 is a schematic flow chart of a data processing method provided in the embodiment of the present application, where the method is applied to an intelligent network card, and the specific flow is as follows:
s201, receiving a data processing request, wherein the data processing request carries instruction information to be processed.
S202, if the to-be-processed indicating information contains the specified protocol field, extracting the target indicating information from the to-be-processed indicating information based on the target protocol format.
It should be noted that, in the embodiment of the present application, the target protocol format refers to a protocol format of an iSCSI protocol. The protocol field is designated as iSCSI header information (iSCSI header).
Referring to FIG. 3, iSCSI is an industry standard for transporting SCSI block commands over IP networks using the Transmission Control Protocol/Internet Protocol (TCP/IP). in the iSCSI protocol, the SCSI command is packaged into an iSCSI packet, then packaged into a TCP message, and finally packaged into an IP message. iSCSI protects reliable delivery of data blocks through TCP connection-oriented protocols. Since iSCSI is based on an IP protocol stack, it can be transported over standard Ethernet devices through a route or switch.
In this embodiment of the present application, the indication information to be processed includes IP header information, TCP header information, iSCSI header information, and SCSI command, and the SCSI command may also be referred to as target indication information.
Because other protocol data may exist in the data received by the intelligent network card, the intelligent network card can filter the received data to obtain the target indication information. Specifically, when the intelligent network card determines that the to-be-processed indication information contains head information of the iSCSI, the to-be-processed indication information can be determined to contain a specified protocol field, and then the target indication information is extracted from the to-be-processed indication information based on a protocol format of the iSCSI protocol.
For example, the intelligent network card receives a data processing request 1, the data processing request carries to-be-processed indication information 1, the to-be-processed indication information 1 includes target indication information 1, and the target indication information 1 is used for indicating to write data.
S203, obtaining a target hash value of the target indication information based on a preset hash algorithm, and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node.
As a possible implementation manner, when the target hash value of the target indication information is obtained based on a preset hash algorithm, the intelligent network card may adopt the steps of:
a1, splitting the target indication information according to the preset block storage size to obtain at least one data block.
In the embodiment of the application, for supporting the block storage service, after the target indication information is obtained, the target indication information may be split according to a preset block storage size, so as to obtain at least one data block. Illustratively, the preset block storage size may be 4 Megabytes (MB).
And A2, performing hash calculation on at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
Specifically, when step a2 is executed, the following methods can be adopted:
mode 1: and directly carrying out hash calculation on the data block.
The intelligent network card directly performs hash calculation on at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
The hash algorithm is used to transform an input of arbitrary length into an output of fixed length through the hash algorithm, and the output is a hash value. The hash function enables a more rapid and efficient access process to a data sequence, by which the data elements are located more quickly. The hash algorithm may employ, but is not limited to: direct addressing, numerical analysis, square-of-medium methods, and the like.
Mode 2: and after the data block is encrypted, performing hash calculation.
The intelligent network card encrypts the at least one data block by adopting a preset encryption algorithm to obtain at least one encrypted data block, and performs hash calculation on the at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
Mode 3: and compressing the data block and then performing hash calculation.
The intelligent network card encrypts at least one data block by adopting a preset encryption algorithm to obtain at least one compressed data block, and performs hash calculation on at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
Mode 4: and encrypting and compressing the data block, and then performing Hash calculation.
The intelligent network card encrypts the at least one data block by adopting a preset encryption algorithm to obtain at least one encrypted data block, encrypts the at least one encrypted data block by adopting the preset encryption algorithm to obtain at least one compressed data block, and performs hash calculation on the at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
Mode 5: and compressing and encrypting the data block, and then performing Hash calculation.
The intelligent network card encrypts the at least one encrypted data block by adopting a preset encryption algorithm to obtain at least one compressed data block, encrypts the at least one compressed data block by adopting the preset encryption algorithm to obtain at least one encrypted data block, and performs hash calculation on the at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
As another possible implementation manner, in this embodiment of the application, after at least one operation of encryption, compression, and the like is performed on the target indication information, the corresponding processed indication information is obtained, then the processed indication information is divided according to a preset block storage size to obtain at least one data block, and then a preset hash algorithm is used to perform hash calculation on the at least one data block to obtain a target hash value of the target indication information. Since the encryption and compression processes are similar to those described above, they will not be described in detail herein.
In the embodiment of the application, hash calculation can be performed on the target indication information directly based on a preset hash algorithm to obtain the target hash value of the target indication information.
The intelligent network card may pre-store mapping relationships between different hash values and the storage nodes, and after the intelligent network card obtains the hash value of the target indication information, the target storage node corresponding to the target hash value may be determined based on the preset mapping relationship between the hash value and the storage node.
For example, assuming that a hash value 1 corresponds to the storage node 1 and a hash value 2 corresponds to the storage node 2, after the smart network card obtains the hash value of the target indication information, if the hash value of the target indication information is the same as the value of the hash value 1, it is determined that the target storage node corresponding to the target hash value is the storage node 1.
S204, sending the target indication information to the target storage node based on the routing information of the target storage node, so that the target storage node performs data processing based on the target indication information.
Specifically, the intelligent network card encapsulates the target indication information based on a specified protocol format to obtain the encapsulated indication information, and then sends the encapsulated indication information to the target storage node based on the routing information of the target storage node.
The specified protocol format may be a TCP/Remote Direct Memory Access (RDMA) protocol.
For example, the intelligent network card encapsulates the target indication information based on a protocol format of an RDMA protocol to obtain the encapsulated indication information, and then sends the encapsulated indication information to the target storage node based on the routing information of the target storage node.
In some implementations, in order to improve processing efficiency, after the intelligent network card receives the data processing request, the data processing request may be stored in the target network card queue first, and then the data processing request may be obtained from the target network card queue according to the queue order, so as to process the data processing request.
Specifically, the intelligent network card determines a target network card queue corresponding to the destination address contained in the indication information to be processed based on the mapping relationship between the address and the network card queue, stores the data processing request in the target network card queue, and acquires the data processing request from the target network card queue.
It should be noted that, in this embodiment of the present application, the destination address may be represented by an IP address, and may also be represented by an IP address and a port number.
For example, assume that, in the mapping relationship between the addresses and the network card queues, address 1 corresponds to network card queue 1, address 2 corresponds to network card queue 2, the destination address included in the indication information to be processed is address 1, and the intelligent network card determines, based on the mapping relationship between the addresses and the network card queues, that the destination network card queue included in the indication information to be processed and corresponding to the destination address is network card queue 1, then stores the data processing request in network card queue 1, and further obtains the data processing request from network card queue 1.
The present application will be described below with reference to a specific example.
Referring to fig. 4, a network card driver, a Work (Work) thread, an iSCSI thread, a block storage client, an encryption module, a compression module, and the like are deployed in the intelligent network card, and specifically, the data processing flow is as follows:
and receiving a data processing request of a user through the network card drive, determining a target network card queue corresponding to the destination address based on the mapping relation between the address and the network card queue, and sending the target network card queue to the destination network card queue according to the route.
And acquiring a data processing request from the target network card queue through the Work thread, determining whether the indication information to be processed contains a specified protocol field, and if the indication information to be processed contains the specified protocol field, sending the indication information to be processed to the iSCSI thread. The method for acquiring the data processing request from the target network card queue by the word thread can adopt a poll () method, and the poll () method is used for taking out a first queue element from the head of the queue.
And extracting target indication information from the indication information to be processed through an iSCSI thread based on a target protocol format, and sending the target indication information to the block storage routing client.
And splitting the target indication information according to the preset block storage size through the block storage routing client to obtain at least one data block. At least one data block is sent to the encryption module via the block store routing client.
And the encryption module is used for delivering at least one data block to a third party for encryption or a local storage agent for encryption according to a preset encryption algorithm to obtain at least one encrypted data block, and sending the at least one encrypted data block to the compression module.
And compressing the at least one encrypted data block by the compression module according to a preset compression algorithm to obtain at least one compressed data block.
And obtaining a target hash value of the target indication information based on a preset hash algorithm through the block storage routing client, and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node.
And packaging the target indication information based on a specified protocol format through a word thread to obtain packaged indication information, and sending the packaged indication information to the target storage node through a network card queue based on the routing information of the target storage node.
Therefore, in the embodiment of the application, after the iSCSI gateway service and block storage client is unloaded to the intelligent network card, protocol stacks do not need to be switched, the multi-queue and zero-copy advantages of the intelligent network card are utilized, the packet receiving capacity based on the iSCSI gateway can be greatly improved, the utilization rate of the network card is improved, and after the iSCSI protocol stacks are unloaded to the intelligent network card, a large amount of CPU resources are saved, and the cost of a data center is reduced.
In addition, the gap between the computing power of the CPU and the transmission rate of the network is continuously expanding, the smart network card can carry diversified functions such as virtual switching, storage, data and network encryption, and after the iSCSI gateway service is offloaded to the smart network card, the features provided by software (e.g., encryption and decryption, etc.) can be provided by hardware (e.g., such as a Field Programmable Gate Array (FPGA)) without the participation of the CPU, and the smart network card can be plugged and used in various heterogeneous environments.
The iSCSI protocol is unloaded to the intelligent network card, the front-end client and the rear-end storage cluster network are opened, an integrated gateway solution is provided, the intelligent network card provides iSCSI service, the iSCSI service is directly connected with the rear-end storage, an IO Path ByPass CPU is connected, no software stack switching is needed, the characteristic of multiple queues of the intelligent network card is utilized, the iSCSI protocol packet is zero-copied to a block storage client, and the bandwidth capacity of the network card is fully exerted.
Based on the same inventive concept, in the embodiment of the present application, a data processing apparatus 500 is provided, and referring to fig. 5, the data processing apparatus 500 at least includes:
a receiving unit 501, configured to receive a data processing request, where the data processing request carries instruction information to be processed;
a protocol processing unit 502, configured to, if the to-be-processed indication information includes a specified protocol field, extract target indication information from the to-be-processed indication information based on a target protocol format;
the route processing unit 503 is configured to obtain a target hash value of the target indication information based on a preset hash algorithm, and determine a target storage node corresponding to the target hash value based on a mapping relationship between a preset hash value and a storage node;
a sending unit 504, configured to send the target indication information to the target storage node based on the routing information of the target storage node, so that the target storage node performs data processing based on the target indication information.
As a possible implementation manner, the to-be-processed indication information includes a destination address, and the protocol processing unit 502 is further configured to:
determining a target network card queue corresponding to the destination address based on a mapping relation between the address and the network card queue, and storing the data processing request in the target network card queue;
and acquiring the data processing request from the target network card queue.
As a possible implementation manner, when the target hash value of the target indication information is obtained based on a preset hash algorithm, the route processing unit 503 is specifically configured to:
splitting the target indication information according to a preset block storage size to obtain at least one data block;
and performing hash calculation on the at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
As a possible implementation manner, after extracting the target indication information from the to-be-processed indication information based on the target protocol format, the route processing unit 503 is further configured to perform at least one of the following operations:
encrypting the target indication information by adopting a preset encryption algorithm to obtain processed indication information; alternatively, the first and second electrodes may be,
and compressing the target indication information by adopting a preset encryption algorithm to obtain the processed indication information.
As a possible implementation manner, when the target hash value of the target indication information is obtained based on a preset hash algorithm, the route processing unit 503 is specifically configured to:
and performing hash calculation on the processed indication information by adopting a preset hash algorithm to obtain the processed indication information, and taking the target hash value as the target hash value of the target indication information.
As a possible implementation manner, when the target indication information is sent to the target storage node based on the routing information of the target storage node, the sending unit 504 is specifically configured to:
based on a specified protocol format, packaging the target indication information to obtain packaged indication information;
and sending the encapsulated indication information to the target storage node based on the routing information of the target storage node.
Based on the same inventive concept, referring to fig. 6, an embodiment of the present application provides an electronic device, which at least includes:
a memory 601 for storing executable instructions;
a processor 602 configured to read and execute executable instructions stored in the memory to implement the data processing method in the embodiment of the present disclosure, such as the steps shown in fig. 2 or fig. 4.
A transceiver 603 for receiving and transmitting data under the control of the processor 602.
Wherein in fig. 6 the bus architecture may comprise any number of interconnected buses and bridges, with one or more processors, represented in particular by processor 602, and various circuits of memory, represented by memory 601, being linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The transceiver 603 may be a plurality of elements, including a transmitter and a transceiver, providing a means for communicating with various other apparatus over a transmission medium. The processor 602 is responsible for managing the bus architecture and general processing, and the memory 601 may store data used by the processor 602 in performing operations.
Based on the same inventive concept, the present application provides a storage medium, and when instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods implemented by the data processing apparatus in the above-mentioned flow.
Based on the same inventive concept, the present application also provides a computer program product or a computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data processing method provided in the various optional implementation manners in the above embodiments.
In some possible embodiments, various aspects of the detection method of the root cause task provided by the present application may also be implemented in the form of a program product including a computer program for causing a computer device to perform the steps in the data processing method according to various exemplary embodiments of the present application described above in this specification when the program product is run on the computer device, for example, the computer device may perform the steps as shown in fig. 2 or fig. 3.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The program product of the embodiments of the present application may be a CD-ROM and include program code and may run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with a command execution system, apparatus, or device.
A readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with a command execution system, apparatus, or device. While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
For the system/apparatus embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for relevant points.
It is to be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or operation from another entity or operation without necessarily requiring or implying any actual such relationship or order between such entities or operations.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A data processing method is applied to an intelligent network card and comprises the following steps:
receiving a data processing request, wherein the data processing request carries indication information to be processed;
if the to-be-processed indicating information contains a specified protocol field, extracting target indicating information from the to-be-processed indicating information based on a target protocol format;
obtaining a target hash value of the target indication information based on a preset hash algorithm, and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node;
and sending the target indication information to the target storage node based on the routing information of the target storage node so as to enable the target storage node to perform data processing based on the target indication information.
2. The method of claim 1, wherein the pending indication information includes a destination address;
after the receiving the data processing request, before determining that the to-be-processed indication information includes a specified protocol field, the method further includes:
determining a target network card queue corresponding to the destination address based on a mapping relation between the address and the network card queue, and storing the data processing request in the target network card queue;
and acquiring the data processing request from the target network card queue.
3. The method as claimed in claim 1 or 2, wherein the obtaining the target hash value of the target indication information based on a preset hash algorithm comprises:
splitting the target indication information according to a preset block storage size to obtain at least one data block;
and performing hash calculation on the at least one data block by adopting a preset hash algorithm to obtain a target hash value of the target indication information.
4. The method according to claim 1 or 2, wherein after extracting the target indication information from the indication information to be processed based on the target protocol format, the method further comprises at least one of the following operations:
encrypting the target indication information by adopting a preset encryption algorithm to obtain processed indication information; alternatively, the first and second electrodes may be,
and compressing the target indication information by adopting a preset encryption algorithm to obtain the processed indication information.
5. The method as claimed in claim 4, wherein the obtaining the target hash value of the target indication information based on a preset hash algorithm comprises:
and performing hash calculation on the processed indication information by adopting a preset hash algorithm to obtain the processed indication information, and taking the target hash value as the target hash value of the target indication information.
6. The method of claim 1 or 2, wherein the sending the target indication information to the target storage node based on the routing information of the target storage node comprises:
based on a specified protocol format, packaging the target indication information to obtain packaged indication information;
and sending the encapsulated indication information to the target storage node based on the routing information of the target storage node.
7. A data processing apparatus, comprising:
the device comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving a data processing request which carries indication information to be processed;
the protocol processing unit is used for extracting target indication information from the indication information to be processed based on a target protocol format if the indication information to be processed contains a specified protocol field;
the route processing unit is used for obtaining a target hash value of the target indication information based on a preset hash algorithm and determining a target storage node corresponding to the target hash value based on a mapping relation between the preset hash value and the storage node;
and the sending unit is used for sending the target indication information to the target storage node based on the routing information of the target storage node so as to enable the target storage node to perform data processing based on the target indication information.
8. The apparatus of claim 7, wherein the pending indication information includes a destination address, and the protocol processing unit is further configured to:
determining a target network card queue corresponding to the destination address based on a mapping relation between the address and the network card queue, and storing the data processing request in the target network card queue;
and acquiring the data processing request from the target network card queue.
9. An electronic device, characterized in that it comprises a processor and a memory, wherein the memory stores a computer program which, when executed by the processor, causes the processor to carry out the steps of the method of any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that it comprises a computer program for causing an electronic device to carry out the steps of the method of any one of claims 1 to 6, when the computer program is run on the electronic device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111643559.2A CN114465770A (en) | 2021-12-29 | 2021-12-29 | Data processing method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111643559.2A CN114465770A (en) | 2021-12-29 | 2021-12-29 | Data processing method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114465770A true CN114465770A (en) | 2022-05-10 |
Family
ID=81408115
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111643559.2A Pending CN114465770A (en) | 2021-12-29 | 2021-12-29 | Data processing method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114465770A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114969851A (en) * | 2022-05-31 | 2022-08-30 | 浪潮电子信息产业股份有限公司 | Data processing method, device, equipment and medium based on FPGA |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103441948A (en) * | 2013-07-03 | 2013-12-11 | 华为技术有限公司 | Data access method, network card and storage system |
| CN104054071A (en) * | 2012-12-14 | 2014-09-17 | 华为技术有限公司 | Method for accessing storage device and storage device |
| CN106293490A (en) * | 2015-05-12 | 2017-01-04 | 中兴通讯股份有限公司 | Data storage, the method read, Apparatus and system |
| CN110535969A (en) * | 2019-09-16 | 2019-12-03 | 腾讯科技（深圳）有限公司 | Date storage method, device, storage medium and equipment based on block chain network |
| US20200133668A1 (en) * | 2018-10-24 | 2020-04-30 | EMC IP Holding Company LLC | DATA STORAGE OPTIMIZATION USING REPLICATION STATISTICS TO AUTOMATICALLY GENERATE NVMe STREAM IDENTIFIERS |
| CN113014650A (en) * | 2021-03-01 | 2021-06-22 | 中国工商银行股份有限公司 | Processing method and device for data request, computing equipment and medium |
| CN113778320A (en) * | 2020-06-09 | 2021-12-10 | 华为技术有限公司 | Network card and method for processing data by network card |
-
2021
- 2021-12-29 CN CN202111643559.2A patent/CN114465770A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104054071A (en) * | 2012-12-14 | 2014-09-17 | 华为技术有限公司 | Method for accessing storage device and storage device |
| CN103441948A (en) * | 2013-07-03 | 2013-12-11 | 华为技术有限公司 | Data access method, network card and storage system |
| CN106293490A (en) * | 2015-05-12 | 2017-01-04 | 中兴通讯股份有限公司 | Data storage, the method read, Apparatus and system |
| US20200133668A1 (en) * | 2018-10-24 | 2020-04-30 | EMC IP Holding Company LLC | DATA STORAGE OPTIMIZATION USING REPLICATION STATISTICS TO AUTOMATICALLY GENERATE NVMe STREAM IDENTIFIERS |
| CN110535969A (en) * | 2019-09-16 | 2019-12-03 | 腾讯科技（深圳）有限公司 | Date storage method, device, storage medium and equipment based on block chain network |
| CN113778320A (en) * | 2020-06-09 | 2021-12-10 | 华为技术有限公司 | Network card and method for processing data by network card |
| CN113014650A (en) * | 2021-03-01 | 2021-06-22 | 中国工商银行股份有限公司 | Processing method and device for data request, computing equipment and medium |
Non-Patent Citations (2)
| Title |
|---|
| SAMSUNG: "Mobile Terminated IP Services", 3GPP TSG SA WG2 MEETING #76 TD S2-096645, no. 2 * |
| 黄建忠;谢长生;朱光喜;吴伟;: "一种基于iSCSI的对象存储安全系统的设计与实现", 计算机科学, no. 04 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114969851A (en) * | 2022-05-31 | 2022-08-30 | 浪潮电子信息产业股份有限公司 | Data processing method, device, equipment and medium based on FPGA |
| CN114969851B (en) * | 2022-05-31 | 2024-02-23 | 浪潮电子信息产业股份有限公司 | FPGA-based data processing method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11489919B2 (en) | Method, apparatus, and data processing system including controller to manage storage nodes and host operations | |
| CN108028833B (en) | NAS data access method, system and related equipment | |
| US7219121B2 (en) | Symmetrical multiprocessing in multiprocessor systems | |
| WO2022126919A1 (en) | File transmission method and apparatus, and computer device and storage medium | |
| US9015519B2 (en) | Method and system for cluster wide adaptive I/O scheduling by a multipathing driver | |
| CN113326228B (en) | Message forwarding method, device and equipment based on remote direct data storage | |
| CN114201421B (en) | Data stream processing method, storage control node and readable storage medium | |
| US20220244861A1 (en) | Data Access Method and Apparatus, and First Computing Device | |
| US11201836B2 (en) | Method and device for managing stateful application on server | |
| CN108491278B (en) | Method and network device for processing service data | |
| EP4318251A1 (en) | Data access system and method, and device and network card | |
| CN113778320A (en) | Network card and method for processing data by network card | |
| CN114465770A (en) | Data processing method and related device | |
| CN113411363A (en) | Uploading method of image file, related equipment and computer storage medium | |
| CN111225003A (en) | NFS node configuration method and device | |
| CN110609746A (en) | Method, apparatus and computer program product for managing network system | |
| CN108829340B (en) | Storage processing method, device, storage medium and processor | |
| US8041902B2 (en) | Direct memory move of multiple buffers between logical partitions | |
| CN107483637B (en) | NFS-based client link management method and device | |
| US10289550B1 (en) | Method and system for dynamic write-back cache sizing in solid state memory storage | |
| CN114003342A (en) | Distributed storage method and device, electronic equipment and storage medium | |
| WO2017177400A1 (en) | Data processing method and system | |
| CN113422792A (en) | Data transmission method and device, electronic equipment and computer storage medium | |
| CN105874757A (en) | Data processing method and multi-core processor system | |
| CN114327262B (en) | Method and device for maintaining port mapping of intelligent network card |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |